前面写了如何优化TCP服务端，这次来写写如何查看连接状态和排除相关故障。
查看连接状态
这里用netstat和ss命令来查看连接。ss 直接从内核读取数据，比netstat速度更快，信息更具体。
- https://manpages.ubuntu.com/manpages/xenial/man8/ss.8.html
-s: summary-t/-u/-x: tcp, udp, unix-4/-6: ipv4, ipv6-l/-a: listening, all-n/-r: numeric, resolve-p: process-e: extended detail
- https://manpages.ubuntu.com/manpages/xenial/man8/netstat.8.html
- http://manpages.ubuntu.com/manpages/bionic/en/man8/lsof.8.html
netstat
# show interfaces
❯ netstat -i
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
br-a9e62987d96f 1500 0 409106 0 0 0 380745 0 0 0 BMRU
br-bf4cc1133650 1500 0 6290 0 0 0 6771 0 0 0 BMRU
docker0 1500 0 72493 0 0 0 99569 0 0 0 BMRU
ens3 1500 0 688697 0 0 0 769598 0 0 0 BMRU
lo 65536 0 102857 0 0 0 102857 0 0 0 LRU
veth4038775 1500 0 11647 0 0 0 11642 0 0 0 BMRU
# show summary
❯ netstat -s
Ip:
1071739 total packets received
660903 forwarded
1 with unknown protocol
0 incoming packets discarded
410797 incoming packets delivered
1189033 requests sent out
Icmp:
4493 ICMP messages received
33 input ICMP message failed.
ICMP input histogram:
destination unreachable: 58
timeout in transit: 96
echo requests: 4339
4513 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 174
echo replies: 4339
IcmpMsg:
InType3: 58
InType8: 4339
InType11: 96
OutType0: 4339
OutType3: 174
Tcp:
30484 active connections openings
16334 passive connection openings
14586 failed connection attempts
551 connection resets received
32 connections established
471876 segments received
569012 segments send out
94995 segments retransmited
1697 bad segments received.
20975 resets sent
InCsumErrors: 1694
Udp:
20750 packets received
190 packets to unknown port received.
0 packet receive errors
20908 packets sent
UdpLite:
...
IpExt:
InNoRoutes: 38
InOctets: 299257215
OutOctets: 389205073
InNoECTPkts: 1078184
InECT0Pkts: 327
InCEPkts: 171
# show tcp, ipv4, listening, process
❯ netstat -t4lp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 *:https *:* LISTEN 10822/nginx: master
tcp 0 0 localhost:10088 *:* LISTEN 4370/photobox
tcp 0 0 *:http *:* LISTEN 10822/nginx: master
tcp 0 0 *:ssh *:* LISTEN 1139/sshd
tcp 0 0 localhost:3000 *:* LISTEN 1342/dockerd
tcp 0 0 localhost:10200 *:* LISTEN 4416/markdir
# show connections with process detail
❯ netstat -t4ap | grep -v LISTEN
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 v.bitsflow.org:http strasbourg-game-1:25636 SYN_RECV -
tcp 0 0 v.bitsflow.org:https 119.248.174.68:27833 ESTABLISHED 10825/nginx: worker
tcp 0 0 v.bitsflow.org:http 144.207.200.:ingreslock SYN_RECV -
tcp 0 0 v.bitsflow.org:https 57.36.9.191:23203 SYN_RECV -
tcp 0 0 v.bitsflow.org:https 172.21.0.3:55122 ESTABLISHED 10825/nginx: worker
tcp 0 0 localhost:45054 localhost:6379 ESTABLISHED 5210/index:app uWSG
tcp 0 0 v.bitsflow.org:https 172.21.0.3:55104 ESTABLISHED 10825/nginx: worker
tcp 0 0 v.bitsflow.org:http reston-game-1.oct:12757 SYN_RECV -
tcp 0 0 v.bitsflow.org:https 172.21.0.3:55098 ESTABLISHED 10825/nginx: worker
tcp 0 0 v.bitsflow.org:http 153.79.222.185:18148 SYN_RECV -
tcp 0 0 v.bitsflow.org:https 172.21.0.3:55094 ESTABLISHED 10825/nginx: worker
tcp 0 0 v.bitsflow.org:https 172.21.0.3:55136 ESTABLISHED 10825/nginx: worker
...
# show routing table
❯ netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default 45.32.76.1 0.0.0.0 UG 0 0 0 ens3
45.32.76.0 * 255.255.252.0 U 0 0 0 ens3
169.254.169.254 45.32.76.1 255.255.255.255 UGH 0 0 0 ens3
172.17.0.0 * 255.255.0.0 U 0 0 0 docker0
172.21.0.0 * 255.255.0.0 U 0 0 0 br-a9e62987d96f
172.25.0.0 * 255.255.0.0 U 0 0 0 br-bf4cc1133650
# show state statistics, like `ss -s`
❯ netstat -t4an | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
LISTEN 26
SYN_RECV 39
ESTABLISHED 30
FIN_WAIT1 1
FIN_WAIT2 1
TIME_WAIT 21
ss
❯ ss -s
Total: 2556 (kernel 3106)
TCP: 2045 (estab 38, closed 1926, orphaned 0, synrecv 0, timewait 80/0), ports 0
Transport Total IP IPv6
* 3106 - -
RAW 0 0 0
UDP 35 8 27
TCP 119 63 56
INET 154 71 83
FRAG 0 0 0
❯ ss -t4lp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 511 *:https *:* users:(("nginx",pid=10825,fd=7),("nginx",pid=10822,fd=7))
LISTEN 0 100 127.0.0.1:20000 *:* users:(("uwsgi",pid=23675,fd=3),("uwsgi",pid=23674,fd=3),("uwsgi",pid=23671,fd=3))
LISTEN 0 65535 127.0.0.1:1024 *:* users:(("album",pid=4418,fd=3))
...
❯ ss -t4ap | grep -v LISTEN
State Recv-Q Send-Q Local Address:Port Peer Address:Port
SYN-RECV 0 0 45.32.78.161:http 45.77.95.134:6138
SYN-RECV 0 0 45.32.78.161:http 119.241.179.244:9194
SYN-RECV 0 0 45.32.78.161:http 46.105.42.71:20793
SYN-RECV 0 0 45.32.78.161:http 74.91.119.24:46336
TIME-WAIT 0 0 127.0.0.1:38432 127.0.0.1:5000
SYN-RECV 0 0 45.32.78.161:http 139.99.234.45:12935
TIME-WAIT 0 0 127.0.0.1:54278 127.0.0.1:20000
ESTAB 0 0 45.32.78.161:https 172.21.0.3:58014 users:(("nginx",pid=10825,fd=29))
ESTAB 0 0 127.0.0.1:45054 127.0.0.1:6379 users:(("uwsgi",pid=5210,fd=6))
ESTAB 0 0 45.32.78.161:https 172.21.0.3:58038 users:(("nginx",pid=10825,fd=40))
SYN-RECV 0 0 45.32.78.161:http 45.85.188.18:43467
TIME-WAIT 0 0 127.0.0.1:8099 127.0.0.1:15120
SYN-RECV 0 0 45.32.78.161:http 139.99.234.46:26533
TIME-WAIT 0 0 127.0.0.1:53568 127.0.0.1:http-alt
ESTAB 0 0 127.0.0.1:17574 127.0.0.1:6379 users:(("cute",pid=4367,fd=6))
ESTAB 0 0 45.32.78.161:https 183.11.29.134:4970 users:(("nginx",pid=10825,fd=22))
TIME-WAIT 0 0 127.0.0.1:10088 127.0.0.1:19286
SYN-RECV 0 0 45.32.78.161:http 195.103.116.249:11408
SYN-RECV 0 0 45.32.78.161:http 172.105.102.212:44387
TIME-WAIT 0 0 127.0.0.1:17236 127.0.0.1:6379
ESTAB 0 0 127.0.0.1:15682 127.0.0.1:5678 users:(("nginx",pid=10825,fd=16))
...
lsof
❯ lsof -i -n -P
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postgres 693 postgres 6u IPv6 18619 0t0 TCP [::1]:5433 (LISTEN)
postgres 693 postgres 7u IPv4 18620 0t0 TCP 127.0.0.1:5433 (LISTEN)
postgres 693 postgres 11u IPv6 18670 0t0 UDP [::1]:49392->[::1]:49392
sshd 1139 root 3u IPv4 20007 0t0 TCP *:22 (LISTEN)
node 1500 root 19u IPv4 23340 0t0 TCP *:2081 (LISTEN)
master 1683 root 12u IPv4 23025 0t0 TCP 127.0.0.1:25 (LISTEN)
master 1683 root 13u IPv6 23026 0t0 TCP [::1]:25 (LISTEN)
superviso 4155 root 4u IPv4 32031 0t0 TCP 127.0.0.1:9001 (LISTEN)
markdir 4416 root 3u IPv4 34567 0t0 TCP 127.0.0.1:10200 (LISTEN)
python 4479 root 3u IPv4 38437 0t0 TCP 127.0.0.1:5000 (LISTEN)
uwsgi 4488 root 3u IPv4 34951 0t0 TCP 127.0.0.1:8080 (LISTEN)
nginx 10822 root 6u IPv4 75372 0t0 TCP *:80 (LISTEN)
nginx 10822 root 7u IPv4 75373 0t0 TCP *:443 (LISTEN)
nginx 10822 root 8u IPv4 75374 0t0 UDP *:443
sshd 14509 root 3u IPv4 505596 0t0 TCP 45.32.78.161:22->222.125.220.101:42896 (ESTABLISHED)
uwsgi 23675 root 3u IPv4 534390 0t0 TCP 127.0.0.1:20000 (LISTEN)
...
TCP抓包与pcap
fish shell function to capture packages related to specific IP
function tshark-by-ip
sudo tshark --color -i en0 -f "(tcp or udp or icmp) and net $argv[1] $argv[2]"
end
