周六没事在家闲来无事,正看着《赢在中国蓝天碧水间》12位Boss比赛中,此时手机监控收到一台服务器不稳定告警短信,此时登陆服务器能登陆,不过反应很慢慢,敲一个命令就卡半天。检查系统的CPU、内存等性能指标,发现都很低。此时认为硬盘有坏道,并通过检查工具来检查
检查命令: badblocks -v /dev/sda >123.txt
通过命令检查无异常现象,在检查IO性能指标查看业务异常
# iostat 2 10 # iotop
此时dmesg命令查看发现网卡etho模式时常改变,
#dmesg device bond0 left promiscuous mode device eth0 left promiscuous mode device bond0 entered promiscuous mode device eth0 entered promiscuous mode device bond0 left promiscuous mode device eth0 left promiscuous mode device bond0 entered promiscuous mode device eth0 entered promiscuous mode device bond0 left promiscuous mode device eth0 left promiscuous mode device bond0 entered promiscuous mode device eth0 entered promiscuous mode device bond0 left promiscuous mode device eth0 left promiscuous mode device bond0 entered promiscuous mode device eth0 entered promiscuous mode device bond0 left promiscuous mode device eth0 left promiscuous mode
也查看/var/log/messgae 记录的文件中也包含eth0模式改变的信息。
Apr 20 14:11:30 kernel: device bond0 left promiscuous mode Apr 20 14:11:30 kernel: device eth0 left promiscuous mode Apr 20 14:11:33 kernel: device bond0 entered promiscuous mode Apr 20 14:11:33 kernel: device eth0 entered promiscuous mode Apr 20 14:11:34 kernel: device bond0 left promiscuous mode Apr 20 14:11:34 kernel: device eth0 left promiscuous mode Apr 20 14:11:58 kernel: device bond0 entered promiscuous mode Apr 20 14:11:58 kernel: device eth0 entered promiscuous mode Apr 20 14:12:00 kernel: device bond0 left promiscuous mode Apr 20 14:12:00 kernel: device eth0 left promiscuous mode
通过ifconfig -a 查看eth0网卡信息,发现很大error数据包。
eth0 Link encap:Ethernet HWaddr 8C:89:A5:DB:65:14 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:770561850 errors:1872182 dropped:0 overruns:0 frame:1872182 TX packets:701398846 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:790767401025 (736.4 GiB) TX bytes:475192351462 (442.5 GiB) Memory:fbe60000-fbe7ffff
系统使用的是双网卡模式,此时关闭了其中一个网卡观察。
ifconfig eth0 down ifconfig eth0 up
目前观察2小时后系统运行正常,无异常情况 。