由于watchdog超时、StorageGRID 设备SG5700意外重新启动
适用场景
- NetApp StorageGRID 11.5及更高版本
- StorageGRID设备SG5700
问题描述
- 一个或多个节点意外重新启动警报电子邮件：
Unexpected node reboot (1 alert)
A node rebooted unexpectedly within the last 24 hours.
Recommended actions
1. Monitor this alert. The alert will be cleared after 24 hours. However, if the node reboots unexpectedly again, this alert will be triggered again.
2. If you cannot resolve the alert, there might be a hardware failure. Contact technical support.
________________________________________
dc1-sn1
Node dc1-sn1
Site DC1
Severity Major
Time triggered WKD MMM DD hh:mm:ss UTC YYYY
Job miscd- 在导致重新启动的相关节点上报告CPU利用率较高。可以在 StorageGRID网格管理器的节点视图下以及节点或节点(内部使用)视图下查看CPU使用情况。
- 由于监视程序超时、此节点将重新启动：
- 收集 受影响节点上的日志文件和系统数据
- 提取 .tar.gz日志文件
- 找到崩溃dmesg文件：
base-os-logs\run\mount-tmp\pge-actv-root\var\log\storagegrid_crash_dmesg.YYYYMMDDhhmmss.log.gz - 提取并打开崩溃dmesg文件、验证节点是否因看门狗 超时而重新启动：
[sss.uuuuuu] fpga_pci: fpgaIsr: fpgaIsr fired, interruptCount = 1
[sss.uuuuuu] fpga_pci: fpgaIsr: fpgaIsr logging memory usage
...
[sss.uuuuuu] fpga_pci: fpgaIsr: fpgaIsr logging CPU backtrace
...
[sss.uuuuuu] fpga_pci: fpgaIsr: fpgaIsr logging blocked tasks
...
[sss.uuuuuu] fpga_pci: fpgaIsr: fpgaIsr logging all ftrace buffers
...
[sss.uuuuuu] fpga_pci: fpgaIsr: fpgaIsr serviced watchdog