CONTAP-364039：意外心跳丢失导致 FAS8300/AFF A400 MetroCluster IP 系统上的接管事件
问题描述
- 频繁的心跳未命中，这可能导致接管一个节点。
- 在心跳未命中之前，正在进行接管的节点将无响应，在此期间没有节点日志。
- 发生接管的无响应节点不会在问题发生时报告任何硬件错误。
- 接管的合作伙伴节点通常会记录 SCSI IO 传输错误、NVMM 镜像同步失败、QP 错误、HA 错误等。
- 合作伙伴节点日志示例：
Jan 01 01:01:01 [node_name: DR_heartbeat_thread: cf.ic.xferTimedOut:error]: HA interconnect: MCC_DRSOM transfer timed out.
Jan 01 01:01:01 [node_name: DR_heartbeat_thread: rdma.rlib.event.error:debug]: QP drsom_hb event error: client disconnect.
Jan 01 01:01:01 [node_name: iw_cxgbe: nvmm.mirror.aborting:debug]: mirror of sysid 1, partner_type HA Partner and mirror state NVMM_MIRROR_ONLINE is aborted because of reason NVMM_ERR_MIRROR_COMPLETION.
Jan 01 01:01:01 [node_name: rastrace_dump: rastrace.dump.saved:debug]: A RAS trace dump for module IC instance 0 was stored in /etc/log/rastrace/IC_0_20250101_01:01:01:123456.dmp.
Jan 01 01:01:01 [node_name: cf_firmware: cf.ic.xferTimedOut:error]: HA interconnect: OFW transfer timed out.
Jan 01 01:01:03 [node_name: cf_main: cf.fsm.takeoverByPartnerDisabled:error]: Failover monitor: takeover of node_name-01 by node_name-02 disabled (HA interconnect error. Verify that the partner node is running and that the HA interconnect cabling is correct, if applicable. For further assistance, contact technical support).
Jan 01 01:01:04 [node_name: cf_main: cf.fsm.partnerNotResponding:notice]: Failover monitor: partner not responding
Jan 01 01:02:01 [node_name: cf_main: cf.fsm.takeoverCountdown:info]: Failover monitor: takeover scheduled in 10 seconds
Jan 01 01:02:10 [node_name: cf_main: cf.fsm.takeover.noHeartbeat:alert]: Failover monitor: Takeover initiated after no heartbeat was detected from the partner node.