FabricPool ESXi SAN主机修复

适用场景

• ONTAP FabricPool
• ESXi上托管的VMware ESXi和子操作系统
• VMFS
• RDM
• VVOL

问题描述

• 在ONTAP FabricPool S3分层中断期间、ESXi上托管的子操作系统可能会报告：
  • I/O故障、
  • 只读文件系统、
  • 子操作系统崩溃等
• 在某些情况下、VMFS数据存储库可能会变为只读/不可访问状态。  ESXi vmkernel日志会将此错误记录为子操作系统的中等错误和I/O故障。

下面显示了来自ESXi的错误消息示例：

ESXi中型错误

2023-07-03T17:26:35.282Z cpu20:1049308)ScsiDeviceIO: 4161: Cmd(0x4598f5de3cc8) 0x28, CmdSN 0x651 from world 1050920 to dev "naa.600a098038314b39335d554871744465" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc Medium Error, LBA: 1706672

2023-07-03T17:26:35.777Z cpu32:1049302)ScsiDeviceIO: 4161: Cmd(0x4598f5db6cc8) 0x28, CmdSN 0x653 from world 0 to dev "naa.600a098038314b39335d554871744465" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc Medium Error, LBA: 160384

2023-07-03T17:26:37.922Z cpu32:1049302)ScsiDeviceIO: 4161: Cmd(0x4598f5c203c8) 0x28, CmdSN 0x655 from world 0 to dev "naa.600a098038314b39335d554871744465" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc Medium Error, LBA: 160712

2023-07-03T17:26:37.922Z cpu36:1049358)Vol3: 4458: Failed to get object 28 type 3 uuid 643f357d-8ff50e8a-53be-4c52623393dc FD 5000644 gen 28 :I/O error

2023-07-03T17:26:40.391Z cpu32:1049302)ScsiDeviceIO: 4161: Cmd(0x4598f5db62c8) 0x28, CmdSN 0x657 from world 0 to dev "naa.600a098038314b39335d554871744465" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc Medium Error, LBA: 160760

2023-07-03T17:26:40.391Z cpu36:1049358)Vol3: 4458: Failed to get object 28 type 3 uuid 643f357d-8ff50e8a-53be-4c52623393dc FD 5c00644 gen 2b :I/O error

2023-07-03T17:26:41.783Z cpu8:1049303)ScsiDeviceIO: 4161: Cmd(0x4579011028c8) 0x28, CmdSN 0x658 from world 1048865 to dev "naa.600a098038314b39335d554871744465" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc Medium Error, LBA: 1278224

2023-07-03T17:26:41.783Z cpu7:1048865)WARNING: Vol3: 2963: 'FP_45GB_W2K19': Failed to clear journal address since JBC could not be Locked. This could result in leak of journal block at <type 6 addr 9>.

2023-07-03T17:26:41.783Z cpu7:1048865)WARNING: Vol3: 3032: 'FP_45GB_W2K19': Failed to clear journal address in on-disk HB. This could result in leak of journal block at <type 6 addr 9>.

2023-07-03T17:26:41.785Z cpu7:1048865)Vol3: 4367: Error closing the volume: . Eviction fails: Failure

2023-07-03T17:27:18.436Z cpu32:1049198)NMP: nmp_ResetDeviceLogThrottling:3782: last error status from device naa.600a098038314b39335d554871744465 repeated 22 times

2023-07-03T17:27:23.777Z cpu32:1049302)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x2a (0x4598f5c49cc8, 1048870) to dev "naa.600a098038314b39335d554871744543" on path "vmhba3:C0:T1:L3" Failed:

2023-07-03T17:27:23.777Z cpu32:1049302)NMP: nmp_ThrottleLogForDevice:3875: H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc. Act:NONE. cmdId.initiator=0x4304d3426dc0 CmdSN 0x587

2023-07-03T17:27:23.777Z cpu32:1049302)ScsiDeviceIO: 4161: Cmd(0x4598f5c49cc8) 0x2a, CmdSN 0x587 from world 1048870 to dev "naa.600a098038314b39335d554871744543" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc Medium Error, LBA: 17832200

2023-07-03T17:27:23.777Z cpu26:1048870)Fil6: 4072: 'linbtrfsboot': Fil6 file IO (<FD c19 r13>) : I/O error

2023-07-03T17:27:23.777Z cpu26:1048870)Fil6: 4038: ioCtx: 0x4598f597f8c0, world: 1048870, overallStatus: I/O error, token: 0x0, tokenStatus: N, txnToken: 0x0, txnTokenStatus: N, totalIOSize: 2078, maxIOLength: 137246

2023-07-03T17:27:23.777Z cpu26:1048870)Fil6: 4044: numAddrs: 0, numTbz: 0, numPBsToAlloc: 0, numBlocksToAlloc: 0 numTBZBlocks: 0, numPBsToUnmap: 0, numBlocksToUnmap: 0

2023-07-03T17:27:23.777Z cpu26:1048870)Fil6: 4051: opType: 2, ioFlags: 393552, allocFlags: 0, ioStateTxn: 0, ioStateNoTxn: 7, txnState: 2, numResolvedAddrMaps: 1, numTxnPendingAddrMaps: 0

2023-07-03T17:27:23.777Z cpu26:1048870)Fil6: 4056: wakeupOnIOComplete: 0, isIOIssued: 1, isIODone: 1, isRead: 0 isSync: 1, canBlock: 1, lastPartialBlock: 0

2023-07-03T17:27:23.777Z cpu26:1048870)BC: 414: write to vmware.log (f532 28 3 64a149c0 dca15134 524cad38 dc933362 34004c4 17 0 0 0 0 0) 2078 bytes failed: I/O error

2023-07-03T17:27:23.777Z cpu26:1048870)Log: 1577: Generating backtrace for 1048870: bcflushd

2023-07-03T17:27:23.777Z cpu26:1048870)Backtrace for current CPU #26, worldID=1048870, fp=0x100126

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931bad0:[0x420039ce8f55]LogBacktraceInt@vmkernel#nover+0xdd stack: 0x149a00, 0x420039c3e164, 0x45389ba1f140, 0x45388d41f140, 0x45388931f140

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931baf0:[0x420039c3e163]BCFileIO@vmkernel#nover+0x1b4 stack: 0x45388931f140, 0x842e31a8, 0x6c2e657261776d76, 0x45388900676f, 0x45389ba1f140

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931bd60:[0x420039c3eb5b]BCWriteBuffer@vmkernel#nover+0x1fc stack: 0x149a00, 0xffffffffffffffff, 0x453800000000, 0xe1000, 0x12b734000

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931bf70:[0x420039c3ecc7]BCFlushFile@vmkernel#nover+0x44 stack: 0x430707815f10, 0x41fff9d527e0, 0x430707815fd0, 0x420039c3eda7, 0x45388931f000

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931bfa0:[0x420039c3eda6]BCFlushAll@vmkernel#nover+0xc3 stack: 0x45388931f140, 0x0, 0x0, 0x420039c3edf9, 0x0

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931bfd0:[0x420039c3edf8]BCFlusherWorld@vmkernel#nover+0x9 stack: 0x0, 0x0, 0x0, 0x420039cc4a60, 0x0

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931bfe0:[0x420039fb1871]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0, 0x420039cc4a60, 0x0, 0x0, 0x0

2023-07-03T17:27:23.777Z cpu26:1048870)0x45388931c000:[0x420039cc4a5f]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0

2023-07-03T17:27:59.160Z cpu32:1049302)ScsiDeviceIO: 4161: Cmd(0x4598f5c7e4c8) 0x2a, CmdSN 0x598 from world 1048870 to dev "naa.600a098038314b39335d554871744543" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0xc Medium Error, LBA: 17832200

2023-07-03T17:27:59.160Z cpu26:1048870)Fil6: 4072: 'linbtrfsboot': Fil6 file IO (<FD c19 r13>) : I/O error

2023-07-03T17:27:59.160Z cpu26:1048870)Fil6: 4038: ioCtx: 0x4598f597f8c0, world: 1048870, overallStatus: I/O error, token: 0x0, tokenStatus: N, txnToken: 0x0, txnTokenStatus: N, totalIOSize: 2078, maxIOLength: 137246

2023-07-03T17:27:59.160Z cpu26:1048870)Fil6: 4044: numAddrs: 0, numTbz: 0, numPBsToAlloc: 0, numBlocksToAlloc: 0 numTBZBlocks: 0, numPBsToUnmap: 0, numBlocksToUnmap: 0

2023-07-03T17:27:59.160Z cpu26:1048870)Fil6: 4051: opType: 2, ioFlags: 393552, allocFlags: 0, ioStateTxn: 0, ioStateNoTxn: 7, txnState: 2, numResolvedAddrMaps: 1, numTxnPendingAddrMaps: 0

2023-07-03T17:27:59.160Z cpu26:1048870)Fil6: 4056: wakeupOnIOComplete: 0, isIOIssued: 1, isIODone: 1, isRead: 0 isSync: 1, canBlock: 1, lastPartialBlock: 0

2023-07-03T17:27:59.160Z cpu26:1048870)BC: 414: write to vmware.log (f532 28 3 64a149c0 dca15134 524cad38 dc933362 34004c4 17 0 0 0 0 0) 2078 bytes failed: I/O error

2023-07-03T17:27:59.160Z cpu26:1048870)Log: 1577: Generating backtrace for 1048870: bcflushd

2023-07-03T17:27:59.160Z cpu26:1048870)Backtrace for current CPU #26, worldID=1048870, fp=0x100126

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931bad0:[0x420039ce8f55]LogBacktraceInt@vmkernel#nover+0xdd stack: 0x149a00, 0x420039c3e164, 0x45388d41f140, 0x453898e9f140, 0x45388931f140

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931baf0:[0x420039c3e163]BCFileIO@vmkernel#nover+0x1b4 stack: 0x45388931f140, 0xf8cda194, 0x6c2e657261776d76, 0x45388900676f, 0x45388d41f140

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931bd60:[0x420039c3eb5b]BCWriteBuffer@vmkernel#nover+0x1fc stack: 0x149a00, 0xffffffffffffffff, 0x453800000000, 0x25000, 0x8a563f000

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931bf70:[0x420039c3ecc7]BCFlushFile@vmkernel#nover+0x44 stack: 0x430707815f10, 0x41fff9d527e0, 0x430707815fd0, 0x420039c3eda7, 0x45388931f000

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931bfa0:[0x420039c3eda6]BCFlushAll@vmkernel#nover+0xc3 stack: 0x45388931f140, 0x0, 0x0, 0x420039c3edf9, 0x0

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931bfd0:[0x420039c3edf8]BCFlusherWorld@vmkernel#nover+0x9 stack: 0x0, 0x0, 0x0, 0x420039cc4a60, 0x0

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931bfe0:[0x420039fb1871]CpuSched_StartWorld@vmkernel#nover+0x86 stack: 0x0, 0x420039cc4a60, 0x0, 0x0, 0x0

2023-07-03T17:27:59.160Z cpu26:1048870)0x45388931c000:[0x420039cc4a5f]Debug_IsInitialized@vmkernel#nover+0xc stack: 0x0, 0x0, 0x0, 0x0, 0x0