CVO 升级因错误而暂停:节点未处于"waiting for giveback state"状态
适用于
- Cloud Volumes ONTAP (CVO)
- 自动化无中断升级(ANDU)
问题
cluster image show-update-progress 中出现以下错误:AWSNAS08::> cluster image show-update-progress
Estimated Elapsed
Update Phase Status Duration Duration
-------------------- ----------------- --------------- ---------------
Pre-update checks completed 00:10:00 00:01:07
ONTAP updates paused-on-error 01:32:00 01:03:35
Details:
Node name Status Status Description
-------------------- ----------------- --------------------------------------
AWSNAS-01 waiting
AWSNAS-02 failed Error: Node "AWSNAS-02" is not in
"waiting for giveback" state.
Action: Use the "storage failover
show" to verify that node
"AWSNAS-02" is in one of the
following states: "waiting for
giveback" or "connected".
7 entries were displayed.
Status: Paused - An error occurred in "ONTAP updates" phase. The update cannot continue until the error has been resolved. Resolve all issues, then use the "cluster image resume-update" command to resume the update.
的输出 storage failover show :
AWSNAS08::> storage failover show
Takeover
Node Partner Possible State Description
-------------- -------------- -------- -------------------------------------
AWSNAS-01 AWSNAS-02 false In takeover
AWSNAS-02 AWSNAS-01 - Unknown
2 entries were displayed.
输出 node show :
AWSNAS::> node show
Node Health Eligibility Uptime Model Owner Location
--------- ------ ----------- ------------- ----------- -------- ---------------
AWSNAS-01 true true 231 days 22:22 CDvM200
AWSNAS-02 false true - - - -
Warning: Unable to list entries on node AWSNAS-02. RPC: Couldn't make connection [from mgwd on node "AWSNAS-01"(VSID:-1) to mgwd at 10.80.41.145]
2 entries were displayed.
在已关闭节点的云提供商串行控制台上发现磁盘预留冲突 :
Reservation conflict found on this node's disks! Local System ID: xxxxxxxxxx Press Ctrl-C for Maintenance menu to release disks. Boot argument NfsRdmaOk not specified. Default (set) is in effect. Boot argument NfsIpOk not specified. Default (set) is in effect. FWU 2nd trigger point FWU has no post firmware update action registered. sysctl_warn_reuse: can't re-use a leaf (sysvar.DETAIL_mask)! pnso provider init started. Notice : pnso card not detected. hwo: Node is using hardware provider : 1. cryptomod_fips: Executing Crypto FIPS Self Tests. cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CCM' passed. cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed. cryptomod_fips: Crypto FIPS self-test: 'KDF' passed. cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-XTS 128, AES-XTS 256' passed. cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed. Waiting for reservations to clear
原因
Thu Jun 29 06:58:06 +0000 [AWSNAS-01: upgrademgr: upgrademgr.update.started:info]: An automated update of the cluster has been started. – Upgrade startedThu Jun 29 07:02:23 +0000 [AWSNAS-01: cf_slowTimeout: cf.fsm.nfo.shtdwnReqIC:debug]: A node sent a shutdown request over interconnect.
Thu Jun 29 07:02:48 +0000 [AWSNAS-01: cf_main: cf.fsm.nfo.partnerShutdown:notice]: Negotiated failover: partner has shutdown
Thu Jun 29 07:04:28 +0000 [AWSNAS-01: cfdisk_config: cf.diskinventory.sendFailed:debug]: params: {'errorCode': '0', 'reason': 'HA Interconnect down'}
互连状态显示的输出:
AWSNAS08::*> interconnect status show
(system ha interconnect status show)
Node: AWSNAS-01
Link Status: up
IC RDMA Connection: down
Node: AWSNAS-02
Link Status: up
IC RDMA Connection: down
2 entries were displayed.
请向您的云提供商提交工单,以了解HA互连中断的原因。
解决方案
合作伙伴备注
追加信息
内部参考
首先尝试以下每个解决方法,并检查是否可以使用 cluster image resume-update 恢复升级:
- 从云提供商控制台启动和停止关闭节点,以查看是否有任何更改。
- 如果关闭实例并重新打开电源无法解决问题,且实例再次卡在
Waiting for reservations to clear,请从其伙伴节点执行以下操作:
::> node run -node local
> priv set advanced
*> cf giveback -f
*> exit
3.使用以下命令切换上行节点中的 IC 链接,并检查 ic 状态:
AWSNAS::> run -node AWSNAS-01
Type 'exit' or 'Ctrl-D' to return to the CLI
AWSNAS-01>
AWSNAS-01> priv set diag
Warning: These diagnostic commands are for use by NetApp
personnel only.
AWSNAS-01*> ic status
Link : up
IC RDMA connection : down
AWSNAS-01*> ic link off 0
Link 0 is now configured down.
AWSNAS-01*> ic link on 0
Link 0 is now configured up.
如果上述解决方法无效。重新启动 UPNODE 可解决此问题,但在此期间两个节点均关闭,将无法提供数据。请先与客户确认其是否可以接受停机时间。
您可以从云提供商控制台停止和启动UPNODE ,节点启动后,升级将自动恢复。您可以继续监控升级 cluster image show-update-progress 直到完成。