跳转到主内容

CVO 升级因错误而暂停:节点未处于"waiting for giveback state"状态

Views:
75
Visibility:
Public
Votes:
0
Category:
netapp_console
Specialty:
ds_cvo
Last Updated:

适用于

  • Cloud Volumes ONTAP (CVO)
  • 自动化无中断升级(ANDU)

问题

在 CVO 升级期间,接管节点已关闭,升级暂停,cluster image show-update-progress 中出现以下错误:

AWSNAS08::> cluster image show-update-progress

                       Estimated      Elapsed
Update Phase      Status           Duration     Duration
-------------------- ----------------- --------------- ---------------
Pre-update checks   completed         00:10:00     00:01:07
ONTAP updates     paused-on-error      01:32:00     01:03:35

Details:

Node name       Status       Status Description
-------------------- ----------------- --------------------------------------
AWSNAS-01      waiting
AWSNAS-02      failed        Error: Node "AWSNAS-02" is not in
                    "waiting for giveback" state.
                    Action: Use the "storage failover
                    show" to verify that node
                    "AWSNAS-02" is in one of the
                    following states: "waiting for
                    giveback" or "connected".
7 entries were displayed.

Status: Paused - An error occurred in "ONTAP updates" phase. The update cannot continue until the error has been resolved. Resolve all issues, then use the "cluster image resume-update" command to resume the update.

的输出 storage failover show :

AWSNAS08::> storage failover show
                Takeover
Node       Partner     Possible State Description
-------------- -------------- -------- -------------------------------------
AWSNAS-01   AWSNAS-02   false     In takeover
AWSNAS-02   AWSNAS-01   -       Unknown
2 entries were displayed.

输出   node show :

AWSNAS::> node show
Node    Health Eligibility Uptime     Model     Owner   Location
--------- ------ ----------- ------------- ----------- -------- ---------------
AWSNAS-01 true true      231 days 22:22 CDvM200
AWSNAS-02 false true           - -       -     -

Warning: Unable to list entries on node AWSNAS-02. RPC: Couldn't make connection [from mgwd on node "AWSNAS-01"(VSID:-1) to mgwd at 10.80.41.145]
2 entries were displayed.

在已关闭节点的云提供商串行控制台上发现磁盘预留冲突

Reservation conflict found on this node's disks! Local System ID: xxxxxxxxxx Press Ctrl-C for Maintenance menu to release disks. Boot argument NfsRdmaOk not specified. Default (set) is in effect. Boot argument NfsIpOk not specified. Default (set) is in effect. FWU 2nd trigger point FWU has no post firmware update action registered. sysctl_warn_reuse: can't re-use a leaf (sysvar.DETAIL_mask)! pnso provider init started. Notice : pnso card not detected. hwo: Node is using hardware provider : 1. cryptomod_fips: Executing Crypto FIPS Self Tests. cryptomod_fips: Crypto FIPS self-test: 'CPU COMPATIBILITY' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 ECB, AES-256 ECB' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CBC, AES-256 CBC' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 GCM, AES-256 GCM' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-128 CCM' passed. cryptomod_fips: Crypto FIPS self-test: 'CTR_DRBG' passed. cryptomod_fips: Crypto FIPS self-test: 'KDF' passed. cryptomod_fips: Crypto FIPS self-test: 'SHA1, SHA256, SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'HMAC-SHA1, HMAC-SHA256, HMAC-SHA512' passed. cryptomod_fips: Crypto FIPS self-test: 'PBKDF2' passed. cryptomod_fips: Crypto FIPS self-test: 'AES-XTS 128, AES-XTS 256' passed. cryptomod_fips: Crypto FIPS self-test: 'Self-integrity' passed. Waiting for reservations to clear

原因

在 CVO 自动无中断升级 (ANDU)  期间,HA 互连断开,节点关闭。
 
EMS 日志:
 

Thu Jun 29 06:58:06 +0000 [AWSNAS-01: upgrademgr: upgrademgr.update.started:info]: An automated update of the cluster has been started. – Upgrade started​​​​​​Thu Jun 29 07:02:23 +0000 [AWSNAS-01: cf_slowTimeout: cf.fsm.nfo.shtdwnReqIC:debug]: A node sent a shutdown request over interconnect.
Thu Jun 29 07:02:48 +0000 [AWSNAS-01: cf_main: cf.fsm.nfo.partnerShutdown:notice]: Negotiated failover: partner has shutdown
Thu Jun 29 07:04:28 +0000 [AWSNAS-01: cfdisk_config: cf.diskinventory.sendFailed:debug]: params: {'errorCode': '0', 'reason': 'HA Interconnect down'}

互连状态显示的输出:

AWSNAS08::*> interconnect status show
  (system ha interconnect status show)
            Node: AWSNAS-01
         Link Status: up
     IC RDMA Connection: down
            Node: AWSNAS-02
         Link Status: up
     IC RDMA Connection: down
2 entries were displayed.

请向您的云提供商提交工单,以了解HA互连中断的原因。

解决方案

请联系 NetApp 技术支持 或登录 NetApp 支持站点 以创建案例。请参阅本文以获得进一步帮助。

合作伙伴备注

合作伙伴备注文本
 

追加信息

追加信息文本

内部参考

首先尝试以下每个解决方法,并检查是否可以使用 cluster image resume-update 恢复升级:

  1. 从云提供商控制台启动和停止关闭节点,以查看是否有任何更改。
  2. 如果关闭实例并重新打开电源无法解决问题,且实例再次卡在 Waiting for reservations to clear,请从其伙伴节点执行以下操作:

      ::> node run -node local
     > priv set advanced
      *> cf giveback -f
      *> exit

3.使用以下命令切换上行节点中的 IC 链接,并检查 ic 状态:

AWSNAS::> run -node AWSNAS-01
Type 'exit' or 'Ctrl-D' to return to the CLI
AWSNAS-01>
AWSNAS-01> priv set diag
Warning: These diagnostic commands are for use by NetApp
     personnel only.
AWSNAS-01*> ic status
     Link : up
     IC RDMA connection : down
AWSNAS-01*> ic link off 0
Link 0 is now configured down.
AWSNAS-01*> ic link on 0
Link 0 is now configured up.

如果上述解决方法无效。重新启动 UPNODE 可解决此问题,但在此期间两个节点均关闭,将无法提供数据。请先与客户确认其是否可以接受停机时间。

您可以从云提供商控制台停止和启动UPNODE ,节点启动后,升级将自动恢复。您可以继续监控升级 cluster image show-update-progress  直到完成。

Sign in to view the entire content of this KB article.

New to NetApp?

Learn more about our award-winning Support

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.