REST调用返回HTTP 502错误
适用场景
- ONTAP 9
- /API/REST的修补程序SnapMirror
问题描述
- 系统将对SnapMirror <relationship-id> 执行REST API调用
PATCH<relationship-id>、以尝试对关系执行quiesce。通过指定的return_timeout后，ONTAP将返回HTTP/502错误，而不是预期的HTTP/200或HTTP/202：
> curl -ik -s -u <username>:<password> -X PATCH "https://<cluster_mgmt_lif>/api/snapmirror/relationships/<relationship-id>?failover=false&force-failover=false&force=false&failback=false&return_timeout=120" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"state\": \"paused\"}"
HTTP/1.1 502 Proxy Error
Date: Wed, 03 Aug 2022 10:13:41 GMT
Server: Apache
X-Content-Type-Options: nosniff
Cache-Control: no-cache,no-store,must-revalidate
Content-Length: 341
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>502 Proxy Error</title>
</head><body>
<h1>Proxy Error</h1>
<p>The proxy server received an invalid
response from an upstream server.<br />
The proxy server could not handle the request<p>Reason: <strong>Error reading from remote server</strong></p></p>
</body></html>return_timeout 为120秒的10:11:41 执行的(这是ONTAP REST支持的最大值)apache_access.log显示HTTP/502错误，但在相关日志条目的上下文中，5min + return_timeout会延迟该错误：
[Wed Aug 03 10:18:38.554471 2022 +0000] "POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.0" 200 190360 1537002 - 0 - 127.0.0.1 name-mapping-get-iter
[Wed Aug 03 10:13:41.088515 2022 +0000] "PATCH /api/snapmirror/relationships/<relationship-id>?failover=false&force-failover=false&force=false&failback=false&return_timeout=120 HTTP/1.1"502 341 300034196 - 0 + 127.0.0.1 -
[Wed Aug 03 10:18:40.333797 2022 +0000] "POST /servlets/netapp.servlets.admin.XMLrequest_filer HTTP/1.0" 200 191798 1701289 - 0 - 127.0.0.1 name-mapping-get-iter
audit.log显示了在出现return_timeout和HTTP/502错误后长时间返回成功的请求：
Wed Aug 03 2022 10:13:41 +00:00 [kern_audit:info:2561] 8503e80000a75342 :: <cluster_name>:http :: <REST-client-IP-address>:53495 :: <cluster_name>:<user_name> :: PATCH /api/snapmirror/relationships/<relationship-id>?failover=false&force-failover=false&force=false&failback=false&return_timeout=120 : { "state": "paused"} :: Pending
Wed Aug 03 2022 10:13:41 +00:00 [kern_audit:info:3229] 8503e80000a75342 :: <cluster_name>:http :: <REST-client-IP-address>:53495 :: <cluster_name>:<user_name> :: PATCH /api/snapmirror/relationships/<relationship-id>?failover=false&force-failover=false&force=false&failback=false&return_timeout=120 HTTP/1.1 :: Error: 502 Proxy Error
Wed Aug 03 2022 11:14:39 +00:00 [kern_audit:info:2561] 8503e80000a75342 :: <cluster_name>:http :: <REST-client-IP-address>:53495 :: <cluster_name>:<user_name> :: PATCH /api/snapmirror/relationships/<relationship-id>?failover=false&force-failover=false&force=false&failback=false&return_timeout=120 : { "state": "paused"} :: Success:
apache_error.log在原始请求超时5分钟后、从REST API服务127.0.0.1:936报告超时：
[Wed Aug 03 10:18:41.122493 2022 +0000] [proxy_http:error] [pid 12297:tid 34379805440] (70007)The timeout specified has expired: [client <REST-client-IP-address>:53495] AH01102: error reading status line from remote server 127.0.0.1:936
[Wed Aug 03 10:18:41.122529 2022 +0000] [proxy:error] [pid 12297:tid 34379805440] [client <REST-client-IP-address>:53495] AH00898: Error reading from remote server returned by /api/snapmirror/relationships/<relationship-id>- mgwd.log 在
return_timeout之后显示ZSM错误，并且稍后成功返回snapmirror_relationship_rest：
Wed Aug 03 2022 10:13:41 +00:00 [kern_mgwd:info:2561] ZSM: parse error=Document is empty {na_zapi_get_elem_from_raw_xml:1009}
Wed Aug 03 2022 11:13:30 +00:00 [kern_mgwd:info:2561] snapmirror_relationship_rest.get_imp() returns: success (latency: 961.568391s) [<relationship-id>,<source_vserver>:<source_volume>,-,-,-,-,-,-,-,<destination_vserver>:<volume>,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,/api/snapmirror/relationships/<relationship-id>/,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-,-] (memory: -7188440 net, 7902568 max, 2199422456 allocated, 2206610896 freed, 14421787 allocations)