在 storageGRID 存储桶上进行摄取/检索期间出现间歇性错误 504 网关超时

最后更新
另存为PDF

Views:: 36

Visibility:: Public

Votes:: 0

Category:: storagegrid

Specialty:: sgrid

Last Updated:

适用于

StorageGRID OS 11.7.0

问题描述

从 storageGRID 存储桶中获取/检索对象时观察到间歇性错误：

Error: reading blob <>: fetching blob: received unexpected HTTP status: 504 Gateway Time-out

从 storageGRID Graphana 中，我们观察到 504 错误与网关节点上的 499 错误同时发生，这与从应用程序端观察到的问题相匹配：

Failed to pull docker image from bucket: Error timed out

已启用调试级别日志记录
从 StorageGRID 日志中，我们发现了通过网关节点从客户端到存储节点的请求上游过程中 SSL 握手超时的问题：

网关节点 - /var/local/log/nginx-gw/access.log

2024/04/15 06:40:08 [warn] 1327055#1327055: *580890363 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 23 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-01-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 06:40:13 [warn] 1327099#1327099: *580934305 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 31 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 08:19:56 [warn] 1327060#1327060: *581032520 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 24 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix=docker%2Fregistry%2Fv2%2Fblobs%2Fsha256%2F38%2F38c364755503d3fa4adc039af63e5108189b759cf6ad9d26f7219072c427db83%2Fdata HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=docker%2Fregistry%2Fv2%2Fblobs%2Fsha256%2F38%2F38c364755503d3fa4adc039af63e5108189b759cf6ad9d26f7219072c427db83%2Fdata, host: "<Gateway node>:10443"

2024/04/15 06:40:08 [error] 1327055#1327055: *580890363 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 08:34:50 [error] 1327060#1327060: *581032520 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"