间歇性错误504 StorageGRID存储分段上的加载/检索期间网关超时

最后更新
另存为PDF

Views:: 3

Visibility:: Public

Votes:: 0

Category:: storagegrid<a>2009994925</a>

Specialty:: sgrid

Last Updated:

适用场景

StorageGRID OS 11.7.0

问题描述

从StorageGRID存储分段中读取/读取对象时出现间歇性错误：

Error: reading blob <>: fetching blob: received unexpected HTTP status: 504 Gateway Time-out

我们在StorageGRID Graps也观察到 504 了与 499 网关节点上的错误一致的错误、这些错误与从应用程序端观察到的问题描述匹配：

Failed to pull docker image from bucket: Error timed out

已启用调试级别日志记录
从StorageGRID日志中、我们发现在客户端通过网关节点向存储节点发出请求的上游期间、问题描述具有SSL握手超时：

网关节点- /var/local/log/nginx-gw/access.log

2024/04/15 06:40:08 [warn] 1327055#1327055: *580890363 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 23 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-01-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 06:40:13 [warn] 1327099#1327099: *580934305 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 31 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 08:19:56 [warn] 1327060#1327060: *581032520 [lua] sgbalancer.lua:564: on_balance_phase(): Connection to 47f9ba08-bc36-4be5-bfe3-cde14b40f83f failed with code 504 in worker 24 while connecting to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix=docker%2Fregistry%2Fv2%2Fblobs%2Fsha256%2F38%2F38c364755503d3fa4adc039af63e5108189b759cf6ad9d26f7219072c427db83%2Fdata HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=docker%2Fregistry%2Fv2%2Fblobs%2Fsha256%2F38%2F38c364755503d3fa4adc039af63e5108189b759cf6ad9d26f7219072c427db83%2Fdata, host: "<Gateway node>:10443"

2024/04/15 06:40:08 [error] 1327055#1327055: *580890363 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"

2024/04/15 08:34:50 [error] 1327060#1327060: *581032520 upstream timed out (110: Connection timed out) while SSL handshaking to upstream, client: <Client IP>, server: _, request: "GET /**-**-registry?list-type=2&max-keys=1&prefix= HTTP/1.1", upstream: https://<StorageGRID node>:18082/**-**-registry?list-type=2&max-keys=1&prefix=, host: "<Gateway node>:10443"