S3/Swift 请求返回 ServiceUnavailable 失败以及节点停用

StorageGRID OS 11.6

Bycast 日志指示请求失败的原因是 Cassandra TimeoutException：
- HTTP Status Code=503, ErrorMsg=ServiceUnavailable, ErrorType=Client, CustomErrorMessage={<none>}, Details={<none>}
- OBDI: checkForPreExistingObject Cassandra TimeoutException (Failed to execute cql at consistency TWO: SELECT event_time, event, last_access_time, object_lock_mode, object_lock_retain_until_time, object_lock_legal_hold, user_metadata, writetime(user_metadata), content_type, writetime(content_type), restore_start_time, restore_expiry_time, retier_time, object_partially_tiered FROM storagegrid.object_by_uuid WHERE uuid = 5595C096-928D-4CAF-B8D8-E03A4865304F - Cassandra Driver Error(Read timeout):'Operation timed out - received only 14 responses.' Detailed Info:[consistency: ALL, responses_received: 14, responses_required: 15, data_present: 1])
Prometheus 数据表明

正在退役的特定节点的 CPU 使用率非常高。
sum by (instance) (sum by (instance, mode) (irate(node_cpu_seconds_total{instance=~"st.*",mode!="idle"}[5m])) / count by (instance, mode)(node_cpu_seconds_total{instance=~"st.*",mode!="idle"}))
注意： st是所有存储节点的通用首字母。
此特定节点的iowait在退役时会增加 5 倍（10% 至 50%），这意味着磁盘系统是瓶颈。
sum by (mode)(irate(node_cpu_seconds_total{instance="issued storage node name",mode!~'idle|guest|nice'}[5m])) * 100 / count by (mode)(node_cpu_seconds_total{instance="issued storage node name",mode!~'idle|guest|nice'})
此特定节点的所有磁盘的使用率几乎为 100%。
irate(node_disk_io_time_seconds_total{instance="issued storage node name",device=~'^sd.*'}[5m])*100

ASUP ->　STATE-CAPTURE-DATA
 Executing ionShow(99,0,0,0,0,0,0,0,0,0) on controller A:

错误节点：

-> chall 3

Target Read/Write Completions

.Channel :.................R E A D S................:...............W R I T E S................:

Ch H/D : #Success ByteXfered ART(uSec) MRT(uSec) : #Success ByteXfered ART(uSec) MRT(uSec) :#Errs

---- --- :---------- ---------- --------- --------- :---------- ---------- --------- --------- :-----

2 Hst : 51070465 3050503068160 23246 1869666 : 24067972 379745803264 45470 13645260 : 0

3 Hst : 50889777 3049366095360 23310 1760814 : 24248943 380225977344 45183 13645220 : 0

Initiator Read/Write Completions

.Channel :.................R E A D S................:...............W R I T E S................:

Ch H/D : #Success ByteXfered ART(uSec) MRT(uSec) : #Success ByteXfered ART(uSec) MRT(uSec) :#Errs

---- --- :---------- ---------- --------- --------- :---------- ---------- --------- --------- :-----

0 Drv : 256171408 35181547092992 17239 852896 : 82234342 1336298067456 2512 286906 : 0

4 Drv : 288 294912 4258 4241 : 0 0 0 0 : 0

Seconds since statistics cleared: 86411

良好节点：

-> chall 3

Target Read/Write Completions

.Channel :.................R E A D S................:...............W R I T E S................:

Ch H/D : #Success ByteXfered ART(uSec) MRT(uSec) : #Success ByteXfered ART(uSec) MRT(uSec) :#Errs

---- --- :---------- ---------- --------- --------- :---------- ---------- --------- --------- :-----

2 Hst : 27647780 2876604737536 5274 829929 : 11826653 237424963584 131 511517 : 0

3 Hst : 27509975 2877446842368 5303 826519 : 12073420 238340426240 131 620620 : 0

Initiator Read/Write Completions

.Channel :.................R E A D S................:...............W R I T E S................:

Ch H/D : #Success ByteXfered ART(uSec) MRT(uSec) : #Success ByteXfered ART(uSec) MRT(uSec) :#Errs

---- --- :---------- ---------- --------- --------- :---------- ---------- --------- --------- :-----

0 Drv : 136207478 28042508481024 3965 325577 : 7641267 528941565952 4254 45393 : 0

4 Drv : 288 294912 4301 4219 : 0 0 0 0 : 0

Seconds since statistics cleared: 86411

注意：