StorageGRID中存储分段的魔力委员会主管请求失败
适用场景
- NetApp StorageGRID
- Spark应用程序
- S3A魔术委员会
问题描述
- 使用Magic Committer的应用程序在使用存储分段一致性级别
read-after-new-write的多部分上传中遇到HEAD请求失败。 - 显示以下请求/响应的应用程序日志：
文件上传：
25/01/17 04:58:03 66 FILE UPLOAD COMPLETED; UPLOAD COMMIT WRITTEN TO part-XXXX-c000.snappy.parquet.pending
请求： 25/01/17 04:58:03 66 Executor task launch worker for task146.0 in stage 8.0 (TID 406) DEBUG S3AFileSystem: Recursive list of all entries under FileIngestion/ParquetJob/GDP_SPARK/2025-01-17T04:44:05.630014Z/data/__magic/job-XXX/tasks/attempt_X/__base/
响应：
25/01/17 04:58:04 (PAR-XXXX -c000.snappy.parquet和PARE-XXXX-c000.snappy.parquet.pending are not found is list response)
请求： 25/01/17 04:58:04 66 Executor task launch worker for task146.0 in stage 8.0 (TID 406) DEBUG S3AFileSystem: Recursive list of all entriesunder FileIngestion/ParquetJob/GDP_SPARK/2025-01-17T04:44:05.630014Z/data/__magic/job-/tasks/attempt_X/
响应： 25/01/17 04:58:04 (part-XXXX-c000.snappy.parquetand part-XXXX-c000.snappy.parquet.pending are found is LIST response as expected)
- StorageGRID bycast.log
bycast.log记录：
OBDR: Metadata request failed for unknown '<bucketname>/FileIngestion/ParquetJob/GDP_SPARK/2025-01-17T04:44:05.630014Z/data/__magic/job-XXXX/tasks/attempt_X/__base/part-XXXX-c000.snappy.parquet'