iSCSI 吞吐量低于预期
适用场景
- Cisco Unified Computing System ( UCS )
- iSCSI
- MPIO
问题描述
- 在应用程序中观察 ~100 MB/ 秒( LVM )到 200 (单个 LUN ) MB/ 秒的吞吐量,而不是预期的 ~1GB/ 秒吞吐量,并在将 LUN 迁移到 LVM (条带化)配置中的新存储系统后使用性能测试工具 FiO 验证是否发生与应用程序无关的情况。
- LVM
fio -thread -rw=readwrite -rwmixread=50 -bs=64k -size=20G -name=mytest --filename=/xxx/xxx/test1/file1
READ: bw=93.7MiB/s (98.3MB/s), 93.7MiB/s-93.7MiB/s (98.3MB/s-98.3MB/s), io=10.0GiB (10.7GB), run=109260-109260msec
WRITE: bw=93.7MiB/s (98.2MB/s), 93.7MiB/s-93.7MiB/s (98.2MB/s-98.2MB/s), io=9.00GiB (10.7GB), run=109260-109260msec
- 单个 LUN
fio -thread-rw=readwrite -rwmixread=50 -bs=64k -size=20G -name=mytest --filename=/xxx/xxx/test2/file1
READ: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=10.0GiB (10.7GB), run=49574-49574msec
WRITE: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=9.00GiB (10.7GB), run=49574-49574msec
- 受影响的配置(
multipath -ll
):
xxxxxxxxxxxxx dm-8 NETAPP ,LUN C-Mode
size=400G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 4:0:0:5 sdab 65:176 active ready running
| |- 2:0:0:5 sdac 65:192 active ready running
| |- 7:0:0:5 sdad 65:208 active ready running
| `- 5:0:0:5 sdar 66:176 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 0:0:0:5 sdf 8:80 active ready running
|- 1:0:0:5 sdl 8:176 active ready running
|- 3:0:0:5 sdam 66:96 active ready running
`- 6:0:0:5 sdav 66:240 active ready running
- 跟踪时,以下 tcpdump 筛选器会捕获丢弃的数据:
- tcpdump :
# tcpdump -i ibft0 -w [path]ibft0.pcap
- 选择性确认:
tcp.options.sack.count > 0
- 重复确认,(快速)重新传输,无序帧:
tcp.analysis.duplicate_ack || tcp.analysis.retransmission || tcp.analysis.fast_retransmission || tcp.analysis.out_of_order
- 在 UCS 命令行界面( nxos )上,命令 show queuing interface ethernet x/xx (受影响的接口分别适用于相关 LUN )指示入站丢弃,这表示 NIC 缓冲区耗尽:
ucs-sas-A(nxos)# sh queuing interface ethernet 1/14
Ethernet1/14 queuing information:
[...]
RX Queuing
[...]
qos-group 4
q-size: 29760, HW MTU: 9216 (9216 configured)
drop-type: drop, xon: 0, xoff: 29760
Statistics:
Pkts received over the port : 16504320
Ucast pkts sent to the cross-bar : 16453589
Mcast pkts sent to the cross-bar : 0
Ucast pkts received from the cross-bar : 10259502
Pkts sent to the port : 10259502
Pkts discarded on ingress : 50731
Per-priority-pause status : Rx (Inactive), Tx (Inactive)
qos-group 5
q-size: 29760, HW MTU: 9216 (9216 configured)
drop-type: drop, xon: 0, xoff: 29760
Statistics:
Pkts received over the port : 50303345
Ucast pkts sent to the cross-bar : 49818733
Mcast pkts sent to the cross-bar : 0
Ucast pkts received from the cross-bar : 2162905
Pkts sent to the port : 2162905
Pkts discarded on ingress : 484613
Per-priority-pause status : Rx (Inactive), Tx (Inactive)