iSCSI 吞吐量低于预期
适用场景
- Cisco Unified Computing System （ UCS ）
- iSCSI
- MPIO
问题描述
- 在应用程序中观察 ~100 MB/ 秒（ LVM ）到 200 （单个 LUN ） MB/ 秒的吞吐量，而不是预期的 ~1GB/ 秒吞吐量，并在将 LUN 迁移到 LVM （条带化）配置中的新存储系统后使用性能测试工具 FiO 验证是否发生与应用程序无关的情况。
- LVM
fio -thread -rw=readwrite -rwmixread=50 -bs=64k -size=20G -name=mytest --filename=/xxx/xxx/test1/file1
READ: bw=93.7MiB/s (98.3MB/s), 93.7MiB/s-93.7MiB/s (98.3MB/s-98.3MB/s), io=10.0GiB (10.7GB), run=109260-109260msec
WRITE: bw=93.7MiB/s (98.2MB/s), 93.7MiB/s-93.7MiB/s (98.2MB/s-98.2MB/s), io=9.00GiB (10.7GB), run=109260-109260msec- 单个 LUN
fio -thread-rw=readwrite -rwmixread=50 -bs=64k -size=20G -name=mytest --filename=/xxx/xxx/test2/file1
READ: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=10.0GiB (10.7GB), run=49574-49574msec
WRITE: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=9.00GiB (10.7GB), run=49574-49574msec- 受影响的配置（
multipath -ll）：
xxxxxxxxxxxxx dm-8 NETAPP ,LUN C-Modesize=400G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw|-+- policy='service-time 0' prio=50 status=active| |- 4:0:0:5 sdab 65:176 active ready running| |- 2:0:0:5 sdac 65:192 active ready running| |- 7:0:0:5 sdad 65:208 active ready running| `- 5:0:0:5 sdar 66:176 active ready running`-+- policy='service-time 0' prio=10 status=enabled|- 0:0:0:5 sdf 8:80 active ready running|- 1:0:0:5 sdl 8:176 active ready running|- 3:0:0:5 sdam 66:96 active ready running`- 6:0:0:5 sdav 66:240 active ready running- 跟踪时，以下 tcpdump 筛选器会捕获丢弃的数据：
- tcpdump ：
# tcpdump -i ibft0 -w [path]ibft0.pcap
- 选择性确认：
tcp.options.sack.count > 0
- 重复确认，（快速）重新传输，无序帧：
tcp.analysis.duplicate_ack || tcp.analysis.retransmission || tcp.analysis.fast_retransmission || tcp.analysis.out_of_order- 在 UCS 命令行界面（ nxos ）上，命令 show queuing interface ethernet x/xx （受影响的接口分别适用于相关 LUN ）指示入站丢弃，这表示 NIC 缓冲区耗尽：
ucs-sas-A(nxos)# sh queuing interface ethernet 1/14Ethernet1/14 queuing information:
[...]
RX Queuing
[...]
qos-group 4
q-size: 29760, HW MTU: 9216 (9216 configured)
drop-type: drop, xon: 0, xoff: 29760
Statistics:
Pkts received over the port : 16504320
Ucast pkts sent to the cross-bar : 16453589
Mcast pkts sent to the cross-bar : 0
Ucast pkts received from the cross-bar : 10259502
Pkts sent to the port : 10259502
Pkts discarded on ingress : 50731
Per-priority-pause status : Rx (Inactive), Tx (Inactive)
qos-group 5
q-size: 29760, HW MTU: 9216 (9216 configured)
drop-type: drop, xon: 0, xoff: 29760
Statistics:
Pkts received over the port : 50303345
Ucast pkts sent to the cross-bar : 49818733
Mcast pkts sent to the cross-bar : 0
Ucast pkts received from the cross-bar : 2162905
Pkts sent to the port : 2162905
Pkts discarded on ingress : 484613
Per-priority-pause status : Rx (Inactive), Tx (Inactive)