zfs sawtooth performance

[yashjain22]

System information

Type	Version/Name
Distribution Name	Ubuntu Linux
Distribution Version	20.04
Linux Kernel	5.4.0-51-generic
Architecture	x86
ZFS Version	zfs-0.8.3-1ubuntu12.4
SPL Version	zfs-0.8.3-1ubuntu12.4

System Configuration	Value
Ram	128GB
CPU	Intel Xeon Silver 4208 CPU @ 2.10 GHz
No of Cores	16
Disk Type	SSD

Describe the problem you're observing

ZVOL (pool containing one physical disk)

There is a very heavy fluctuations in IOPS during the elapsed time of 300 seconds

IOPS vs Time

ZVOL (pool containing two physical disk)

IOPS vs Time

ZVOL (pool containing four physical disk)

IOPS vs Time

Raw SSD

Much less fluctuation in IOPS during the elapsed time of 300 seconds

IOPS vs Time

Describe how to reproduce the problem

1. Zpool was created with one phsical disk (SSD)
   zpool create -o ashift=12 tank <device1>
2. Zvol was created using this command
   zfs create -o dedup=on -o compression=lz4 -o sync=always -o volblocksize=128K -s -V 300G tank/vol1
3. Primarycache was set to metadata on pool
   zfs set primarycache=metadata tank
4. Install vdbench
5. Fill the disk using this synthetic workload
   dedupratio=4 dedupunit=4k compratio=4 messagescan=no sd=default,openflags=directio,align=4k sd=sd2,lun=/dev/zd0 wd=wd_default,sd=* wd=fill,sd=sd*,xfersize=(4k,3,8k,8,16k,11,32k,12,64k,19,128k,21,256k,26),rdpct=0,seekpct=eof rd=default rd=fill,wd=fill,iorate=max,interval=30,elapsed=9000,forthreads=16
6. after first workload is finished , run the second workload and record performance
   dedupratio=4 dedupunit=4k compratio=4 sd=default,openflags=directio,align=4k sd=sd1,lun=/dev/zd0 wd=wd_vsi_read,rdpct=100,xfersize=(4k,25.68,8k,26.31,16k,6.4,32k,7.52,60k,10.52,128k,9.82,252k,7.31,504k,6.19,984k,0.23,1032k,0.02),seekpct=80,range=(0,100),sd=sd* wd=wd_vsi_write,rdpct=0,xfersize=(4k,59.62,8k,14.81,16k,8.97,32k,6.13,60k,5.35,100k,1.63,212k,1.39,432k,1.95,920k,0.14,1032k,0.01),seekpct=80,range=(50,100),sd=sd* rd=rd_vsi_ramp,wd=wd_vsi*,forrdpct=57.40,iorate=max,warmup=60,interval=1,elapsed=300,forthreads=8

Include any warning/errors/backtraces from the system logs

[bghira]

128k volblocksize? dear god, that might result in a bit of RMW overhead.

[yashjain22]

128k volblocksize? dear god, that might result in a bit of RMW overhead.

But when i start increasing disks in the pool the graph sorts of smooths out and fluctuation reduces

[shodanshok]

ZFS commits writes to the main pool each 5 seconds by default, so you probably are seeing the reduced IO performance due to writes hitting the disks. As I would not define this an issue, and I suggest closing this ticket and writing on the mailing list. Thanks.

[yashjain22]

ZFS commits writes to the main pool each 5 seconds by default, so you probably are seeing the reduced IO performance due to writes hitting the disks. As I would not define this an issue, and I suggest closing this ticket and writing on the mailing list. Thanks.

Thanks for replying, I have updated the issue with two and four physical disks in that case fluctuations are much less so is the behaviour still explainable by txg_delay ?

[shodanshok]

I would say so: after all, your pool now has 2-4x the write bandwidth of the single-vdev one. This means any flush finishes much faster than in the first case.

[yashjain22]

When i prevented metaslabs from unloading, there was no sawtooth behaviour. So I wonder is metaslabs loading/unloading causing this bottleneck ?

[shodanshok]

Yes, metadata trashing can be a co-factor in periodically slowing down your pool.

[yashjain22]

Okay I'll try to track the metaslab load/unload activity