Increasing the maximum I/O size in Linux

This post is really a quick note to myself to remind me how to bump up the maximum IO size on Linux. I have been benchmarking a bit lately and increasing the maximum size of an I/O request from 512kb to 1024kb looked like something worth doing. Especially since it’s also done in Exadata :)

So why would it matter? It matters most for Oracle DSS systems, actually. Why? Take ORION for example-although it’s using mindless I/O as explained by @flashdba and @kevinclosson, at least it gives me a quick way to test the size of a typical I/O request. Let me demonstrate:

[oracle@server1 ~]$ $ORACLE_HOME/bin/orion -run oltp -testname foo
...

So ORION is now busying itself producing numbers based on 8k reads. How do I know? Because I’m using collectl (https://martincarstenbach.wordpress.com/2011/08/05/an-introduction-to-collectl/) in another session:

[oracle@server1 ~]$ perl collectl.pl -sD --iosize --dskfilt sdc
waiting for 1 second sample...

# DISK STATISTICS (/sec)
#          <---------reads---------><---------writes---------><--------averages--------> Pct
#Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size  RWSize  QLen  Wait SvcTim Util
sdc          62464      0 7808    8       0      0    0    0       8     1     0      0   63
sdc          63280      0 7910    8       0      0    0    0       8     1     0      0   57
sdc          64048      0 8005    8       0      0    0    0       8     1     0      0   60
^COuch!

Look at the IO size: 8k just as you would expect. By the way ORION is slightly unrealistic in the OLTP case because it uses io_submit and io_getevents instead of pread but that’s a different story altogether for another blog post.

Now I’d like to run the DSS test. Firing off ORION again, this time with “-run dss” I observe the following:

$ perl collectl.pl -sD --iosize --dskfilt sdc
waiting for 1 second sample...

# DISK STATISTICS (/sec)
#          <---------reads---------><---------writes---------><--------averages--------> Pct
#Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size  RWSize  QLen  Wait SvcTim Util
sdc         1995776      0 3898  512       0      0    0    0     512     1     0      0   73
sdc         2089984      0 4082  512       0      0    0    0     512     1     0      0   77
sdc         1976320      0 3860  512       0      0    0    0     512     1     0      0   73

Aha, my IO size is 512k. That’s good, but not good enough. After what seemed like a very long time and digging through scheduler code on lxr.linux.no and discussing with friends I found out that the clue is in the SYSFS. Here you find lots of parameters determining how the kernel issues IO against a block device. You define the IO scheduler, the queue depth, sector size (those 4k sectors and Oracle!) and other parameters. Amongst these the following are of relevance to this post:

– max_sectors_kb
– max_hw_sectors_kb

max_sectors_kb defines the maximum I/O size, and it logically cannot exceed its maximum. I found 1MB the maximum for many spinning disks, my SATA SSD has more.

Now when I bump that value up to 1M I get larger IO sizes with the DSS benchmark:

$ perl collectl.pl -sD --iosize --dskfilt sdc
waiting for 1 second sample...

# DISK STATISTICS (/sec)
#          <---------reads---------><---------writes---------><--------averages--------> Pct
#Name       KBytes Merged  IOs Size  KBytes Merged  IOs Size  RWSize  QLen  Wait SvcTim Util
sdc         1927047      0 1882 1024       0      0    0    0    1024     1     0      0   51
sdc         1906688      0 1862 1024       0      0    0    0    1024     1     0      0   51

Thanks also to Frits Hoogland for helping me understand Linux IO better!