Performance drop in iozone for reading large files

[Bryce Harrington]

On Wed, Sep 06, 2006 at 10:59:48AM -0400, Dean Hildebrand wrote:
> To prove/disprove the sever caching theory, here are a couple *echm* fun 
> things you can try:
> 1) While writing and reading the file, watch the memory on the server 
> using 'top'.  You should see it rise up an amount roughly equal to the 
> size of the file.  Once the file is deleted by iozone, the memory will 
> drop back down again.
> 2) After writing the file do not delete it (use -w I believe). Umount 
> the server, reboot the server, then mount the server and read the file.  
> This will read the file directly from disk.  Compare this performance to 
> your original read performance (which used the server cache).

I think we've satisfied ourselves that this generally describes what
we're seeing.  However, there is one other odd behavior we think may be
of interest.  Compare the following two charts:

2.6.17 (stock):
    http://crucible.osdl.org/runs/1959/test_output/iozone.sys.log.png
2.6.18-rc1 (stock):
    http://crucible.osdl.org/runs/1955/test_output/iozone.sys.log.png

Notice how the read performance in the second kernel has fallen below
write performance.  We're seeing this "read performance worse than write
performance" in the recently released CITI patch as well as in the git
trees.  For instance, see the 'nfsv4' iozone test cases at
http://crucible.osdl.org/runs/branches.html 

The last CITI kernel patchset prior to 2.6.18-rc1 does not appear to
have this issue:

linux-2.6.17-CITI_NFS4_ALL-1.diff:
    http://crucible.osdl.org/runs/1953/test_output/iozone.sys.log.png

It looks like it may be an issue in nfsd in general, and since it
appears to have happened between CITI's last patch and the next official
kernel release, we're wondering if perhaps it's due to some change
elsewhere in the kernel?  Does this appear to be a regression, or is it
an expected situation?

Thanks,
Bryce

----- Forwarded message from Jason Neighbors <jasonn at osdl.org> -----

Date: Fri, 8 Sep 2006 13:49:34 -0700
From: Jason Neighbors <jasonn at osdl.org>
To: bryce at osdl.org
Subject: 2.6.18 nfs performance

Hey,

So read, re-read, and re-write all seem to be affected by this.  Only write performance is in-line with 2.6.17.  Re-write performance change is actually more obvious than the read change, since like Dean mentioned, read performance drops quite a bit due to server-side cache anyways.  But it shouldn't drop below write performance like it does on 2.6.18-rc*.

We've been restricting the suts to 256 meg for the iozone runs.

The change between 2.6.17 and 2.6.18 can be seen by just viewing the iozone graphs:
2.6.16 (stock): http://crucible.osdl.org/runs/1635/test_output/iozone.sys.log.png
2.6.17 (stock): http://crucible.osdl.org/runs/1634/test_output/iozone.sys.log.png
2.6.18-rc4 (citi): http://crucible.osdl.org/runs/1633/test_output/iozone.sys.log.png

The change actually occurred at 2.6.18-rc1:
2.6.17 (stock): http://crucible.osdl.org/runs/1959/test_output/iozone.sys.log.png
2.6.18-rc1 (stock): http://crucible.osdl.org/runs/1955/test_output/iozone.sys.log.png

It is definitely server-side, since 2.6.18-rc1 server with 2.6.17 client has the problem, while 2.6.17 server with 2.6.18-rc1 client does not.

Samba/CIFS does not experience the same issue when reaching mem size:
2.6.17 (stock): http://crucible.osdl.org/runs/1634/test_output/iozone.smb.log.png
2.6.18-rc4 (citi): http://crucible.osdl.org/runs/1633/test_output/iozone.smb.log.png


-- 
Jason Neighbors
x1939

----- End forwarded message -----