[nfsv4] Performance drop in iozone for reading large files

[Bryce Harrington]

On Thu, Sep 14, 2006 at 05:13:53PM -0700, Andrew Morton wrote:
> On Thu, 14 Sep 2006 16:58:05 -0700
> Bryce Harrington <bryce at osdl.org> wrote:
> 
> > On Wed, Sep 06, 2006 at 10:59:48AM -0400, Dean Hildebrand wrote:
> > > To prove/disprove the sever caching theory, here are a couple *echm* fun 
> > > things you can try:
> > > 1) While writing and reading the file, watch the memory on the server 
> > > using 'top'.  You should see it rise up an amount roughly equal to the 
> > > size of the file.  Once the file is deleted by iozone, the memory will 
> > > drop back down again.
> > > 2) After writing the file do not delete it (use -w I believe). Umount 
> > > the server, reboot the server, then mount the server and read the file.  
> > > This will read the file directly from disk.  Compare this performance to 
> > > your original read performance (which used the server cache).
> > 
> > I think we've satisfied ourselves that this generally describes what
> > we're seeing.  However, there is one other odd behavior we think may be
> > of interest.  Compare the following two charts:
> > 
> > 2.6.17 (stock):
> >     http://crucible.osdl.org/runs/1959/test_output/iozone.sys.log.png
> > 2.6.18-rc1 (stock):
> >     http://crucible.osdl.org/runs/1955/test_output/iozone.sys.log.png
> > 
> > Notice how the read performance in the second kernel has fallen below
> > write performance.  We're seeing this "read performance worse than write
> > performance" in the recently released CITI patch as well as in the git
> > trees.  For instance, see the 'nfsv4' iozone test cases at
> > http://crucible.osdl.org/runs/branches.html 
> 
> It appears that large linear reads of large, well-laid out files on ext3
> have taken a ~20% hit post-2.6.17.  Badari is looking into it.
> 
> If your server-side's filesystem is ext3 I'd suggest that you mount it with
> ext2 and retest.  If that fixes it, it's almost certainly the ext3
> regression.

Oh, cool, we'll give this a try.  Thanks!
 
> Alternatively, you could alter ext3_get_blocks_handle() so that it only
> ever does single-block mappings: I assume the way to do this is:
> 
> 	maxblocks = 1;
> 
> on entry to that function.

Bryce