[osdl_staff_pdx] [nfsv4] Performance drop in iozone for reading large files

[Stephen Hemminger]

On Thu, 14 Sep 2006 23:53:03 -0700
Bryce Harrington <bryce at osdl.org> wrote:

> On Thu, Sep 14, 2006 at 05:13:53PM -0700, Andrew Morton wrote:
> > On Thu, 14 Sep 2006 16:58:05 -0700
> > Bryce Harrington <bryce at osdl.org> wrote:
> > 
> > > On Wed, Sep 06, 2006 at 10:59:48AM -0400, Dean Hildebrand wrote:
> > > > To prove/disprove the sever caching theory, here are a couple *echm* fun 
> > > > things you can try:
> > > > 1) While writing and reading the file, watch the memory on the server 
> > > > using 'top'.  You should see it rise up an amount roughly equal to the 
> > > > size of the file.  Once the file is deleted by iozone, the memory will 
> > > > drop back down again.
> > > > 2) After writing the file do not delete it (use -w I believe). Umount 
> > > > the server, reboot the server, then mount the server and read the file.  
> > > > This will read the file directly from disk.  Compare this performance to 
> > > > your original read performance (which used the server cache).
> > > 
> > > I think we've satisfied ourselves that this generally describes what
> > > we're seeing.  However, there is one other odd behavior we think may be
> > > of interest.  Compare the following two charts:
> > > 
> > > 2.6.17 (stock):
> > >     http://crucible.osdl.org/runs/1959/test_output/iozone.sys.log.png
> > > 2.6.18-rc1 (stock):
> > >     http://crucible.osdl.org/runs/1955/test_output/iozone.sys.log.png
> > > 
> > > Notice how the read performance in the second kernel has fallen below
> > > write performance.  We're seeing this "read performance worse than write
> > > performance" in the recently released CITI patch as well as in the git
> > > trees.  For instance, see the 'nfsv4' iozone test cases at
> > > http://crucible.osdl.org/runs/branches.html 
> > 
> > It appears that large linear reads of large, well-laid out files on ext3
> > have taken a ~20% hit post-2.6.17.  Badari is looking into it.
> > 
> > If your server-side's filesystem is ext3 I'd suggest that you mount it with
> > ext2 and retest.  If that fixes it, it's almost certainly the ext3
> > regression.
> 
> Oh, cool, we'll give this a try.  Thanks!
>  
> > Alternatively, you could alter ext3_get_blocks_handle() so that it only
> > ever does single-block mappings: I assume the way to do this is:
> > 
> > 	maxblocks = 1;
> > 
> > on entry to that function.


Sounds like a candidate for a 'git bisect' if it is that different.