[nfsv4] [nfsv4] Performance drop in iozone for reading large files

[Bryce Harrington]

On Fri, Sep 15, 2006 at 09:01:19AM -0700, Badari Pulavarty wrote:
> On Thu, 2006-09-14 at 23:53 -0700, Bryce Harrington wrote:
> > On Thu, Sep 14, 2006 at 05:13:53PM -0700, Andrew Morton wrote:
> > > > I think we've satisfied ourselves that this generally describes what
> > > > we're seeing.  However, there is one other odd behavior we think may be
> > > > of interest.  Compare the following two charts:
> > > > 
> > > It appears that large linear reads of large, well-laid out files on ext3
> > > have taken a ~20% hit post-2.6.17.  Badari is looking into it.
> > > 
> > > If your server-side's filesystem is ext3 I'd suggest that you mount it with
> > > ext2 and retest.  If that fixes it, it's almost certainly the ext3
> > > regression.
> > 
> > Oh, cool, we'll give this a try.  Thanks!
> 
> Can you try with following ext3 patch ?

Thanks, we've tried again with this patch, but got the same result:

2.6.18-rc6+patch:
    http://crucible.osdl.org/runs/2077/test_output/iozone.sys.log.png

We had also wondered if it was an ext3 issue, so tested against xfs and
reiser (see links below), but saw the same effect.

Notice that re-write behavior is affected but not write.  We would have
expected either write and re-write to be unaffected, or for read,
re-read, write, and re-write to all be affected the same way.  Could
this be a clue?

Thanks,
Bryce



------------------------------------------------------------------------

Hey Bryce,

Our file size max is 1 gig, so possibly not large enough for the issue Badari's patch addresses?  Or maybe since our tests are across NFS mounts.

The run with Badari's ext3 patch has completed the iozone sys test:
2.6.18-rc6+patch: http://crucible.osdl.org/runs/2077/test_output/iozone.sys.log.png

Stock runs for comparison:
2.6.18-rc1: http://crucible.osdl.org/runs/1959/test_output/iozone.sys.log.png
2.6.17: http://crucible.osdl.org/runs/1955/test_output/iozone.sys.log.png

Previous 2.6.18-rc6 stock runs on ext3, xfs, and reiserfs (xeons w/ scsi):
ext3: http://crucible.osdl.org/runs/1947/test_output/iozone.sys.log.png
xfs: http://crucible.osdl.org/runs/1948/test_output/iozone.sys.log.png
reiserfs: http://crucible.osdl.org/runs/1949/test_output/iozone.sys.log.png


Summary of our problem:
NFS Performance drop on iozone files larger than server memory size for 2.6.18-rc* compared to 2.6.17.  Read, re-read, and re-write all seem to be affected.  Only write performance is in-line with 2.6.17.  Re-write performance change is actually more obvious than the read change, since like Dean mentioned, read performance drops quite a bit due to server-side cache anyways.  But it shouldn't drop below write performance like it does on 2.6.18-rc*.

Since it occurs when server-side memory is exhausted, we've been restricting the suts to 256 meg for the iozone runs.  It is definitely server-side, since 2.6.18-rc1 server with 2.6.17 client has the problem, while 2.6.17 server with 2.6.18-rc1 client does not.

Samba/CIFS does not experience the problem when reaching mem size:
2.6.17 (stock): http://crucible.osdl.org/runs/1634/test_output/iozone.smb.log.png
2.6.18-rc4 (citi): http://crucible.osdl.org/runs/1633/test_output/iozone.smb.log.png


-- 
Jason Neighbors
x1939