[pnfs] poor read performance

[J. Bruce Fields]

On Tue, Oct 02, 2007 at 10:57:13PM -0400, J. Bruce Fields wrote:
> On Tue, Oct 02, 2007 at 07:39:22PM -0700, dean hildebrand wrote:
> > I'm new to the sessions code, but it seems that in xdr_read_pages
> > there is a check which will call xdr_shrink_bufhead (which will call
> > memmove)
> > ...
> >         /* Realign pages to current pointer position */
> >         iov  = buf->head;
> >         shift = iov->iov_len + (char *)iov->iov_base - (char *)xdr->p;
> >         if (shift > 0)
> >                 xdr_shrink_bufhead(buf, shift);
> > ...
> > 
> > Placing some debug statements in xdr_read_pages, it seems that
> > xdr_shrink_bufhead is only called when the sessions code is used,
> > which might be the CPU hog.  With sessions in our system the shift is
> > 44 on every read request.
> > 
> > Any ideas?  How could sessions affect the read buffer?  Has anyone
> > seen a read performance problem?  Something to do with slots? (always
> > a good guess)
> 
> I think it's in the function that encodes the xdr read that you'll find
> an estimate of the offset into the reply that where the read data is

More specifically: take a look at the call to xdr_inline_pages() from
nfs4_xdr_enc_read(), and the comment right before it.  Note some of the
other read-link functions (readdir, readlink, getacl, fs_locations) do
the same thing.

--b.

> expected to be found (assuming it turns out to be a succesful read).
> That estimate is what allows the client to arrange to have the read data
> received directly into the page cache instead of having to first receive
> it into a separate xdr buffer and then copy it all.
> 
> That calculation probably hasn't been updated to take into account
> whatever extra new session op it is that the start of each read
> operation, probably adding a word or two to the start of each read
> reply....
> 
> --b.
> _______________________________________________
> pNFS mailing list
> pNFS at linux-nfs.org
> http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs