[pnfs] poor read performance

[J. Bruce Fields]

On Tue, Oct 02, 2007 at 07:39:22PM -0700, dean hildebrand wrote:
> I'm new to the sessions code, but it seems that in xdr_read_pages
> there is a check which will call xdr_shrink_bufhead (which will call
> memmove)
> ...
>         /* Realign pages to current pointer position */
>         iov  = buf->head;
>         shift = iov->iov_len + (char *)iov->iov_base - (char *)xdr->p;
>         if (shift > 0)
>                 xdr_shrink_bufhead(buf, shift);
> ...
> 
> Placing some debug statements in xdr_read_pages, it seems that
> xdr_shrink_bufhead is only called when the sessions code is used,
> which might be the CPU hog.  With sessions in our system the shift is
> 44 on every read request.
> 
> Any ideas?  How could sessions affect the read buffer?  Has anyone
> seen a read performance problem?  Something to do with slots? (always
> a good guess)

I think it's in the function that encodes the xdr read that you'll find
an estimate of the offset into the reply that where the read data is
expected to be found (assuming it turns out to be a succesful read).
That estimate is what allows the client to arrange to have the read data
received directly into the page cache instead of having to first receive
it into a separate xdr buffer and then copy it all.

That calculation probably hasn't been updated to take into account
whatever extra new session op it is that the start of each read
operation, probably adding a word or two to the start of each read
reply....

--b.