[pnfs] poor read performance

[dean hildebrand]

Hi All,

We've noticed that performing a read using standard NFSv4 (or pNFS)
causes the CPU on the client to skyrocket.  On an older machine, the
read performance is then drastically reduced (reduced from 50 Mb/s to
5 Mb/s on one machine).

To look into the problem, we tried with the plain 2.6.18.3 kernel and
this problem did not occur.  Also, the problem does not appear using
nfsv3.  Write performance is fine for every test we ran.

The problem does appear when using the 4.1-sessions branch, so this is
a problem related to the sessions code (and not pNFS).

We profiled the kernel with sessions and noticed that the memmove
function seems to be hogging all the cpu.  Here is our output from
OProfile with sessions.
samples  %        image name               app name                 symbol name
4196413  94.2417  vmlinux                  vmlinux                  memmove
48413     1.0872  vmlinux                  vmlinux
__copy_to_user_ll
35828     0.8046  vmlinux                  vmlinux
skb_copy_bits
15305     0.3437  oprofiled                oprofiled                (no symbols)
11313     0.2541  vmlinux                  vmlinux                  kmap_atomic
6753      0.1517  tg3.ko                   tg3                      tg3_poll
...

With standard NFSv4 in 2.6.18 memmove is way down the list:
samples  %        image name               app name                 symbol name
41099    19.8331  vmlinux                  vmlinux
__copy_to_user_ll
35068    16.9228  vmlinux                  vmlinux
skb_copy_bits
6495      3.1343  tg3.ko                   tg3                      tg3_poll
5438      2.6242  vmlinux                  vmlinux                  tcp_v4_rcv
...

I'm new to the sessions code, but it seems that in xdr_read_pages
there is a check which will call xdr_shrink_bufhead (which will call
memmove)
...
        /* Realign pages to current pointer position */
        iov  = buf->head;
        shift = iov->iov_len + (char *)iov->iov_base - (char *)xdr->p;
        if (shift > 0)
                xdr_shrink_bufhead(buf, shift);
...

Placing some debug statements in xdr_read_pages, it seems that
xdr_shrink_bufhead is only called when the sessions code is used,
which might be the CPU hog.  With sessions in our system the shift is
44 on every read request.

Any ideas?  How could sessions affect the read buffer?  Has anyone
seen a read performance problem?  Something to do with slots? (always
a good guess)

Dean Hildebrand
Research Staff Member
IBM