nfsd4_stateowners memory leak

Mon Aug 24 21:09:35 EDT 2009

Hello All,

I know nfs developers probably have enough to deal with without going
over problems which have already been resolved, so hearing about old
bugs must be pretty tedious, but, we have a couple of servers running
latest versions of centos (redhat enterprise linux server clone) ie
version 5.3, and one of them seems to have a nasty memory leak, of
nfsd4_stateowners (and stateids to smaller extent)

We're running Centos kernels even though they are not on the bleeding
edge, because stability is important to us and going to brand-new
kernels to avoid old bugs - well, it risks even worse consequences of
new bugs that havent been ironed out. These centos kernels are patched
with many nfs patches and are supposed to work, noone else seems to have
any problems with them... but it seems they dont work for us.. and so I
would really appreciate any advice on how to proceed to resolve this
issue without having to move to brand new/untested kernels.

The problem persists on PAE (32bit, large mem) kernels from latest
centos 5.3 updated kernel 2.6.128.2.1 right back to base centos 5.2
kernel 2.6.18-92

Basically, the problem is that slabtop shows this sort of thing, after
server has been running a while:
825422 825422 100%    0.07K  15574       53     62296 K nfsd4_stateids
825154 825154 100%    0.34K  75014       11    300056 K
nfsd4_stateowners 
300 MB... and it just continues to grow. after 7-8 days the stateowners
have bloated the slab up to the point where lowmem is exhausted and
server will lockup, hard.

eg, after some time, when we have most of our clients exit, this might
cause 100 MB or more to be freed from slab on server, but it will still
leave a lot of "leaked", unfreed memory behind, eg:

[root at nas1 ~]# date ; cat /proc/meminfo | grep -i slab
Mon Aug 24 14:21:35 CST 2009
Slab:           445836 kB

then if I restart nfs it instantly frees up another 380 MB:
[root at nas1 ~]# date ; cat /proc/meminfo | grep -i slab
Mon Aug 24 14:22:07 CST 2009
Slab:            66208 kB

So, restarting nfs on server does free up all this slab memory and will
allow server to continue.

The problem looks similar to that described in:
http://linux-nfs.org/pipermail/nfsv4/2007-November/007124.html
this was related to server not incrementing seqid, when client does - on
failed lock operation, whuch was then causing memory to never be freed
on server. 
"> The memory "leak" occurs when the client tries to close the file.
The
> nfs request to close the file fails because the nfs server has not
> incremented its sequence counter.  This results in the NFS server not
> processing the NFS request to close the file, and not releasing the
> memory from the SLAB until the NFS server is shut down."

There was a patch proposed:

> Cc: Benny Halevy <bhalevy at panasas.com
<http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4> >
> Cc: Steven Wilton <steven.wilton at team.eftel.com.au
<http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4> >
> Cc: Trond Myklebust <trond.myklebust at fys.uio.no
<http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4> >
> Signed-off-by: J. Bruce Fields <bfields at citi.umich.edu
<http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4> >
> ---
>  fs/nfsd/nfs4state.c |    7 +++----
>  1 files changed, 3 insertions(+), 4 deletions(-)
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index 102c95b..b36f912 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -2094,8 +2094,10 @@ nfs4_preprocess_seqid_op(struct svc_fh 
> *current_fh, u32 seqid, stateid_t *statei
>  		goto check_replay;
>  	}
>  
> +	*stpp = stp;
> +	*sopp = sop = stp->st_stateowner;
> +
>  	if (lock) {
> -		struct nfs4_stateowner *sop = stp->st_stateowner;
>  		clientid_t *lockclid = &lock->v.new.clientid;
>  		struct nfs4_client *clp = sop->so_client;
>  		int lkflg = 0;
> @@ -2125,9 +2127,6 @@ nfs4_preprocess_seqid_op(struct svc_fh 
> *current_fh, u32 seqid, stateid_t *statei
>  		return nfserr_bad_stateid;
>  	}
>  
> -	*stpp = stp;
> -	*sopp = sop = stp->st_stateowner;
> -
>  	/*
>  	*  We now validate the seqid and stateid generation numbers.
>  	*  For the moment, we ignore the possibility of 
> -- 
> 1.5.3.5.561.g140d

I have looked in the latest kernel source and it looks like nfs4state.c
has been rewritten in such a way that the above patch no longer applies.

I've tried turning on some rpcdebug, and i do see some messages like
this:
Aug 25 09:06:38 nas1 kernel: nfsd: write complete err=32768
Aug 25 09:06:42 nas1 kernel: nfsd: write complete err=260

and somne of these:
Aug 25 08:31:48 nas1 kernel: NFSD: setlease failed [-11], no delegation

and tcpdump shows some of these:
> necros.538976288: reply ERR 1448
10:13:25.300606 IP (tos 0x0, ttl  64, id 39829, offset 0, flags [DF],
proto: TCP (6), length: 1500) nas1.nfs

I guess my question is... given the symptoms we're seeing, and that
Centos 5.3  runs a 2.6.18 kernel (with many many patches), does anyone
know if the above patch will work and be safe on 2.6.18 Centos kernel,
or of another patch specifically vs 21.6.18 kernel which will resolve
this memory leak? 

Or have any other advice on how to resolve this issue without going to
brand new kernel, which may cause other problems?

Might there be something I can do to pinpoint the problem and somehow
prevent the clients from making the calls which are going to trigger
this memory leak? or change any server configuration options so the
errors, and therefore the leaks, do not occur?

We need help to resolve this or i may soon be starting to look round for
a sturdy tree-branch or beam - something able to support about 100 kg
suspended from a thick rope and noose :(

Any advice would be very much appreciated!

John

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://linux-nfs.org/pipermail/nfsv4/attachments/20090825/66d7e735/attachment.html>