Kernel Panic on Backup

[Norman Elton]

Bruce,

Thanks again for your help. NFSv4 has been working beautifully until  
now!

> Just to be clear--these servers are nfsv4 *clients*, right?

Correct. The server has been stable.

> Thanks for the picture!  If there were some way to just get a couple
> more lines above the top of the screen, that'd have a few more
> details....

Unfortunately I'm a bit limited here. I don't see a panic log  
anywhere, but I can setup diskdump to dump the contents of memory to  
disk. Let me know if this would be helpful.

> What kernel version is this?

We're running RedHat 2.6.18-53.1.14.el5.

Of the 10-12 production boxes running NFSv4, only three have been  
affected so far. They are in a group of four boxes all running a Java  
application. It could be a complete coincidence... but interesting  
nonetheless. These four boxes were all rebuilt with RHEL5 & NFSv4 in  
the past two weeks. The other boxes, running the same OS and NFS  
configuration, have been stable.

It is possible that it's a firmware issue on the servers themselves. I  
believe these four boxes are a little older than our other servers.  
I'll double check this tomorrow. Also, I'm going to setup a dummy  
system that just sits there backing itself up all day long in the  
hopes of reproducing this in the lab.

Thanks again for all your help. Let me know if there's anything else I  
can do.

Norman Elton



On Mar 30, 2008, at 11:10 AM, J. Bruce Fields wrote:
> On Sat, Mar 29, 2008 at 10:25:35AM -0400, Norman Elton wrote:
>> Hello,
>>
>> We've begun rolling out NFSv4 for our servers. We mainly use it to
>> mount home directories for the sysadmins who SSH into the box, as  
>> well
>> as for a shared "scratch" space. We're not really pushing tons of  
>> data
>> here.
>>
>> I have a cron job that backs up relevant directories to the NFS mount
>> nightly using rsync:
>>
>> rsync -rlptD --stats --exclude-from=/etc/backup.conf -- 
>> exclude='*' / /
>> nfs/backup/$HOSTNAME
>>
>> Everything was working fine for the past few weeks, until I recently
>> rebuilt four servers to use NFSv4.
>
> Just to be clear--these servers are nfsv4 *clients*, right?
>
>> Three of these have locked up so
>> far, two of which died about the time of the nightly backup. I was
>> able to get a picture of the kernel panic on the third server:
>
> Thanks for the picture!  If there were some way to just get a couple
> more lines above the top of the screen, that'd have a few more
> details....
>
>>
>> http://wnelto.people.wm.edu/crash.jpg
>>
>> The panic seems to have some scary GSS / NFS messages in there, but
>> I'm unsure what to do next.
>
> Yes, so the top of the stack is:
>
> 	auth_rpcgss:gss_setup_upcall+0x91/0xf3
> 	auth_rpgss:gss_refresh+0x7c/0x12d
>
> and then nfs4_proc_setclientid is in there too.
>
> What kernel version is this?
>
> --b.
>
>>
>> Any ideas?
>>
>> Thanks,
>>
>> Norman Elton
>> College of William & Mary
>> Williamsburg VA
>> _______________________________________________
>> NFSv4 mailing list
>> NFSv4 at linux-nfs.org
>> http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4