rpc.idmapd hangs after 'bad type in upcall'

[Ash Stirling]

On 01/05/08 5:25 PM, "Jeff Layton" <jlayton at redhat.com> wrote:

> On Thu, 1 May 2008 15:22:58 -0400
> "Kevin Coffman" <kwc at citi.umich.edu> wrote:
> 
>> On Thu, May 1, 2008 at 2:23 PM, Stirling, Ashley <ashley at mun.ca> wrote:
>>> 
>>> We have recently been experiencing a problem where idmapd will log a 'bad
>>> type in upcall' message and thereafter the nfs server will respond to all
>>> requests with NFS4ERR_DELAY.  Restarting rpc.idmapd returns nfs to a
>>> functioning state.
>>> 
>>> The nfs server is running RedHat Enterprise Linux 4:
>>>   kernel 2.6.9-42
>>>   nfs-utils 1.0.6-80
>>> 
>>> Attached are:
>>>   -  the syslogs around the time of the most recent hang
>>>   -  output from strace of the idmapd daemon
>>>   -  packet capture around the time of idmapd hanging
>>> (user names are blanked out to comply with our privacy policy)
>>> 
>>> 
>>> I don't really know what to look for in the attached logs.  Any assistance
>>> or direction you could provide would be greatly appreciated.
>>> 
>>> Thanks,
>>> - Ash
>> 
>> This version of idmapd doesn't properly handle a zero-length read.  I
>> see this was fixed in nfs-utils-1.0.8.  I think the kernel is stuck
>> waiting for an answer, and idmapd is not sending one.  I don't know if
>> there is a kernel change that doesn't send an empty request.
>> 
>> I'm not sure what your options are for upgrading.
>> 
>> Below is the part of the nfs-utils patch to idmapd to handle the
>> zero-length read:
>> 
>> ---------------------------- utils/idmapd/idmapd.c
>> ----------------------------
>> index 5712edb..158feaf 100644
>> @@ -547,9 +547,10 @@ nfsdcb(int fd, short which, void *data)
>>         if (which != EV_READ)
>>                 goto out;
>> 
>> -       if ((len = read(ic->ic_fd, buf, sizeof(buf))) == -1) {
>> +       if ((len = read(ic->ic_fd, buf, sizeof(buf))) <= 0) {
>>                 idmapd_warnx("nfsdcb: read(%s) failed: errno %d (%s)",
>> -                            ic->ic_path, errno, strerror(errno));
>> +                            ic->ic_path, len?errno:0,
>> +                            len?strerror(errno):"End of File");
>>                 goto out;
>>         }
> 
> 
> Thanks Kevin. Ashley, I recommend opening a BZ (or even better, a RH
> support case) and we'll see if we can get that fixed...
> 
> Thanks,

Kevin, Jeff; thank you both for you help.

I will open an RH support case today.  If I get a resolution through those
channels, I will post back to the list for future googlers.

Thanks again,
- Ash