rpc.idmapd hangs after 'bad type in upcall'

[Jeff Layton]

On Thu, 1 May 2008 15:22:58 -0400
"Kevin Coffman" <kwc at citi.umich.edu> wrote:

> On Thu, May 1, 2008 at 2:23 PM, Stirling, Ashley <ashley at mun.ca> wrote:
> >
> > We have recently been experiencing a problem where idmapd will log a 'bad
> > type in upcall' message and thereafter the nfs server will respond to all
> > requests with NFS4ERR_DELAY.  Restarting rpc.idmapd returns nfs to a
> > functioning state.
> >
> > The nfs server is running RedHat Enterprise Linux 4:
> >   kernel 2.6.9-42
> >   nfs-utils 1.0.6-80
> >
> > Attached are:
> >   -  the syslogs around the time of the most recent hang
> >   -  output from strace of the idmapd daemon
> >   -  packet capture around the time of idmapd hanging
> > (user names are blanked out to comply with our privacy policy)
> >
> >
> > I don't really know what to look for in the attached logs.  Any assistance
> > or direction you could provide would be greatly appreciated.
> >
> > Thanks,
> > - Ash
> 
> This version of idmapd doesn't properly handle a zero-length read.  I
> see this was fixed in nfs-utils-1.0.8.  I think the kernel is stuck
> waiting for an answer, and idmapd is not sending one.  I don't know if
> there is a kernel change that doesn't send an empty request.
> 
> I'm not sure what your options are for upgrading.
> 
> Below is the part of the nfs-utils patch to idmapd to handle the
> zero-length read:
> 
> ---------------------------- utils/idmapd/idmapd.c ----------------------------
> index 5712edb..158feaf 100644
> @@ -547,9 +547,10 @@ nfsdcb(int fd, short which, void *data)
>         if (which != EV_READ)
>                 goto out;
> 
> -       if ((len = read(ic->ic_fd, buf, sizeof(buf))) == -1) {
> +       if ((len = read(ic->ic_fd, buf, sizeof(buf))) <= 0) {
>                 idmapd_warnx("nfsdcb: read(%s) failed: errno %d (%s)",
> -                            ic->ic_path, errno, strerror(errno));
> +                            ic->ic_path, len?errno:0,
> +                            len?strerror(errno):"End of File");
>                 goto out;
>         }


Thanks Kevin. Ashley, I recommend opening a BZ (or even better, a RH
support case) and we'll see if we can get that fixed...

Thanks,
-- 
Jeff Layton <jlayton at redhat.com>