rpc.idmapd hangs after 'bad type in upcall'

[Kevin Coffman]

On Thu, May 1, 2008 at 2:23 PM, Stirling, Ashley <ashley at mun.ca> wrote:
>
> We have recently been experiencing a problem where idmapd will log a 'bad
> type in upcall' message and thereafter the nfs server will respond to all
> requests with NFS4ERR_DELAY.  Restarting rpc.idmapd returns nfs to a
> functioning state.
>
> The nfs server is running RedHat Enterprise Linux 4:
>   kernel 2.6.9-42
>   nfs-utils 1.0.6-80
>
> Attached are:
>   -  the syslogs around the time of the most recent hang
>   -  output from strace of the idmapd daemon
>   -  packet capture around the time of idmapd hanging
> (user names are blanked out to comply with our privacy policy)
>
>
> I don't really know what to look for in the attached logs.  Any assistance
> or direction you could provide would be greatly appreciated.
>
> Thanks,
> - Ash

This version of idmapd doesn't properly handle a zero-length read.  I
see this was fixed in nfs-utils-1.0.8.  I think the kernel is stuck
waiting for an answer, and idmapd is not sending one.  I don't know if
there is a kernel change that doesn't send an empty request.

I'm not sure what your options are for upgrading.

Below is the part of the nfs-utils patch to idmapd to handle the
zero-length read:

---------------------------- utils/idmapd/idmapd.c ----------------------------
index 5712edb..158feaf 100644
@@ -547,9 +547,10 @@ nfsdcb(int fd, short which, void *data)
        if (which != EV_READ)
                goto out;

-       if ((len = read(ic->ic_fd, buf, sizeof(buf))) == -1) {
+       if ((len = read(ic->ic_fd, buf, sizeof(buf))) <= 0) {
                idmapd_warnx("nfsdcb: read(%s) failed: errno %d (%s)",
-                            ic->ic_path, errno, strerror(errno));
+                            ic->ic_path, len?errno:0,
+                            len?strerror(errno):"End of File");
                goto out;
        }