I've been bashing my head against the keyboard all Sunday long over this bizarre issue. Tried strace'ing things, tail -f'ing syslog and messages. And, hell, I've even spent an hour looking at ominous tcpdump output, to no avail.
Thing is: at my workplace, we had an Debian Jessie, running ZFS and sharing homes manually (i.e. not using zfs share) over NFS. We grew tired of running ancient software and ZFS was streamed down to another server, running Debian Buster, which now operates as NFS server.
I designed the transition to be fairly seamless: a few changes in DNS and voilá, nfs.domain points to the new server. But things aren't work as expected. While it mounts over krb5p, and transfer rates are fairly good, it hiccups while opening some files.
Examples:
When logging in, there's a unacceptable five to ten seconds delay to load .bashrc and show the actual $PS1 prompt.
Also, even vim takes up several seconds (more than thirty sometimes) to open. And closing it (no pun intended) takes another forty.
Sometimes, when cat'ing .bashrc, it hangs a while (like five seconds) at an openat syscall.
Code: Select all
execve("/usr/bin/cat", ["cat", ".bashrc"], 0x7ffdf3d44398 /* 23 vars */) = 0
brk(NULL) = 0x557af9f21000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=43568, ...}) = 0
mmap(NULL, 43568, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f8979b97000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260A\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1824496, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8979b95000
mmap(NULL, 1837056, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f89799d4000
mprotect(0x7f89799f6000, 1658880, PROT_NONE) = 0
mmap(0x7f89799f6000, 1343488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7f89799f6000
mmap(0x7f8979b3e000, 311296, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16a000) = 0x7f8979b3e000
mmap(0x7f8979b8b000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b6000) = 0x7f8979b8b000
mmap(0x7f8979b91000, 14336, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f8979b91000
close(3) = 0
arch_prctl(ARCH_SET_FS, 0x7f8979b96540) = 0
mprotect(0x7f8979b8b000, 16384, PROT_READ) = 0
mprotect(0x557af833f000, 4096, PROT_READ) = 0
mprotect(0x7f8979bc9000, 4096, PROT_READ) = 0
munmap(0x7f8979b97000, 43568) = 0
brk(NULL) = 0x557af9f21000
brk(0x557af9f42000) = 0x557af9f42000
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=3031632, ...}) = 0
mmap(NULL, 3031632, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f89796ef000
close(3) = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}) = 0
openat(AT_FDCWD, ".bashrc", O_RDONLY
But there's a catch: It only happens when mounting things with krb5p security, and krb5i fared even worse, surprisingly. Exporting and mounting with "sec=sys" works okay: no slow downs, hiccups or troubles of the same sort. Kerberos seems not to be working adequately, but, as I said already, there's nothing relevant on logs. No errors, no warnings, nothing. The darned NFS thing just slows down for no reason at all.
What's happening?
Thanks,
-trp