[Solved] Hiccups on NFS over krb5p

Kernels & Hardware, configuring network, installing services

[Solved] Hiccups on NFS over krb5p

Postby prppedro » 2020-03-09 05:40

Good night, fellow sysadmins

I've been bashing my head against the keyboard all Sunday long over this bizarre issue. Tried strace'ing things, tail -f'ing syslog and messages. And, hell, I've even spent an hour looking at ominous tcpdump output, to no avail.

Thing is: at my workplace, we had an Debian Jessie, running ZFS and sharing homes manually (i.e. not using zfs share) over NFS. We grew tired of running ancient software and ZFS was streamed down to another server, running Debian Buster, which now operates as NFS server.

I designed the transition to be fairly seamless: a few changes in DNS and voilá, nfs.domain points to the new server. But things aren't work as expected. While it mounts over krb5p, and transfer rates are fairly good, it hiccups while opening some files.

Examples:
When logging in, there's a unacceptable five to ten seconds delay to load .bashrc and show the actual $PS1 prompt.
Also, even vim takes up several seconds (more than thirty sometimes) to open. And closing it (no pun intended) takes another forty.

Sometimes, when cat'ing .bashrc, it hangs a while (like five seconds) at an openat syscall.
Code: Select all
execve("/usr/bin/cat", ["cat", ".bashrc"], 0x7ffdf3d44398 /* 23 vars */) = 0
brk(NULL)                               = 0x557af9f21000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=43568, ...}) = 0
mmap(NULL, 43568, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f8979b97000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260A\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1824496, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8979b95000
mmap(NULL, 1837056, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f89799d4000
mprotect(0x7f89799f6000, 1658880, PROT_NONE) = 0
mmap(0x7f89799f6000, 1343488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7f89799f6000
mmap(0x7f8979b3e000, 311296, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16a000) = 0x7f8979b3e000
mmap(0x7f8979b8b000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b6000) = 0x7f8979b8b000
mmap(0x7f8979b91000, 14336, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f8979b91000
close(3)                                = 0
arch_prctl(ARCH_SET_FS, 0x7f8979b96540) = 0
mprotect(0x7f8979b8b000, 16384, PROT_READ) = 0
mprotect(0x557af833f000, 4096, PROT_READ) = 0
mprotect(0x7f8979bc9000, 4096, PROT_READ) = 0
munmap(0x7f8979b97000, 43568)           = 0
brk(NULL)                               = 0x557af9f21000
brk(0x557af9f42000)                     = 0x557af9f42000
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=3031632, ...}) = 0
mmap(NULL, 3031632, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f89796ef000
close(3)                                = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0), ...}) = 0
openat(AT_FDCWD, ".bashrc", O_RDONLY


For the next few minutes, the file is cached and is always pretty easily read.

But there's a catch: It only happens when mounting things with krb5p security, and krb5i fared even worse, surprisingly. Exporting and mounting with "sec=sys" works okay: no slow downs, hiccups or troubles of the same sort. Kerberos seems not to be working adequately, but, as I said already, there's nothing relevant on logs. No errors, no warnings, nothing. The darned NFS thing just slows down for no reason at all.

What's happening?

Thanks,
-trp
Linux User #457653

My PC specs: a bunch of old Intel chips, some memory, a just good enough GPU. I guess it ran Crysis, though UPS didn't quite like it.

In my experience, there's no such thing as Year of Linux...
User avatar
prppedro
 
Posts: 28
Joined: 2017-03-13 16:09
Location: São Paulo/SP

Re: [Solved] Hiccups on NFS over krb5p

Postby prppedro » 2020-03-09 06:21

Well, nervermind.

Seems like I got it.

Initially, I've copied over the keytab file from the old server and added a few other keytabs to it. Now, I removed "nfs/nfs.domain" and "host/nfs.domain", then re-added them, thus increasing the kvno. Everything's working fine, now, it seems.
Linux User #457653

My PC specs: a bunch of old Intel chips, some memory, a just good enough GPU. I guess it ran Crysis, though UPS didn't quite like it.

In my experience, there's no such thing as Year of Linux...
User avatar
prppedro
 
Posts: 28
Joined: 2017-03-13 16:09
Location: São Paulo/SP


Return to System configuration

Who is online

Users browsing this forum: No registered users and 12 guests

fashionable