“Cannot allocate memory” despite free reporting “available”

[aglomerator]

This is a question for a linux kernel or sys admin guru.

I got this error from qemu, trying to start a VM with 3GB memory:

ioctl(KVM_CREATE_VM) failed: 12 Cannot allocate memory
failed to initialize KVM: Cannot allocate memory

To my knowledge this can be because there isn't enough memory or the commit limit is too low, but apparently not... There is 5.9 GB available by dumping caches and no commit limit:

$ free -m
total used free shared buff/cache available
Mem: 7696 1287 135 139 6274 5973
Swap: 2892 525 2367

$ cat /proc/sys/vm/overcommit_memory
1

So then I wrote a c++ program to allocate successively bigger chunks. I found that it failed to allocate more than 2.1 GB. (N.B. it was compiled as 64-bit.) That's consistent with Qemu not starting up, but WHY???

So then I modified it to write to the memory as well. This resulted in some caches being dumped, with free reporting this at about 2 GB allocated:

$ free -m
total used free shared buff/cache available
Mem: 7696 2988 288 143 4420 4268
Swap: 2892 525 2367

...and this when the program terminated:

$ free -m
total used free shared buff/cache available
Mem: 7696 1258 2253 147 4185 5994
Swap: 2892 525 2367

Now I tried to start Qemu and magically it worked! Free reports:

$ free -m
total used free shared buff/cache available
Mem: 7696 2438 4451 147 806 4834
Swap: 2892 530 2362

So it looks like the kernel wasn't willing to free some caches when asked for a big chunk, but was willing to let them go in several steps. What's going on?

I am running Debian testing: Linux - 4.11.0-1-amd64 #1 SMP Debian 4.11.6-1 (2017-06-19) x86_64 GNU/Linux