KVM/QEMU-guest, very slow disk performance?

If none of the more specific forums is the right place to ask

KVM/QEMU-guest, very slow disk performance?

Postby slavka » 2017-09-07 09:06

I'm really confused of QEMU/KVM behavior. I have a server - about 100GB RAM, 2 CPU - 6 core + HT, and three HDDs - one for root, and two for BTRFS-mirror for virtual machines (HGST HDN724030ALE640 if it's matter). OS - Debian 9.1, QEMU (disk type - raw, IO - native).

So, server is idle - it's run only one VM. This virtual machine (Debian 9.1, 16GB RAM, 4 VCPU), which runs PostgreSQL and MySQL databases. Queries receive only one - PostgreSQL (few insert per second). And... it takes lot of time, to do this inserts. I wrote simple script, which insert data to very simple table, whitout any constraints or indexes (three varchar fields). Script performance - about 1-2 inserts per second. It's extremely slow!

At first, I thought it's was Postgres trouble. But I rewrote my script for MySQL, and it shows same result. Then, I create new VM on FreeBSD, install PostgreSQL, create same database and table from Linux VM, and check performance again - 1-2 inserts per second.

Okay. Let's see on iostat and iotop. iotop shows, that Postgres process, which serves my insert queries, consume about 60-70% of IO (for simple inserts!) on VM. iostat -x says, that now uses about 40-45 writes/sec on VM, and about 80-90 w/s in host machine.

I tried to rate write speed with dd utility, like this:

Code: Select all
dd if=/dev/urandom of=/tmp/fooo.bar bs=1M count=1000


and it's done about 5-6 secs.

Next, while script execute about 100 inserts, I check file:

Code: Select all
/proc/[Postgres_PID]/io

rchar: 138544
wchar: 114689
syscr: 39
syscw: 15
read_bytes: 0
write_bytes: 540672
cancelled_write_bytes: 0


What I'm doing wrong? Why I have so slow disk performance?

UPD: I'm copied remote file (about 4,5 GB) to this VM via SFTP. It was copied on 15-20 MB/s speed (about 3-4 minutes). I really don't understand it. I thought, that it was KVM problem (2 different database have same troubles, different VM on FreeBSD have same troubled behavior, wich Linux VM), but remote file was copied fast, iotop shows 20-30% load, and iostat shows w/s about 150-200 (sometimes about 300). What's happening? Please, help someone...
slavka
 
Posts: 1
Joined: 2017-09-07 08:31

Re: KVM/QEMU-guest, very slow disk performance?

Postby pylkko » 2017-09-07 11:00

I don not know, and I am not sure if I understand what you are doing..

But btrfs is a copy-on-write (COW) file system and often the guides say that do not do COW on COW. So do not run virtual machine images that are written into the middle of on a COW system and do not run databases either. So, could it be that you have a COW btrfs partition with a COW vm image in it with a database in it and are getting massive i/o overhead?
User avatar
pylkko
 
Posts: 1045
Joined: 2014-11-06 19:02


Return to General Questions

Who is online

Users browsing this forum: No registered users and 18 guests

fashionable