How calculate used mem with /proc/meminfo?

If none of the more specific forums is the right place to ask

How calculate used mem with /proc/meminfo?

Postby bedtime » 2018-02-15 18:21

I understand very well that I could just use the 'free' command. The reason I would like to do this is efficiency. Also, it's a good learning experience. :wink:

In the calculation below, I only use the awk command once to read a file. I believe that this is much more efficient than using the free command if one had to compare side-by-side. Perhaps, after, I could do a test to find out? :)

Here is what I have so far:

Code: Select all
awk '/MemTotal/ {total=$2} /MemFree/ {free=$2} /Buffers/ {buffers=$2} $1 ~ /^Cached/ {cached=$2} END {printf "%.0f\n",(total - (free + buffers + cached))/1000}' /proc/meminfo


Here is what is going on above (so you don't have to rack your brains too much):

awk grabs MemTotal, MemFree, Buffers (has to make sure it matches the whole word or it grabs the wrong number), and Cached (again, match the whole word...). The calculation used is: MemTotal - (MemFree + Buffers + Cached) as per https://access.redhat.com/solutions/406773. The result is divided by 1000 to make human readable (should it be 1024 or another number?) Then rounded to 1s (i.e., "%.0f/n").

Code: Select all
$ cat /proc/meminfo
MemTotal:        3914524 kB
MemFree:         2573412 kB
MemAvailable:    2987780 kB
Buffers:           34068 kB
Cached:           937708 kB
SwapCached:            0 kB
Active:           524700 kB
Inactive:         760272 kB
Active(anon):     319028 kB
Inactive(anon):   175120 kB
Active(file):     205672 kB
Inactive(file):   585152 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4062204 kB
SwapFree:        4062204 kB
Dirty:                48 kB
Writeback:             0 kB
AnonPages:        308904 kB
Mapped:           120608 kB
Shmem:            180960 kB
Slab:              34704 kB
SReclaimable:      23920 kB
SUnreclaim:        10784 kB
KernelStack:        1472 kB
PageTables:         4196 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6019464 kB
Committed_AS:     711844 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
AnonHugePages:     83968 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
DirectMap4k:       46580 kB
DirectMap2M:     4018176 kB


The result:
370


Comparing to the 'free' command:
Code: Select all
$ free -m
              total        used        free      shared  buff/cache   available
Mem:           3822         337        2512         176         972        2916
Swap:          3966           0        3966


It's 33mb off. This is the issue: how do I make this result match the free -m command?

Sorry about the long post, but I wanted on the info there at once. As well, just to make this more complex than it already is, I am using Transparent Huge Pages set to [Always] if that has any effect on this? It is kernel enabled.

A reading closer to the one used with the htop command might be attained with (as per https://codedump.io/share/eSQbQNrsUB8O/ ... -like-htop):

Code: Select all
awk '/MemTotal/ {total=$2} /MemFree/ {free=$2} /Buffers/ {buffers=$2} $1 ~ /^Cache/ {cached=$2} /SReclaimable/ {reclaim=$2} /Shmem:/ {shmem=$2} END {printf "%.0f", ((total - free) - (buffers + cached + reclaim - shmem)) / 1000}' /proc/meminfo


This time it is about 15mb off when I divide by 1000, and right on when I divide by 1024.
OS: Debian GNU/Linux (Buster—testing); Kernel: 4.16.0-2-amd64; Shell: sh; WM: i3
Machine: HP ProDesk 600 G3—i7-7700, 8GB RAM, 256 SSD, Intel HD Graphics 630

bedtime
 
Posts: 93
Joined: 2012-12-16 19:34

Re: How calculate used mem with /proc/meminfo?

Postby p.H » 2018-02-15 19:17

The used memory is just (MemTotal - MemFree). Any other calculation is wrong.
p.H
 
Posts: 645
Joined: 2017-09-17 07:12

Re: How calculate used mem with /proc/meminfo?

Postby debiman » 2018-02-16 05:24

bedtime wrote:In the calculation below, I only use the awk command once to read a file. I believe that this is much more efficient than using the free command if one had to compare side-by-side.
why do you believe that?
free does essentially the same:
The information is gathered by parsing /proc/meminfo.
...only it is likely coded in C specifically for that purpose.
that is most likely more efficient.

anyhow...
This time it is about 15mb off when I divide by 1000, and right on when I divide by 1024.
so all's good?
User avatar
debiman
 
Posts: 3064
Joined: 2013-03-12 07:18

Re: How calculate used mem with /proc/meminfo?

Postby bedtime » 2018-02-16 09:57

p.H wrote:The used memory is just (MemTotal - MemFree). Any other calculation is wrong.

In the grand scheme, I agree; but I would think that most people use the 'free -m' command as their common reference.

why do you believe that? free does essentially the same: The information is gathered by parsing /proc/meminfo. only it is likely coded in C specifically for that purpose. that is most likely more efficient.

I ran tests to find out the validity of this. I admit it was a crude test. If anyone could think of a better one, let me know:

Hypothesis:

Running the 'free -m' command will take more resources, time, and cpu.

Test:

Run a command 10000 times and take measure of time and cpu.

Code: Select all
$ cat test.sh
#!/bin/mksh

#run this test with: time ./test.sh

count=0

while [ $count -le 10000 ]
do

   #free -m | awk '/^Mem/ {print $3}'
   
   awk '/MemTotal/ {total=$2} /MemFree/ {free=$2} /Buffers/ {buffers=$2} $1 ~ /^Cache/ {cached=$2} /SReclaimable/ {reclaim=$2} /Shmem:/ {shmem=$2} END {printf "%.1f", ((total - free) - (buffers + cached)) / 1024}' /proc/meminfo

   count=$((count+1))
done


Results (lower the better):

Code: Select all
free -m | awk '/^Mem/ {print $3}'

0m14.04s real 0m09.76s user 0m08.38s system
cpu usage during test: 70%

Code: Select all
awk '/MemTotal/ {total=$2} /MemFree/ {free=$2} /Buffers/ {buffers=$2} $1 ~ /^Cache/ {cached=$2} /SReclaimable/ {reclaim=$2} /Shmem:/ {shmem=$2} END {printf "%.1f", ((total - free) - (buffers + cached)) / 1024}' /proc/meminfo

0m06.94s real 0m03.82s user 0m02.12s system
cpu usage during test: 52%

Conclusion:

The 'free -m' command took 2.02 times longer to perfrom the same task and used 34.6% more cpu whilst doing so.

Notes:

I did not measure memory as I very oddly could not find the 'free' process anywhere. :shock: When I did it said it took 0mb, which could not possibly be true. In any event, it likely takes more memory as it uses both 'awk' and 'free' and mine uses only 'awk'.

I am reminded of science class! :D

This time it is about 15mb off when I divide by 1000, and right on when I divide by 1024.
so all's good?

I have the calculation for the 'htop' command, but I would like to know the formula that the 'free -m' command uses.
OS: Debian GNU/Linux (Buster—testing); Kernel: 4.16.0-2-amd64; Shell: sh; WM: i3
Machine: HP ProDesk 600 G3—i7-7700, 8GB RAM, 256 SSD, Intel HD Graphics 630

bedtime
 
Posts: 93
Joined: 2012-12-16 19:34

Re: How calculate used mem with /proc/meminfo?

Postby sunrat » 2018-02-16 21:20

bedtime wrote:I am reminded of science class! :D

It does indeed sound rather academic. I'll keep this thread in mind next time I want to run free 10,000 times. :wink:
“ computer users can be divided into 2 categories:
Those who have lost data
...and those who have not lost data YET ”
Remember to BACKUP!
User avatar
sunrat
 
Posts: 2495
Joined: 2006-08-29 09:12
Location: Melbourne, Australia

Re: How calculate used mem with /proc/meminfo?

Postby Head_on_a_Stick » 2018-02-17 10:34

debiman wrote:free does essentially the same:
The information is gathered by parsing /proc/meminfo.
...only it is likely coded in C specifically for that purpose.
that is most likely more efficient

+1

I made a slightly modified version of the OP's script to aid testing:
Code: Select all
alpine:~$ cat ~/bin/memtest
#!/bin/sh
count=0
while [ $count -le 10000 ]; do
   case "$1" in
      "free") free -m | awk '/^Mem/ {print $3}' > /dev/null;;
      "awk") awk '/MemTotal/ {total=$2} /MemFree/ {free=$2} /Buffers/ {buffers=$2} $1 ~ /^Cache/ {cached=$2} /SReclaimable/ {reclaim=$2} /Shmem:/ {shmem=$2} END {printf "%.1f", ((total - free) - (buffers + cached)) / 1024}' /proc/meminfo > /dev/null;;
   esac
   count=$((count+1))
done
alpine:~$

This accepts either "free" or "awk" as an argument to the `memtest` command and will run the appropriate backend accordingly.

On my Alpine Linux system we see:
Code: Select all
alpine:~$ time memtest free
    0m08.05s real     0m10.16s user     0m03.12s system
alpine:~$

With htop showing ~50% CPU and:
Code: Select all
alpine:~$ time memtest awk
    0m14.87s real     0m11.80s user     0m03.14s system
alpine:~$

At ~33% CPU.

So on my box procps appears to be more efficient.

EDIT: tried this from my Debian stretch virtual machine (QEMU/KVM, Alpine Linux host):
Code: Select all
empty@virtlab:~ $ time memtest awk
    0m13.89s real     0m00.33s user     0m01.44s system
empty@virtlab:~ $ time memtest free
    0m23.83s real     0m00.43s user     0m02.24s system
empty@virtlab:~ $

The `free` version ran at ~40% CPU (in the guest) with `awk` holding to ~28%

WTF is wrong with Debian's procps? :?
dbruce wrote:Ubuntu forums try to be like a coffee shop in Seattle. Debian forums strive for the charm and ambience of a skinhead bar in Bacau. We intend to keep it that way.
User avatar
Head_on_a_Stick
 
Posts: 8315
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: How calculate used mem with /proc/meminfo?

Postby debiman » 2018-02-17 11:51

bedtime wrote:free -m | awk '/^Mem/ {print $3}'


Conclusion:

The 'free -m' command took 2.02 times longer to perfrom the same task and used 34.6% more cpu whilst doing so.

wrong.
what you timed is 2 commands: free piped into awk - and you compare this to a pure awk script.
compare just 'free -m' to your awk script, you'll get different results.

i do understand that the scripts output is what you want, though.
User avatar
debiman
 
Posts: 3064
Joined: 2013-03-12 07:18

Re: How calculate used mem with /proc/meminfo?

Postby Head_on_a_Stick » 2018-02-17 14:36

debiman wrote:compare just 'free -m' to your awk script, you'll get different results

Are you not curious enough to try yourself?

On my system it improves the time by about half a second so it doesn't seem to make much difference.

How does your Arch system's procps compare to the awk script?

bedtime wrote:I would like to know the formula that the 'free -m' command uses

It's buried away in the code — how's your C?

https://gitlab.com/procps-ng/procps/raw/master/free.c

:mrgreen:
dbruce wrote:Ubuntu forums try to be like a coffee shop in Seattle. Debian forums strive for the charm and ambience of a skinhead bar in Bacau. We intend to keep it that way.
User avatar
Head_on_a_Stick
 
Posts: 8315
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: How calculate used mem with /proc/meminfo?

Postby bedtime » 2018-02-17 17:36

sunrat wrote:
bedtime wrote:I am reminded of science class! :D

It does indeed sound rather academic. I'll keep this thread in mind next time I want to run free 10,000 times. :wink:


Most people will run this far more often if they choose to use it.

For example:

I have it running in my dwm bar. It refreshes every second. I want to make sure that when it does refresh, it's making as little an interruption as possible to the system; almost like an impreceptable blip. After 24 hrs, it will have exceeded 86400 'blips' (24 x 60 x 60).

Now, there are people that love their 'blips', that's why they created Kde and Gnome... :P

Head_on_a_Stick wrote:
debiman wrote:free does essentially the same:
The information is gathered by parsing /proc/meminfo.
...only it is likely coded in C specifically for that purpose.
that is most likely more efficient

Thank you so much for giving it a test too! This made my day! :D

I made a slightly modified version of the OP's script to aid testing:
Code: Select all
alpine:~$ cat ~/bin/memtest
#!/bin/sh
count=0
while [ $count -le 10000 ]; do
   case "$1" in
      "free") free -m | awk '/^Mem/ {print $3}' > /dev/null;;
      "awk") awk '/MemTotal/ {total=$2} /MemFree/ {free=$2} /Buffers/ {buffers=$2} $1 ~ /^Cache/ {cached=$2} /SReclaimable/ {reclaim=$2} /Shmem:/ {shmem=$2} END {printf "%.1f", ((total - free) - (buffers + cached)) / 1024}' /proc/meminfo > /dev/null;;
   esac
   count=$((count+1))
done
alpine:~$

This accepts either "free" or "awk" as an argument to the `memtest` command and will run the appropriate backend accordingly.

Nice. I'll be using it for further studies. :)

On my Alpine Linux system we see:
With htop showing ~50%

I've seen 'top' being used around the net for this very purpose, and it makes me shudder. It was the first solution I saw and I refused to even try it. I just knew... :| But, it is great for something to compare to. Just seems like it has a lot to load up compared to 'free' and others.


The `free` version ran at ~40% CPU (in the guest) with `awk` holding to ~28%

WTF is wrong with Debian's procps? :?

The free command itself is just fine, I gather. I believe that the apps are built to work on all computers and solve many bugs. This all takes additional code and junk that most of us don't need but can come in handy. I'm betting you'd see quite a runaround in the code. I'm actually surprised that it did so well, considering.

debiman wrote:
bedtime wrote:free -m | awk '/^Mem/ {print $3}'
Conclusion:

The 'free -m' command took 2.02 times longer to perfrom the same task and used 34.6% more cpu whilst doing so.

wrong.
what you timed is 2 commands: free piped into awk - and you compare this to a pure awk script.
compare just 'free -m' to your awk script, you'll get different results.

i do understand that the scripts output is what you want, though.

Yes, if free -m could do all this itself, then I would not need to use awk with it; but that is not really rational to expect such a thing.

Head_on_a_Stick wrote:
bedtime wrote:I would like to know the formula that the 'free -m' command uses

It's buried away in the code — how's your C?

https://gitlab.com/procps-ng/procps/raw/master/free.c
:mrgreen:

I've dabbled in C++ for a time, but it's really not my interest anymore. Would be nice to know, though. Or, to even perheps make a 'sound' formula with the data from /proc/meminfo. I almost feel like getting in touch with the developer, but I don't think it's an appropriate considering the reason.
OS: Debian GNU/Linux (Buster—testing); Kernel: 4.16.0-2-amd64; Shell: sh; WM: i3
Machine: HP ProDesk 600 G3—i7-7700, 8GB RAM, 256 SSD, Intel HD Graphics 630

bedtime
 
Posts: 93
Joined: 2012-12-16 19:34

Re: How calculate used mem with /proc/meminfo?

Postby Head_on_a_Stick » 2018-02-17 17:51

bedtime wrote:
On my Alpine Linux system we see:
With htop showing ~50%

I've seen 'top' being used around the net for this very purpose, and it makes me shudder. It was the first solution I saw and I refused to even try it. I just knew... :| But, it is great for something to compare to. Just seems like it has a lot to load up compared to 'free' and others

Ah no, you misunderstand me — I was using htop to show the average CPU usage while running the tests.

The free command itself is just fine, I gather

Yes indeed but my point was why is the free command faster than your awk script under Alpine Linux?

Your script is clearly faster in Debian (as seen in my VM) but why?

@debiman has an Arch box so perhaps if that also has a slow `free` command then it may be down to the libc versions (Alpine has musl, Debian has GNU).

I have it running in my dwm bar. It refreshes every second. I want to make sure that when it does refresh, it's making as little an interruption as possible to the system

If you want to use a script to feed dwm's status bar then be sure to call the loop with a /bin/sh shebang rather than bash or mksh (dash is lighter than either).

Here's my startup script:
Code: Select all
alpine:~$ cat .xinitrc
#!/bin/sh
setxkbmap gb
feh --bg-center ~/Pictures/dwm.png
urxvtd -q -f -o
while sleep 1;do xsetroot -name "$(cat /sys/class/power_supply/BAT0/capacity)%  •  $(date +'%F  •  %T')";done&
exec dwm
alpine:~$

If #!/bin/sh is removed then my default shell (loksh) is used for the while loop and it is (slightly) heavier than the POSIX shell (/bin/sh is linked to busybox in Alpine).

For an even lighter solution, try slstatus instead:

https://github.com/drkhsh/slstatus

This is coded in pure C (much better than silly old C++) and allows the desired modules to be loaded as required, it is _very_ nice indeed :)
dbruce wrote:Ubuntu forums try to be like a coffee shop in Seattle. Debian forums strive for the charm and ambience of a skinhead bar in Bacau. We intend to keep it that way.
User avatar
Head_on_a_Stick
 
Posts: 8315
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: How calculate used mem with /proc/meminfo?

Postby None1975 » 2018-02-19 15:29

Head_on_a_Stick wrote:For an even lighter solution, try slstatus instead:https://github.com/drkhsh/slstatusThis is coded in pure C (much better than silly old C++) and allows the desired modules to be loaded as required, it is _very_ nice indeed :)

Yes. thumbs up for slstatus! It usesWM_NAME or stdin to fill the status bar.
User avatar
None1975
 
Posts: 698
Joined: 2015-11-29 18:23
Location: Lithuania, Vilnius


Return to General Questions

Who is online

Users browsing this forum: No registered users and 7 guests

fashionable