Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

How calculate used mem with /proc/meminfo?

If none of the specific sub-forums seem right for your thread, ask here.
Post Reply
Message
Author
bedtime
Posts: 146
Joined: 2012-12-16 19:34
Has thanked: 1 time
Been thanked: 6 times

How calculate used mem with /proc/meminfo?

#1 Post by bedtime »

I understand very well that I could just use the 'free' command. The reason I would like to do this is efficiency. Also, it's a good learning experience. :wink:

In the calculation below, I only use the awk command once to read a file. I believe that this is much more efficient than using the free command if one had to compare side-by-side. Perhaps, after, I could do a test to find out? :)

Here is what I have so far:

Code: Select all

awk '/MemTotal/ {total=$2} /MemFree/ {free=$2} /Buffers/ {buffers=$2} $1 ~ /^Cached/ {cached=$2} END {printf "%.0f\n",(total - (free + buffers + cached))/1000}' /proc/meminfo
Here is what is going on above (so you don't have to rack your brains too much):

awk grabs MemTotal, MemFree, Buffers (has to make sure it matches the whole word or it grabs the wrong number), and Cached (again, match the whole word...). The calculation used is: MemTotal - (MemFree + Buffers + Cached) as per https://access.redhat.com/solutions/406773. The result is divided by 1000 to make human readable (should it be 1024 or another number?) Then rounded to 1s (i.e., "%.0f/n").

Code: Select all

$ cat /proc/meminfo
MemTotal:        3914524 kB
MemFree:         2573412 kB
MemAvailable:    2987780 kB
Buffers:           34068 kB
Cached:           937708 kB
SwapCached:            0 kB
Active:           524700 kB
Inactive:         760272 kB
Active(anon):     319028 kB
Inactive(anon):   175120 kB
Active(file):     205672 kB
Inactive(file):   585152 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4062204 kB
SwapFree:        4062204 kB
Dirty:                48 kB
Writeback:             0 kB
AnonPages:        308904 kB
Mapped:           120608 kB
Shmem:            180960 kB
Slab:              34704 kB
SReclaimable:      23920 kB
SUnreclaim:        10784 kB
KernelStack:        1472 kB
PageTables:         4196 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6019464 kB
Committed_AS:     711844 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
AnonHugePages:     83968 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
DirectMap4k:       46580 kB
DirectMap2M:     4018176 kB
The result:
370
Comparing to the 'free' command:

Code: Select all

$ free -m
              total        used        free      shared  buff/cache   available
Mem:           3822         337        2512         176         972        2916
Swap:          3966           0        3966
It's 33mb off. This is the issue: how do I make this result match the free -m command?

Sorry about the long post, but I wanted on the info there at once. As well, just to make this more complex than it already is, I am using Transparent Huge Pages set to [Always] if that has any effect on this? It is kernel enabled.

A reading closer to the one used with the htop command might be attained with (as per https://codedump.io/share/eSQbQNrsUB8O/ ... -like-htop):

Code: Select all

awk '/MemTotal/ {total=$2} /MemFree/ {free=$2} /Buffers/ {buffers=$2} $1 ~ /^Cache/ {cached=$2} /SReclaimable/ {reclaim=$2} /Shmem:/ {shmem=$2} END {printf "%.0f", ((total - free) - (buffers + cached + reclaim - shmem)) / 1000}' /proc/meminfo
This time it is about 15mb off when I divide by 1000, and right on when I divide by 1024.

p.H
Global Moderator
Global Moderator
Posts: 3049
Joined: 2017-09-17 07:12
Has thanked: 5 times
Been thanked: 132 times

Re: How calculate used mem with /proc/meminfo?

#2 Post by p.H »

The used memory is just (MemTotal - MemFree). Any other calculation is wrong.

User avatar
debiman
Posts: 3063
Joined: 2013-03-12 07:18

Re: How calculate used mem with /proc/meminfo?

#3 Post by debiman »

bedtime wrote:In the calculation below, I only use the awk command once to read a file. I believe that this is much more efficient than using the free command if one had to compare side-by-side.
why do you believe that?
free does essentially the same:
The information is gathered by parsing /proc/meminfo.
...only it is likely coded in C specifically for that purpose.
that is most likely more efficient.

anyhow...
This time it is about 15mb off when I divide by 1000, and right on when I divide by 1024.
so all's good?

bedtime
Posts: 146
Joined: 2012-12-16 19:34
Has thanked: 1 time
Been thanked: 6 times

Re: How calculate used mem with /proc/meminfo?

#4 Post by bedtime »

p.H wrote:The used memory is just (MemTotal - MemFree). Any other calculation is wrong.
In the grand scheme, I agree; but I would think that most people use the 'free -m' command as their common reference.
why do you believe that? free does essentially the same: The information is gathered by parsing /proc/meminfo. only it is likely coded in C specifically for that purpose. that is most likely more efficient.
I ran tests to find out the validity of this. I admit it was a crude test. If anyone could think of a better one, let me know:

Hypothesis:

Running the 'free -m' command will take more resources, time, and cpu.

Test:

Run a command 10000 times and take measure of time and cpu.

Code: Select all

$ cat test.sh
#!/bin/mksh

#run this test with: time ./test.sh

count=0

while [ $count -le 10000 ]
do

	#free -m | awk '/^Mem/ {print $3}'
	
   awk '/MemTotal/ {total=$2} /MemFree/ {free=$2} /Buffers/ {buffers=$2} $1 ~ /^Cache/ {cached=$2} /SReclaimable/ {reclaim=$2} /Shmem:/ {shmem=$2} END {printf "%.1f", ((total - free) - (buffers + cached)) / 1024}' /proc/meminfo

	count=$((count+1))
done
Results (lower the better):

Code: Select all

free -m | awk '/^Mem/ {print $3}'
0m14.04s real 0m09.76s user 0m08.38s system
cpu usage during test: 70%

Code: Select all

awk '/MemTotal/ {total=$2} /MemFree/ {free=$2} /Buffers/ {buffers=$2} $1 ~ /^Cache/ {cached=$2} /SReclaimable/ {reclaim=$2} /Shmem:/ {shmem=$2} END {printf "%.1f", ((total - free) - (buffers + cached)) / 1024}' /proc/meminfo
0m06.94s real 0m03.82s user 0m02.12s system
cpu usage during test: 52%

Conclusion:

The 'free -m' command took 2.02 times longer to perfrom the same task and used 34.6% more cpu whilst doing so.

Notes:

I did not measure memory as I very oddly could not find the 'free' process anywhere. :shock: When I did it said it took 0mb, which could not possibly be true. In any event, it likely takes more memory as it uses both 'awk' and 'free' and mine uses only 'awk'.

I am reminded of science class! :D
This time it is about 15mb off when I divide by 1000, and right on when I divide by 1024.
so all's good?
I have the calculation for the 'htop' command, but I would like to know the formula that the 'free -m' command uses.

User avatar
sunrat
Administrator
Administrator
Posts: 6382
Joined: 2006-08-29 09:12
Location: Melbourne, Australia
Has thanked: 115 times
Been thanked: 456 times

Re: How calculate used mem with /proc/meminfo?

#5 Post by sunrat »

bedtime wrote:I am reminded of science class! :D
It does indeed sound rather academic. I'll keep this thread in mind next time I want to run free 10,000 times. :wink:
“ computer users can be divided into 2 categories:
Those who have lost data
...and those who have not lost data YET ”
Remember to BACKUP!

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: How calculate used mem with /proc/meminfo?

#6 Post by Head_on_a_Stick »

debiman wrote:free does essentially the same:
The information is gathered by parsing /proc/meminfo.
...only it is likely coded in C specifically for that purpose.
that is most likely more efficient
+1

I made a slightly modified version of the OP's script to aid testing:

Code: Select all

alpine:~$ cat ~/bin/memtest
#!/bin/sh
count=0
while [ $count -le 10000 ]; do
   case "$1" in
      "free") free -m | awk '/^Mem/ {print $3}' > /dev/null;;
      "awk") awk '/MemTotal/ {total=$2} /MemFree/ {free=$2} /Buffers/ {buffers=$2} $1 ~ /^Cache/ {cached=$2} /SReclaimable/ {reclaim=$2} /Shmem:/ {shmem=$2} END {printf "%.1f", ((total - free) - (buffers + cached)) / 1024}' /proc/meminfo > /dev/null;;
   esac
   count=$((count+1))
done
alpine:~$
This accepts either "free" or "awk" as an argument to the `memtest` command and will run the appropriate backend accordingly.

On my Alpine Linux system we see:

Code: Select all

alpine:~$ time memtest free
    0m08.05s real     0m10.16s user     0m03.12s system
alpine:~$
With htop showing ~50% CPU and:

Code: Select all

alpine:~$ time memtest awk 
    0m14.87s real     0m11.80s user     0m03.14s system
alpine:~$
At ~33% CPU.

So on my box procps appears to be more efficient.

EDIT: tried this from my Debian stretch virtual machine (QEMU/KVM, Alpine Linux host):

Code: Select all

empty@virtlab:~ $ time memtest awk
    0m13.89s real     0m00.33s user     0m01.44s system
empty@virtlab:~ $ time memtest free
    0m23.83s real     0m00.43s user     0m02.24s system
empty@virtlab:~ $
The `free` version ran at ~40% CPU (in the guest) with `awk` holding to ~28%

WTF is wrong with Debian's procps? :?
deadbang

User avatar
debiman
Posts: 3063
Joined: 2013-03-12 07:18

Re: How calculate used mem with /proc/meminfo?

#7 Post by debiman »

bedtime wrote:free -m | awk '/^Mem/ {print $3}'


Conclusion:

The 'free -m' command took 2.02 times longer to perfrom the same task and used 34.6% more cpu whilst doing so.
wrong.
what you timed is 2 commands: free piped into awk - and you compare this to a pure awk script.
compare just 'free -m' to your awk script, you'll get different results.

i do understand that the scripts output is what you want, though.

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: How calculate used mem with /proc/meminfo?

#8 Post by Head_on_a_Stick »

debiman wrote:compare just 'free -m' to your awk script, you'll get different results
Are you not curious enough to try yourself?

On my system it improves the time by about half a second so it doesn't seem to make much difference.

How does your Arch system's procps compare to the awk script?
bedtime wrote:I would like to know the formula that the 'free -m' command uses
It's buried away in the code — how's your C?

https://gitlab.com/procps-ng/procps/raw/master/free.c

:mrgreen:
deadbang

bedtime
Posts: 146
Joined: 2012-12-16 19:34
Has thanked: 1 time
Been thanked: 6 times

Re: How calculate used mem with /proc/meminfo?

#9 Post by bedtime »

sunrat wrote:
bedtime wrote:I am reminded of science class! :D
It does indeed sound rather academic. I'll keep this thread in mind next time I want to run free 10,000 times. :wink:
Most people will run this far more often if they choose to use it.

For example:

I have it running in my dwm bar. It refreshes every second. I want to make sure that when it does refresh, it's making as little an interruption as possible to the system; almost like an impreceptable blip. After 24 hrs, it will have exceeded 86400 'blips' (24 x 60 x 60).

Now, there are people that love their 'blips', that's why they created Kde and Gnome... :P
Head_on_a_Stick wrote:
debiman wrote:free does essentially the same:
The information is gathered by parsing /proc/meminfo.
...only it is likely coded in C specifically for that purpose.
that is most likely more efficient
Thank you so much for giving it a test too! This made my day! :D
I made a slightly modified version of the OP's script to aid testing:

Code: Select all

alpine:~$ cat ~/bin/memtest
#!/bin/sh
count=0
while [ $count -le 10000 ]; do
   case "$1" in
      "free") free -m | awk '/^Mem/ {print $3}' > /dev/null;;
      "awk") awk '/MemTotal/ {total=$2} /MemFree/ {free=$2} /Buffers/ {buffers=$2} $1 ~ /^Cache/ {cached=$2} /SReclaimable/ {reclaim=$2} /Shmem:/ {shmem=$2} END {printf "%.1f", ((total - free) - (buffers + cached)) / 1024}' /proc/meminfo > /dev/null;;
   esac
   count=$((count+1))
done
alpine:~$
This accepts either "free" or "awk" as an argument to the `memtest` command and will run the appropriate backend accordingly.
Nice. I'll be using it for further studies. :)
On my Alpine Linux system we see:
With htop showing ~50%
I've seen 'top' being used around the net for this very purpose, and it makes me shudder. It was the first solution I saw and I refused to even try it. I just knew... :| But, it is great for something to compare to. Just seems like it has a lot to load up compared to 'free' and others.

The `free` version ran at ~40% CPU (in the guest) with `awk` holding to ~28%

WTF is wrong with Debian's procps? :?
The free command itself is just fine, I gather. I believe that the apps are built to work on all computers and solve many bugs. This all takes additional code and junk that most of us don't need but can come in handy. I'm betting you'd see quite a runaround in the code. I'm actually surprised that it did so well, considering.
debiman wrote:
bedtime wrote:free -m | awk '/^Mem/ {print $3}'
Conclusion:

The 'free -m' command took 2.02 times longer to perfrom the same task and used 34.6% more cpu whilst doing so.
wrong.
what you timed is 2 commands: free piped into awk - and you compare this to a pure awk script.
compare just 'free -m' to your awk script, you'll get different results.

i do understand that the scripts output is what you want, though.
Yes, if free -m could do all this itself, then I would not need to use awk with it; but that is not really rational to expect such a thing.
Head_on_a_Stick wrote:
bedtime wrote:I would like to know the formula that the 'free -m' command uses
It's buried away in the code — how's your C?

https://gitlab.com/procps-ng/procps/raw/master/free.c
:mrgreen:
I've dabbled in C++ for a time, but it's really not my interest anymore. Would be nice to know, though. Or, to even perheps make a 'sound' formula with the data from /proc/meminfo. I almost feel like getting in touch with the developer, but I don't think it's an appropriate considering the reason.

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: How calculate used mem with /proc/meminfo?

#10 Post by Head_on_a_Stick »

bedtime wrote:
On my Alpine Linux system we see:
With htop showing ~50%
I've seen 'top' being used around the net for this very purpose, and it makes me shudder. It was the first solution I saw and I refused to even try it. I just knew... :| But, it is great for something to compare to. Just seems like it has a lot to load up compared to 'free' and others
Ah no, you misunderstand me — I was using htop to show the average CPU usage while running the tests.
The free command itself is just fine, I gather
Yes indeed but my point was why is the free command faster than your awk script under Alpine Linux?

Your script is clearly faster in Debian (as seen in my VM) but why?

@debiman has an Arch box so perhaps if that also has a slow `free` command then it may be down to the libc versions (Alpine has musl, Debian has GNU).
I have it running in my dwm bar. It refreshes every second. I want to make sure that when it does refresh, it's making as little an interruption as possible to the system
If you want to use a script to feed dwm's status bar then be sure to call the loop with a /bin/sh shebang rather than bash or mksh (dash is lighter than either).

Here's my startup script:

Code: Select all

alpine:~$ cat .xinitrc
#!/bin/sh
setxkbmap gb
feh --bg-center ~/Pictures/dwm.png
urxvtd -q -f -o
while sleep 1;do xsetroot -name "$(cat /sys/class/power_supply/BAT0/capacity)%  •  $(date +'%F  •  %T')";done&
exec dwm
alpine:~$
If #!/bin/sh is removed then my default shell (loksh) is used for the while loop and it is (slightly) heavier than the POSIX shell (/bin/sh is linked to busybox in Alpine).

For an even lighter solution, try slstatus instead:

https://github.com/drkhsh/slstatus

This is coded in pure C (much better than silly old C++) and allows the desired modules to be loaded as required, it is _very_ nice indeed :)
deadbang

User avatar
None1975
df -h | participant
df -h | participant
Posts: 1387
Joined: 2015-11-29 18:23
Location: Russia, Kaliningrad
Has thanked: 45 times
Been thanked: 64 times

Re: How calculate used mem with /proc/meminfo?

#11 Post by None1975 »

Head_on_a_Stick wrote:For an even lighter solution, try slstatus instead:https://github.com/drkhsh/slstatusThis is coded in pure C (much better than silly old C++) and allows the desired modules to be loaded as required, it is _very_ nice indeed :)
Yes. thumbs up for slstatus! It usesWM_NAME or stdin to fill the status bar.
OS: Debian 12.4 Bookworm / DE: Enlightenment
Debian Wiki | DontBreakDebian, My config files on github

Post Reply