Sleuthing CPU/load/memory increases

[MarmiteSandwich]

Hi,
I have a Raspberry Pi which runs 24/7 as a music server and it periodically becomes unexpectedly unresponsive and/or crashes completely. OMV provides historic graphics of cpu, load and memory (via RRDtool?).

LoadWeek.png (37.23 KiB) Viewed 785 times

MemoryWeek.png (23.85 KiB) Viewed 785 times

.

These graphics suggest that something is leaking memory or processes or something. There are also various logs available in OMV e.g.

, but I don't know where to start with finding the culprit process/service. I use winscp and putty to bypass OMV if necessary. Any suggestions welcome.
Marmite

Raspberry Pi 3B
ARMv7 Processor rev 4 (v7l)
Linux 5.10.103-v7+
OMV 6.9.14-1 (Shaitan)
Java (11?)
MinimServer 2.2
BubbleUPnP Server 0.9-update49

[wizard10000]

I think the first thing you'd need is a baseline from a clean boot. My suggestion:

Reboot the machine, give it like five minutes to settle down and then dump top's output to a text file like this:

Code: Select all

top -o %MEM > ~/top.txt

This will sort applications by RAM use and stick the result in your home directory.

Then give it a couple days or more, close all applications your user opened and dump again to a different text file:

Code: Select all

top -o %MEM > ~/top2.txt

Compare the two text files and it should tell you which application(s) may be leaking.

Dumping top to a text file is mostly human-readable and is more accurate than using ps. ps will tell you how much RAM your app *could* use, top will tell you how much your application *is* using.

[MarmiteSandwich]

great - gonna give it a try

[Aki]

@MarmiteSandwich:
Have you identified the affected programs ?

[MarmiteSandwich]

I set up a scheduled task in OMV which performs a top at 3am every day and adds the output to a.txt file. There are 2 java based applications running permanently (bubbleupnp server and Minimserver), both of which seem to be gradually increasing their memory usage every day. Was going to leave it a few more days and then check with the developers and Google for memory issues with Java. Could post the results if you are interested.
Marmite.

[Aki]

Could post the results if you are interested.

Yes, thanks.

[MarmiteSandwich]

Herewith 5 days of top output (top.txt). Day 1 is 17:00 when it was started, the remaining days are 03:00.

Using a bit of jiggery pokery in basic and excel I produced the pdf output of the development over time (topout.pdf).

The line called bubbleu+ is just the one application, whereas the line "pi" is mainly minimserver in java plus some other stuff.

Strange thing about this analysis is that the total memory seems to be going down, while the 2 suspect programs are steadily going up. Most of the time of this sample I have been away from home, so the 2 main programs have been pretty much dormant.

Interested in anything you can read into these.
Regards,
Marmite

[Aki]

Hello,

The two java program are consuming more and more ram.

You can set a memory limit for each program (or adjust the current limit) [1] to avoid saturating resources (if there's a risk of that happening). Also, the JVM should trigger garbage collection more often with tighter memory limits.

Hope this helps.

[1] https://stackoverflow.com/questions/149 ... ge-for-jvm
--
note: moving to "Off-Topic" sub-forum because the topic is about a derivative distribution.

[MarmiteSandwich]

Thanks for the suggestions.
Marmite