Help with RAM Usage on debian Server

[Paul6552]

Hello community,

I have a debian server with docker and java installed.

docker 3 containers:

• swag (Linuxserver container for nginx (webserver) and ssl-certificate)
• postgresql
• keycloak (Authentification-Management)

2 Java-Programs:
I start these programs with these linux commands:
export JAVA_OPTS="-Duser.timezone=UTC"
./java/jdk/bin/java $JAVA_OPTS -Xmx10g -jar portal-backend-1.0-SNAPSHOT-runner.jar 2>&1 >> log/portal.log &
and
export JAVA_OPTS="-Duser.timezone=UTC"
./java/jdk/bin/java $JAVA_OPTS -Xmx10g -jar pea-backend-2.0.0-SNAPSHOT-runner.jar 2>&1 >> log/pea.log &

These 2 programs are heavy calculation programs written in hibernate, quarkus ...

Server has 16GB RAM and I am running out of RAM every (around) 6 hours. Then I have to rebot the server and the next 6 hours start.

Sadly I do not really understand the linux command -Xmx10g. Anything with Virtual (or not) Ram usage.

Attached you can find a picture from htop. Maybe some more experienced linux server can help me. Is the problem the Xmx10g command or do I have a too small server or or or
Any help appreciated

 

[CaffeineAddict]

Server has 16GB RAM and I am running out of RAM every (around) 6 hours.

Hello and welcome to Linux.org!
What you're most likely experiencing is memory leak:

a memory leak is a type of resource leak that occurs when a computer program incorrectly manages memory allocations[1] in a way that memory which is no longer needed is not released.

Memory leak - Wikipedia

en.wikipedia.org

I suggest you investigate which software is responsible and then either report a bug to developers or find an alternative.

EDIT:
@Paul6552
If you look at htop you'll notice that memory bar is mostly in orange color, orange color indicates cache which suggest the software requires large amounts of cache for performance reasons.

Therefore if it's not memory leak it's possible to solve it by creating a swap partition to store the cache there and release memory for computation, for 16GB memory it's suggested to have 24GB swap partition.

You currently have no swap partition.

 

[Paul6552]

thank you for your fast response.

2 more questions:
1. Can the problem also be the linux command -Xmx10g? I read in the intenet about it. I took over this server but for me the 10g is also not corret. There are two Java programs that each run with 10g. But the server has only 16GB RAM. Still possible?
2. Okay Swap partition would be great. No idea if I have one or not but I am believing you :-D Can I create the swap partition during the server and programs are running?
I found these tutorial how to create a swap partition. Thanks for the 24GB suggestion

Swap Partition: What It Is & How to Create and Manage It

This article explains what a swap partition is, as well as how to create and manage a swap partition in Linux.

phoenixnap.com

 

[CaffeineAddict]

Can the problem also be the linux command -Xmx10g?

Yes, the option Xmx from what I see is the option to specify heap size
Heap memory is memory that the program dynamically allocates during runtime, ex. not stack memory which is very limited per process and only temporary.

In this case the program is allowed to reserve 10GB of heap memory, which is somewhere exactly what's shown in htop, therefore you could reduce that too to a lower value.
This might result is the program having to perform the task longer.

There are two Java programs that each run with 10g. But the server has only 16GB RAM. Still possible?

Yeah, therefore for 2 programs where each requires 10GB of heap + memory required by system and other processes you would need somewhere 25GB of memory but have only 16GB.

Can I create the swap partition during the server and programs are running?

Not sure but nothing bad would happen if you make it and then see if cache gets stored to swap.
It depends on those programs whether they'll detect swap during runtime. if not you'll need to restart them.

 

[Paul6552]

Ok. Many thanks.

The next time the server is struggeling I will first lower the heap memory and in the next maintenance window I will try to create a swap partition.

Not sure if nothing bad would happen X) because I have to reformat the current disk, which is heavely used. I read in the tutorial that is is possible to create a swap partition beside the primary partition, so that I dont have to touch the primary partition but I am going to read the tutorial a bit more.

Thank you a lot for the tipps and especially for the swap partition

 

[CaffeineAddict]

Not sure if nothing bad would happen X) because I have to reformat the current disk

There is no need to reformat entire disk, if there is no free unused disk space then existing partition (probably root) needs to be shrinked to make some space.
But I'm not sure if that's possible while system is running on root partition, if not then it needs to be done from live USB which can lead to data loss if not done properly.

If you're using KDE then I'd suggest using KDE partition manager which has UI.
you can also use fdisk from CLI but be careful not to delete what you don't want to.

 

[dos2unix]

Server has 16GB RAM and I am running out of RAM every (around) 6 hours. Then I have to rebot the server and the next 6 hours start.

Sadly I do not really understand the linux command -Xmx10g. Anything with Virtual (or not) Ram usage.

The -Xmx10g, mean the Java heap size can be up to 10GB. It's "usually" recommended not to goa bove 50% of the RAM on heap size. In your case, that would be 8GB. You can change this line.

By default docker apps don't use compressed RAM. That means if you're using say 4GB of RAM in container A, and 8 GB of RAM in container B, that only leaves 4GB of RAM for your host OS.

 

[Paul6552]

KDE would be great, but I have only ssh access. Is it possible to install KDE on my local linux machine (linux mint) and connect via SSH to the debian server and format the partition over ssh?

@dos2unix
When the recommended way is to use 50% of RAM. Is it meant in sum? Because I have two Java Programs. That means every java program can only have 4GB of java heap size according to your description. Or can every java program have 50% of the total Ram. So 8GB?

//EDIT
Sry, after thinking about it. It is 4Gb for every java program. Sry for the stupid question

 

[CaffeineAddict]

In your case, that would be 8GB. You can change this line

There are btw. 2 programs each reserving 10GB of heap, so each should be really reserving max 5GB

Is it possible to install KDE on my local linux machine (linux mint) and connect via SSH to the debian server and format the partition over ssh?

You can't use locally installed partition manager to partition remote disk.
You're pretty much limited to fdisk trough SSH

But first tell, does the Debian server use LVM or is it installed to directly to disk?
It's not straightforward to shrink root partition btw. see link below:

How to shrink root filesystem without booting a livecd

I find myself needing to rearrange a system's partitions to move data previously under the root filesystem into dedicated mount points. The volumes are all in LVM, so this is relatively easy: creat...

unix.stackexchange.com

 

[Paul6552]

Took me a while to figure out if I was using LVM. But no.

I have a root server from netcup more precisely: Netcup root server

So I will stay with fdisk. Thanks so far. Once I've read up more, I'll probably come back to discuss my fdisk settings. Thank you very much

 

[KGIII]

Instead of a swap partition, OP can probably get away with a swapfile. Yes? No?

In my experience, the two are functionally the same and I get the same results.

 

[Paul6552]

Hello everyone, here my experience so far:
1. Approach
I changed -Xmx to 4g without SwapFile. I got a lot of Java error out of memory exceptions (in the java application) and in the server log I found java out of memory exceptions heaps space error
2. Approach
I created a Swap FILE (Thanks for the tip). Restarted the server, started java applications again with Xmx4g but still many Memory exceptions.
3. Approach
I changed the java applications back to -Xmx10g and every queries worked, but when I check memory ... with htop I get the following result (pls see attached the picture). I can see the swap file, but I have no idea why it isn`t used. It stays with 0K used. Now I am again afraid that the server is going down after a few hours x)

How I created the swap-File:
--------------------------------------

• fallocate -l 24G /swapfile
• chown root:root /swapfile
• chmod 0600 /swapfile

Format swapfile:

• mkswap /swapfile

Activate swapfile:

• swapon /swapfile

In der Datei "/etc/fstab" hinzufügen:
/swapfile swap swap defaults 0 0
Status:

• swapon -s

When I type "swapon -s" I get the following status:
Filename Type Size Used Priority
/swapfile file 25165820 0 -2

//EDIT
One question more:
If I leave the settings as "-Xmx10g" and the server runs out of physical memory, does it access the SwapFile after that or is the -Xmx command only for physical memory?