We are running a python application on an RHEL 7 VM machine hosted in Azure. Machine has 8GB of memory & 2GB of swap space configured as swap file. Below the output of free command from the server.
Here is the output of top command with list of top processing
All of them are python processes eating up whole of swap space. Whats puzzling for me is that memory is under utilized and swap is 100% utilized. We are planning to increase the swap space to 8 GB to see if system resources(memory) are optimally utilized.
I would like to understand this behavior in depth. Why is swap space getting consumed first than memory. What needs to be done for making the system performance better?
Regards,
Veeresh
Moderator's Comments:
Please do wrap your codes and samples in CODE TAGS as per forum rules. After having 30+ posts it is expected from you to follow all Rules of community.
Thanks & Regards,
MODs team
Last edited by RavinderSingh13; 02-16-2020 at 03:01 AM..
Location: Asia Pacific, Cyberspace, in the Dark Dystopia
Posts: 19,118
Thanks Given: 2,351
Thanked 3,359 Times in 1,878 Posts
Dear Veeresh,
Not answering your question directly, but speaking honestly......
Swap space, especially on Linux systems in the year 2020 is a "relic of the past" (kind of like vacuum tubes before the transistor was invented) .
Why?
Because years ago, computer RAM was very expensive, so the best alternative was disk swap.
However, that equation changed many years ago; RAM became so cheap, most people (including me) never use swap because we have enough RAM; and if the machine started or "needed" to really "swap" that would be an "indicator" to upgrade RAM, because RAM is so cheap. We do not want our systems to swap, ever.
Not knowing the full details of your application, except that it is a python application running in a VM, I suggest you turn off swap and see what happens.
This is easy enough on Linux:
But before you do this, it is a good idea to review the man page... here is one for Suse, but should be the same (or very similar) for RedHat:
Let me predict what will happen without swap:
the machine will have spikes of high load, and during the spikes will pause its productive work.
Perhaps it will even invoke the OOM process killer.
My observation: a Linux kernel needs (little) swap to smoothly handle certain workloads.
Because it never frees memory without "pressure", and because of the "unused RAM is wasted RAM" philosophy.
So, for safety have a few GB of swap! Even if the kernel.org developers plus a million followers swear these situations do not exist.
In comparison, the Solaris kernel tries to free memory as a background task, in order to avoid a sudden memory pressure. Solaris can run without swap.
However, this was obscured by ZFS, that was developed with a "only free cache under pressure" philosophy. So in Solaris with ZFS either limit the ZFS cache or have some swap!
Location: Asia Pacific, Cyberspace, in the Dark Dystopia
Posts: 19,118
Thanks Given: 2,351
Thanked 3,359 Times in 1,878 Posts
Every system is different depending on the application
MadeInGermany does not have enough information to make a prediction.... accurate predictions require more knowledge than just a "guess" without system knowledge.
No one can make a prediction based on the very little information provided in the original post ( a python app running in a vm.... is very little information..... really)
In fact, I will illustrate this by turning off swap for a day on two different Linux Ubuntu server, one in a VM and one dedicated hardware (see below)
Location: Asia Pacific, Cyberspace, in the Dark Dystopia
Posts: 19,118
Thanks Given: 2,351
Thanked 3,359 Times in 1,878 Posts
For example...
Can I run without swap? Is further tuning possible?
Quote:
Systems without swap can make sense and are supported by Red Hat - just be sure the behaviour of such a system under memory pressure is what you want. In most environments, a bit of swap makes sense.
It should be clear to anyone, that if you have enough memory (and memory is very cheap) that you do not need swap on a modern Linux system. This is fairly well documented; although some swap is recommended.
There is no harm in the original poster turning off swap and observing themselves, what happens.
Location: Asia Pacific, Cyberspace, in the Dark Dystopia
Posts: 19,118
Thanks Given: 2,351
Thanked 3,359 Times in 1,878 Posts
To further illustrate this point, I have a Linux (Ubuntu) server running (in VM mode) only MQTT services and Node-RED (for the most part), and I just did this on a real, live server in production:
I am also graphing the performance, which currently looks like this:
I will post the same graph in one day to compare.
Then, after that, I will do the same for an even busier Linux server (Ubuntu, dedicated server, running many web services and sites, MQTT, backups, and more)..... to illustrate.
After turning off swap on this system:
24 hours from now I will post the graphs and update on this VM Ubuntu instance.
Nothing but the facts .... no guesses, no opinions, and no personal preferences in this test. Just "turn off swap" and show the results at the same time, a day later.
Location: Asia Pacific, Cyberspace, in the Dark Dystopia
Posts: 19,118
Thanks Given: 2,351
Thanked 3,359 Times in 1,878 Posts
And tomorrow, I will take this dedicated Ubuntu server, currently looks like this (runs MQTT, mysql, apache2 and a number of web sites, does backups, etc):
Will do the same... will turn off swap complete for a day and compare before and after.
In this manner, everyone can see two very different real-world, live Linux server examples of running with swap on and off; in both cases, there is plenty of memory. In fact, the server in this post (the dedicated server) shows zero swap being used even though the uptime is:
Hello, I am using Linux os.
$ df -k /dev/shm
Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 2023256 1065000 958256 53% /dev/shm
$
Based on my google this, it is shared memory. What is this shared memory and where exactly it is used? Can you... (5 Replies)
Dear Friends ,
I am confused about shared memory in Linux . plz look @ the below output :
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 126G 61G 59G 52% /
tmpfs 1.7G 0 1.7G 0% /dev/shm
Here , I see that every time... (1 Reply)
Hi ,
As I am new to Linux server, I am facing some doubts like: On linux server virtual memory usage goes to 99%, but I have Threshold limit of 95%. So it crossed the threshold limit and alarmd.
Yesterday I moniterd the server using TOP command, and found some of Tibco process was consuming... (4 Replies)
I am trying to create an application that will be able to sniff memory of other applications.
I am not completely new to systems programming but I am not sure how to go about this task. I understand that accomplishing this mainly require these steps.
1: Get a list of processes
2: Find the... (2 Replies)
Hi All,
We are using the linux servers and need to track the memory utilization of the box. Could anyone advice how the same can be achived.
:) (1 Reply)
Hi,
I having problem with my linux machine
it have 6Gb physical memory and somehow it always almost coming to the bottom neck and than it start writing to the swap memory
you can see that there is more than 4G in cahce, is there any way to clean the cache or to limit it to 2Gb?
host1... (6 Replies)
What's the best way to find out how much memory is being used/available? I tried using free, but I didn't quite understand the output. Can someone explain it?
$ free
total used free shared buffers cached
Mem: 16304536 16256376 48160 0 ... (6 Replies)
HI
I'm a rookie in C programming and I'm working in Monta Vista Linux. I have to write a program that displays free memory. I have memtester(allready written by someone else) and now I have to type how much amount of memory tester will test and I want that memtester finds out himself how much of... (11 Replies)
Hi,
Does any one know what tool to use to visualize how is memory layed out for C on linux systems. I mean how much stack portion is used in functional call.
Where exactly does the argument to function sit in memory ?
I have written small program pasted below. But I am not able to infer... (3 Replies)