Login or Register to Ask a Question and Join Our Community


Linux

Underutilization of Linux memory

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems Linux Underutilization of Linux memory
# 1  
Old 02-16-2020
Underutilization of Linux memory
Hi All,

We are running a python application on an RHEL 7 VM machine hosted in Azure. Machine has 8GB of memory & 2GB of swap space configured as swap file. Below the output of free command from the server.

Code:
#-> free -h
              total        used        free      shared  buff/cache   available
Mem:           7.8G        3.4G        582M        381M        3.8G        3.6G
Swap:          2.0G        2.0G          0B

Here is the output of top command with list of top processing

Code:
top - 11:58:22 up 104 days,  2:40,  2 users,  load average: 1.03, 0.78, 0.91
Tasks: 205 total,   1 running, 203 sleeping,   0 stopped,   1 zombie
%Cpu(s): 24.5 us,  2.0 sy,  0.0 ni, 73.3 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem :  8157308 total,   605732 free,  3562428 used,  3989148 buff/cache
KiB Swap:  2097148 total,        0 free,  2097148 used.  3789188 avail Mem

  SWAP    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 97076  77330 root      20   0  728384  75128   2556 S   0.0  0.9 279:47.05 /app/xbot_venv/bin/python /app/xbot_worke+
 93040  77334 root      20   0  747624  92296   2556 S   0.0  1.1 348:13.00 /app/xbot_venv/bin/python /app/xbot_worke+
 90964  77318 root      20   0  732748  92008   2556 S   0.0  1.1 278:46.58 /app/xbot_venv/bin/python /app/xbot_worke+
 90916  77314 root      20   0  742220  48488   2556 S   0.0  0.6 157:59.16 /app/xbot_venv/bin/python /app/xbot_worke+
 90708  77312 root      20   0  801544 126112   2556 S   0.0  1.5 412:10.42 /app/xbot_venv/bin/python /app/xbot_worke+
 88384  77309 root      20   0  809980 177044   2556 S   0.0  2.2 546:48.45 /app/xbot_venv/bin/python /app/xbot_worke+
 85624  77319 root      20   0  739172  96352   2556 S   0.0  1.2 298:26.25 /app/xbot_venv/bin/python /app/xbot_worke+
 84780  77325 root      20   0  676940  47644   2556 S   0.0  0.6 157:53.64 /app/xbot_venv/bin/python /app/xbot_worke+
 83444  77306 root      20   0  735792  93004   2556 S   0.0  1.1 282:51.07 /app/xbot_venv/bin/python /app/xbot_worke+
 83120  77320 root      20   0  667980  42984   2556 S   0.0  0.5 157:30.00 /app/xbot_venv/bin/python /app/xbot_worke+
 81536  77322 root      20   0  730912  89452   2556 S   0.0  1.1 280:43.66 /app/xbot_venv/bin/python /app/xbot_worke+
 80504  77311 root      20   0  666956  39956   2556 S   0.0  0.5 156:57.16 /app/xbot_venv/bin/python /app/xbot_worke+
 79392  77323 root      20   0  666444  33728   2556 S   0.0  0.4 156:03.71 /app/xbot_venv/bin/python /app/xbot_worke+
 77804  77300 root      20   0  723180  92972   2556 S   0.0  1.1 610:43.01 /app/xbot_venv/bin/python /app/xbot_worke+
 76852  77305 root      20   0  813904 152728   2556 S   0.0  1.9 546:59.20 /app/xbot_venv/bin/python /app/xbot_worke+
 75864  77307 root      20   0  661824  41188   2556 S   0.0  0.5 157:21.67 /app/xbot_venv/bin/python /app/xbot_worke+

All of them are python processes eating up whole of swap space. Whats puzzling for me is that memory is under utilized and swap is 100% utilized. We are planning to increase the swap space to 8 GB to see if system resources(memory) are optimally utilized.

I would like to understand this behavior in depth. Why is swap space getting consumed first than memory. What needs to be done for making the system performance better?

Regards,
Veeresh

Moderator's Comments:
Mod Comment Please do wrap your codes and samples in CODE TAGS as per forum rules. After having 30+ posts it is expected from you to follow all Rules of community.

Thanks & Regards,
MODs team
Last edited by RavinderSingh13; 02-16-2020 at 03:01 AM..
# 2  
Old 02-16-2020
Dear Veeresh,

Not answering your question directly, but speaking honestly......

Swap space, especially on Linux systems in the year 2020 is a "relic of the past" (kind of like vacuum tubes before the transistor was invented) .

Why?

Because years ago, computer RAM was very expensive, so the best alternative was disk swap.

However, that equation changed many years ago; RAM became so cheap, most people (including me) never use swap because we have enough RAM; and if the machine started or "needed" to really "swap" that would be an "indicator" to upgrade RAM, because RAM is so cheap. We do not want our systems to swap, ever.

Not knowing the full details of your application, except that it is a python application running in a VM, I suggest you turn off swap and see what happens.

This is easy enough on Linux:

Code:
swapoff -a

But before you do this, it is a good idea to review the man page... here is one for Suse, but should be the same (or very similar) for RedHat:

https://www.unix.com/man-page/suse/8/swapoff/

https://www.unix.com/man-page/suse/8/swapoff/
This User Gave Thanks to Neo For This Post:
veeresh_15
# 3  
Old 02-16-2020
Let me predict what will happen without swap:
the machine will have spikes of high load, and during the spikes will pause its productive work.
Perhaps it will even invoke the OOM process killer.

My observation: a Linux kernel needs (little) swap to smoothly handle certain workloads.
Because it never frees memory without "pressure", and because of the "unused RAM is wasted RAM" philosophy.
So, for safety have a few GB of swap! Even if the kernel.org developers plus a million followers swear these situations do not exist.

In comparison, the Solaris kernel tries to free memory as a background task, in order to avoid a sudden memory pressure. Solaris can run without swap.
However, this was obscured by ZFS, that was developed with a "only free cache under pressure" philosophy. So in Solaris with ZFS either limit the ZFS cache or have some swap!
# 4  
Old 02-16-2020
Every system is different depending on the application

MadeInGermany does not have enough information to make a prediction.... accurate predictions require more knowledge than just a "guess" without system knowledge.

No one can make a prediction based on the very little information provided in the original post ( a python app running in a vm.... is very little information..... really)

In fact, I will illustrate this by turning off swap for a day on two different Linux Ubuntu server, one in a VM and one dedicated hardware (see below)
# 5  
Old 02-16-2020
For example...

Can I run without swap? Is further tuning possible?

Quote:
Systems without swap can make sense and are supported by Red Hat - just be sure the behaviour of such a system under memory pressure is what you want. In most environments, a bit of swap makes sense.
REF:

Do we really need swap on modern systems?

It should be clear to anyone, that if you have enough memory (and memory is very cheap) that you do not need swap on a modern Linux system. This is fairly well documented; although some swap is recommended.

There is no harm in the original poster turning off swap and observing themselves, what happens.
# 6  
Old 02-16-2020
To further illustrate this point, I have a Linux (Ubuntu) server running (in VM mode) only MQTT services and Node-RED (for the most part), and I just did this on a real, live server in production:

Code:
root@localhost:~# swapoff -a

I am also graphing the performance, which currently looks like this:

Underutilization of Linux memory-screen-shot-2020-02-16-71101-pmjpg


I will post the same graph in one day to compare.

Then, after that, I will do the same for an even busier Linux server (Ubuntu, dedicated server, running many web services and sites, MQTT, backups, and more)..... to illustrate.

After turning off swap on this system:

Code:
Tasks:  93 total,   1 running,  52 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.7 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  2041224 total,   229536 free,   403428 used,  1408260 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  1457936 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                                                                            
25673 mosquit+  10 -10   48028   5708   5044 S  0.3  0.3  19:05.65 mosquitto                                                                                                          
29531 node-red   5 -15 1364496 177400  23296 S  0.3  8.7  66:17.99 node-red                                                                                                           
    1 root      20   0  225412   9208   6704 S  0.0  0.5   2:34.75 systemd   

....
...
...

24 hours from now I will post the graphs and update on this VM Ubuntu instance.

Nothing but the facts .... no guesses, no opinions, and no personal preferences in this test. Just "turn off swap" and show the results at the same time, a day later.

Code:
ubuntu_vm# uptime
 13:15:58 up 116 days, 11:56,  1 user,  load average: 0.00, 0.00, 0.00

# 7  
Old 02-16-2020
And tomorrow, I will take this dedicated Ubuntu server, currently looks like this (runs MQTT, mysql, apache2 and a number of web sites, does backups, etc):

Code:
top - 06:40:25 up 35 days, 6 min,  1 user,  load average: 0.05, 0.12, 0.16
Tasks: 224 total,   1 running, 223 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.5 us,  0.9 sy,  0.0 ni, 97.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32652952 total, 16986476 free,   694036 used, 14972440 buff/cache
KiB Swap:  7996412 total,  7996412 free,        0 used. 31071156 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                           
  950 mysql     20   0 2749912 333616  19552 S   0.7  1.0 542:11.32 mysqld                                                                                                            
 8346 www-data  20   0  479644  32704  21104 S   0.7  0.1   0:00.02 apache2                                                                                                           
 8353 www-data  20   0  477684  24696  13064 S   0.7  0.1   0:00.02 apache2                                                                                                           
  901 www-data  20   0  343920  25120   2356 S   0.3  0.1  73:38.94 memcached                                                                                                         
 7461 www-data  20   0  477100  15708   4556 S   0.3  0.0   0:00.05 apache2                                                                                                           
 7607 www-data  20   0  477100  15708   4556 S   0.3  0.0   0:00.03 apache2                                                                                                           
 8205 www-data  20   0  477100  15708   4556 S   0.3  0.0   0:00.01 apache2                                                                                                           
    1 root      20   0  185144   5792   4040 S   0.0  0.0   2:55.18 systemd       
..
...
...
...

Will do the same... will turn off swap complete for a day and compare before and after.

In this manner, everyone can see two very different real-world, live Linux server examples of running with swap on and off; in both cases, there is plenty of memory. In fact, the server in this post (the dedicated server) shows zero swap being used even though the uptime is:

Code:
ubuntu# uptime
 06:44:46 up 35 days, 11 min,  1 user,  load average: 0.31, 0.22, 0.19
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

Shared memory in linux

Hello, I am using Linux os. $ df -k /dev/shm Filesystem 1K-blocks Used Available Use% Mounted on tmpfs 2023256 1065000 958256 53% /dev/shm $ Based on my google this, it is shared memory. What is this shared memory and where exactly it is used? Can you... (5 Replies)
Discussion started by: govindts
5 Replies

2. Red Hat

about shared memory space in Linux !

Dear Friends , I am confused about shared memory in Linux . plz look @ the below output : $ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 126G 61G 59G 52% / tmpfs 1.7G 0 1.7G 0% /dev/shm Here , I see that every time... (1 Reply)
Discussion started by: shipon_97
1 Replies

3. Linux

Memory Usage on Linux server

Hi , As I am new to Linux server, I am facing some doubts like: On linux server virtual memory usage goes to 99%, but I have Threshold limit of 95%. So it crossed the threshold limit and alarmd. Yesterday I moniterd the server using TOP command, and found some of Tibco process was consuming... (4 Replies)
Discussion started by: Jaywantmm
4 Replies

4. Programming

Memory sniffing in linux

I am trying to create an application that will be able to sniff memory of other applications. I am not completely new to systems programming but I am not sure how to go about this task. I understand that accomplishing this mainly require these steps. 1: Get a list of processes 2: Find the... (2 Replies)
Discussion started by: mosey
2 Replies

5. Linux

Linux Memory Track

Hi All, We are using the linux servers and need to track the memory utilization of the box. Could anyone advice how the same can be achived. :) (1 Reply)
Discussion started by: haitorajesh
1 Replies

6. Red Hat

share memory on linux

how to list the orphaned shared memory? how to kill them so that shared mem is free again. thanks (9 Replies)
Discussion started by: melanie_pfefer
9 Replies

7. UNIX for Advanced & Expert Users

Memory managment - linux

Hi, I having problem with my linux machine it have 6Gb physical memory and somehow it always almost coming to the bottom neck and than it start writing to the swap memory you can see that there is more than 4G in cahce, is there any way to clean the cache or to limit it to 2Gb? host1... (6 Replies)
Discussion started by: Igal Malka
6 Replies

8. Red Hat

Linux memory usage

What's the best way to find out how much memory is being used/available? I tried using free, but I didn't quite understand the output. Can someone explain it? $ free total used free shared buffers cached Mem: 16304536 16256376 48160 0 ... (6 Replies)
Discussion started by: junkmail426
6 Replies

9. Programming

Determining free(available) memory in MV linux

HI I'm a rookie in C programming and I'm working in Monta Vista Linux. I have to write a program that displays free memory. I have memtester(allready written by someone else) and now I have to type how much amount of memory tester will test and I want that memtester finds out himself how much of... (11 Replies)
Discussion started by: trancedeejay
11 Replies

10. Programming

memory layout in C on linux

Hi, Does any one know what tool to use to visualize how is memory layed out for C on linux systems. I mean how much stack portion is used in functional call. Where exactly does the argument to function sit in memory ? I have written small program pasted below. But I am not able to infer... (3 Replies)
Discussion started by: parasa
3 Replies
Login or Register to Ask a Question