Linux

Thanks Neo for your comments and taking out time to perform those tests. I will also try checking my system's performance by turning off swap.
But if you or anyone else can help me understanding my current system behaviour, it will help me understand Linux Memory management better.
As part of my python application, we are making some rest api calls to Azure and using various azure services. Please let me know what other information i can provide for commenting on this behaviour?

Regards,
Veeresham

What value do you have for swappiness:

Code:

# cat /proc/sys/vm/swappiness
60

You can try tweaking this back a lower percentage (eg add/change vm.swappyness=10 in /etc/sysctl.conf and reboot) will cause more pages to be dropped from cache in preference to using swap.

Dear Veeresham,

I think you may have completely missed the point in my first post. MadeInGermany obviously did as well.

Let's take the price of this RAM on AliExpress:

Samsung server memory DDR3 8GB 16GB 1333MHz 1600MHz 1866 ECC REG DDR3 PC3-12800R Register DIMM RAM 240pin 12800 8G 2RX4 X58 X79

In the link above, the price of server RAM is about $20 USD for 8 GB.

Now, I am not sure how much time you have spent working on this issue and how much you think your time is worth to you; but I can tell you that already I have spend time and effort on answering your question for you with a value of at least $40 USD, including replying to you and others, doing some quick Internet searches, and logging into a remote system and turning off swap (to illustrate a point for the "naysayers" who remain in love with swap from the yesterdays of expensive RAM nostalgia).

Let's say that the time you and others have spent on the problem (on your end) is also work about $40 USD. This means that $80 USD total human capital has already been spent on this problem.

$80 USD divided by $20 USD turns out to be roughly the same in terms of resources, as purchasing 32 GB of Samsung server RAM for your application. Of course, this is different in a VM service, but that is a different topic; but the same idea applies (just differently, and I tend NOT to use VMs and cloud services for security, and command and control reasons)

Let's say you try tweaking swappiness (which I have done many times before on Linux systems) and playing around with more Linux system parameters (like cache clearing, refresh rates, etc) to try to further optimize. You are spending more resources, all of which could be used to add RAM to you system, if you value your time (in the least). Only in an "academic" context, doing a study for the sake of learning (not operations) is all this optimization an efficient use of time and energy. You have indicated you are working on a real-world application and are not in academics.

This is my point.

RAM is very cheap (and it gets cheaper every year). If your system is swapping, that is an indicator, and a very strong indicator, that you need more RAM. Tweaking and tweaking your system parameters and encouraging others to engage in this kind of tweaking with you, is an inefficient use of human resources and capital. It's more "academic" in nature, versus practical, in operational terms.

In answering you in this reply, this reply is at least work 8 GB of server RAM in 2020; but let's be very conservative, and say all this thinking and typing and error checking (my in-browser spell checker seems to be in a plot against me, LOL) is only worth 4GB of RAM, either way; you are talking about your system which you say has 8GB of RAM installed. That RAM is worth about $20 USD, today (in hardware terms, not inflated, artificial VM terms).

If you want better performance, go out and spend $20 more USD and buy 8GB more and you will have 16GB of RAM.

This is a better use of your time and your energy. Your application will run faster and better. The world will be a better place.

The reason I have turned off swap on one server (yesterday), and am posting the results tonight in this thread, is to prove a simple point to others who are "swap nostalgic". This is not "a test" for me. I have been working with Linux since before Slackware 0.8, which was released before 1993. This translates to using Linux for around 27 years, continuously. I know the results before I turn off swap. There will be no change in system performance because the server and the applications I am running on that server are (1) not constrained by RAM and (2) there are no memory leaks. Linux runs well and I always configure all my Linux systems to have plenty of RAM and upgrade if RAM becomes an issue.

There are a lot of people who like to talk about the virtue of swap. This is like saying that when the glass of milk is full, I will go to the shelf and get another glass to store my milk until the main glass is less full, as I pour into the main glass. While this might be good if you are pouring milk very fast into your glass; most people would get a bigger glass.

The same in true for computer RAM.

RAM is so cheap these days (and has been for many years), you are better off getting more RAM versus spending a lot of time and energy trying to tweak a system to squeeze more performance from your system, which is asking you for more RAM because it is starting to swap.

You asked for ideas; so here is mine, put very simply. I would never run a Linux server application with 8GB of RAM and 2GB of swap because I do not want my systems to swap, ever, for any reason. I run 64 GB RAM on my desktop. I only use 8 GB VMs for "playgrounds", not production apps. If I built an app in a 8GB sandbox and it then became memory constrained, I would move it to a more robust server or upgrade it (if in VM, but I tend to move real-world apps to dedicated hardware, but that is just me because I like having more control over the OS and the upgrade cycles, etc). This is because time is worth more than money. We cannot gain back time. The arrow of time runs only in one direction (in this universe we are in now).

What I do is to always make sure my computer systems, all of them, remote servers, desktops all of them; have more than enough RAM to do the job because RAM is so cheap and I have better things to do with my time, which I consider valuable and a limited resource which I have little control.

Tonight I will post the graph of my MQTT / Node-RED server with swap off for 24 hours. You will see that there is no issue at all because there is enough RAM. If your system is memory constrained and you turn off swap, you will more-than-likely have some performance issues; but my point is not "turn off swap".. my point is "upgrade and add more RAM". For me "swap" is a "big red, flashing LED" telling me "please upgrade".

Now, in the VM case, RAM is a bit more expensive because when running in a VM and we are paying more for the RAM; but that is an entirely different discussion (that is a business issue, not a technology issue); but the point is still the same.

If you system is swapping, install more RAM.

You don't need to turn swap on and off. I was trying to make a point. If your system is memory bound (constrained) it will perform poorly even if you use swap; although it will perform better with swap than without swap (generally speaking) when you are really constrained by RAM. Like in my milk metaphor, it is OK to swap glasses when you are pouring milk to fast into a small glass; but it's even better (and smarter) to get a bigger glass.

The best choice, as I said in my first post, is to upgrade your RAM and move on to other tasks, unless (1) you are doing this as an academic exercise or (2) your time has very little value.

This is why I have replied this way. Time is an arrow which only moves in one directly. Use your time and energy wisely.

As promised, here is the chart of a Linux Ubuntu server 24 hours or so after turning off swap (it is still off).

As mentioned, on this server there is plenty of memory (for the applications) so there is zero issue having swap turned off:



I can continue this for a few of my "busy servers", some very busy, but all have plenty of RAM, so all will perform just fine with swap turned off (since they do not really use any swap with it turned on).

Anyway, I don't want to beat this dead horse any longer...

Computer RAM is cheap.... if your system is swapping, buy more RAM.

Cheers.

And here I come by thinking that swap nowadays was used to save whats in the memory (ram) on hibernate/suspend.

At least on 'personal' machines that are not used as servers.

My 2 cents, please correct me if I'm wrong on this.

You are not wrong that swap is used for hibernation in systems that are configured to use swap-based hibernation. But, I am confused by your reply, to be honest. This is not a question of "right and wrong". Not everyone hibernates, in fact, hibernation is a something some people choose to use and others choose not to hibernate. This is the same as any computer configuration. For example in Ubuntu, hibernation is not enabled by default; and as I recall, Ubuntu stopped supporting hibernation (in general) and only makes it available for users who want it.

Personally, I have zero systems (desktop or server) configured to hibernate. That is my choice (right?); others may like hibernation. I do not. The original poster did not ask a question about hibernation, to be honest; and it is very likely the OP does not use hibernation, since on Linux systems (at least the ones I am familiar with, like Ubuntu), hibernation is not enabled by default.

The title of this thread by the OP is "Underutilization of Linux memory" not "How to Enable Hibernation in Linux", isn't that correct?

As for me, I have a number of very busy Ubuntu servers in production and I do not enable hibernation on a single one of them.

My reply early discusses how RAM in 2020 is so cheap, that when you have system swapping for performance reasons (not for those people who configuration their systems to hibernate), that is a sign that you should invest in more RAM.

That is my choice and this is how I always advice others to admin their systems after over 25 years of hands on Linux work since Slackware 0.8. I like lots of deep, cheap RAM and never hibernate.

As a side, I just checked and Slackware 1.0 was released in 1993. I started deploying Linux for the United States Air Force at 27 Air Combat Command bases worldwide as Slackware 0.8, so that was at least 27 years ago. Perhaps over 27 years of using Linux in production has informed me "just a little" on how to configure systems.

Code:

https://en.wikipedia.org/wiki/Slackware

Everyone has a choice. You can build your system with plenty of "deep, cheap" RAM or not. Maybe you like having you systems move bits to disk and like to be "frugal" with RAM, LOL. However, I prefer RAM, it is CHEAP; and I never enable hibernation on any system. That is my choice and my advise to most people. If you system is swapping for performance reasons (this is not a hibernation discussion) it is a blinking light to deploy more RAM.

So, If anyone wants to use swap and you like having less RAM and you like your system thrashing pushing bits from RAM and disk and back, and that is your idea of "good computing", then go for it. I am not telling or advising anyone to use or not to use swap or to enable nor disable it. Everyone has a choice, right? I have posted what I do, and I have been working with Linux on a daily basic in production since Slackware 0.8, for over two decades (nearly three decades).

This means, quite simply, have told you that for me, I find RAM cheap and when a system is swapping, that is generally a very clear sign for me to increase RAM (this is not a discussion about hibernation, and never was, to be frank). However, on all Linux systems I have deployed for the past 10 years, I deploy them with a lot of RAM, so they never need to swap. None of my systems and servers swap, and none of my systems and server hibernate.

If anyone wants to swap and hibernate, then you should do so. Everyone has a choice when they configure their system. Many people advise based on "theory". I am telling you what I do based on 27 years of hands on production experience with Linux servers deployed. Most of my systems tend to have a lot of RAM. If I have a Linux system with a small amount of RAM deployed (like a VM slice), and it gets busy because the app grows and it starts to swap, I will deploy more RAM. I never configure hibernation.

As a final thought (hopefully), I can recall many people here at unix.com who were not happy when I gave Linux and Unix "equal status" at unix.com many years ago. There were a lot of "hard core" UNIX users (back then) who were really negative about Linux (and they did not use Linux back then either). I find it interesting that, now, so many of those same Unix-diehards of "yester-years" are now Linux experts today.

There is a big difference between "computing in theory" and "operational system deployment, administration and maintenance" (real-life operations).

A lot of people "talk to walk" but very few people actually "walk the talk". This is especially true today, in my view. We have a lot of "arm chair" experts in computer science and IT security (bloggers, writers, YT video makers), who have never really ran an operational servers in a busy production environment, fought hackers and DDOS attacks "in the real world" or been forced to make data center change in a day or two when a hosting business goes belly up.

There is a huge different between "theory" and "practice" and in server operations, the same is true, for sure! The world today is "pregnant" with arm-chair IT experts, bloggers, commenters, influencers, YT video makers, forum posters, who have never actually ran a production server application.

Moderator's Comments:

Thread closed since it is moving far off topic. This was not a discussion about hibernation when it started. If you want to discuss hibernation, please start a new thread.