大鸟上撒点盐,生活才能多姿多彩。

Swap usage is at 100% and need to find out what processes are using swap and reduce swap usage

Solution Verified – UpdatedJanuary 29 2017 at 9:04 PM-

Environment

  • Red Hat Enterprise Linux 4
  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • swap

Issue

  • Swap memory usage is at 100%, which is above the error threshold (50%). Please assist to find out the cause and suggest steps of how to decrease swap usage.

Resolution

Overview

Regarding full swap, multiple points should be taken into consideration:
0. On RHEL6, a bug which did lead to increased swap usage is resolved in kernel-2.6.32-504.el6 (RHSA-2014-1392) and later. Private RHBZ #949166 has details.
1. Increasing the system’s physical memory will make it less likely that swap has to be used
2. A further approach tries to find the process using most swap (using below script) and kill that process. This is not desirable in most situations.
3. The available swap space can be increased in creating a swapfile, this can be done without service downtime. Please refer to How do I add a swap file to my Red Hat Enterprise Linux system? for details.
4. If it is not desired that current processes have pages which are also in swap, thenswapoff -a, followed by aswapon -acan be performed.

Resolution 1: Increase system physical memory

In some cases, swap full issue causes thrashing(continuously heavy swap in/out activity). When sar file (the files are located in /var/log/sa,sysstatpackage must be installed andsysstatservice is working) exists in the system and sar file contains continuous pswpin and pswpout record. It indicates that the system does not have enough physical memory in current workload. The system requires additional system memory whose size is the size of current total swap size at least. Thrashing usually causes system performance degradation because thrashing cause heavy disk I/O. Recent systems with SSD might not show such system performance degradation.
Example of sar output

Raw

Resolution 2: Find process’s memory regions that are using the most swap

To find out what 10 process’s memory regions are using the most swap space, copy and paste these commands onto a bash command line terminal and then press <Enter> to execute it:

Raw

This will display the 10 process’s memory regions that were using the most swap since they were started, sorted in decreasing swap usage order. This script does not typically display total swap in use by a process. If you kill some of these processes that own the listed memory segments then you will find that swap usage will decrease. The processes killed may either be a direct contributor or a victim of the root cause of high swap usage. Note 1: If the above script produces no output, then it could be that none of the currently running processes in /proc/*/smaps are using swap. You can test that by simply running:

Raw

Important thing to keep in mind is that the aforementioned script will only show the active processes that have memory swapped at that point in time, meaning at the time when the script was run. It might be quite possible that the system has already swapped a chunk of memory and that is visible onfreeoutput but the script shows no output. Point being, this script shows the current swapping activity and cannot be used for historical data gathering. For that purpose, sar can be used. Note 2: To change the number of processes displayed to something other than 10, modify ‘tail -10’ to the desired number. Note 3: There is no way to know how much swap space is used by a process in kernel versions prior to version 2.6.18-128.el5 (RHEL 5 update 3). So in any prior RHEL versions (all of RHEL 3, RHEL 4, and RHEL 5 up to and including RHEL 5 update 2) the necessary kernel code for determining how much swap space is used by individual processes is not present. Note 4: This can also be accomplished by running thetopcommand and adding the SWAP column to the outputs. To add this column pressf, to enter the field editor,pto enable the SWAP column, andenterto return to thetopoutput. One may expect this field to shows how much a program is swapped out.
Unfortunately,topnatively shows this information by using this formula:

Raw

Note 5 System V IPC maintains a list of swapped out shared memory (shmem) pages on theshmem_swaplistlist. Before Since RHEL 5 update 8 (2.6.18-293.el5) swapped out System V IPC shared memory segments are not exposed to user space. Under newer kernels this information is exported through the/proc/sysvipc/shmfile.
It is also possible to inspect the swap usage with “top”:
1) Open top, press “f” and then select the swap field
2) Press “ESC” , ant then “W” and confirm writing to “~/.toprc”
3) Quit top and run “top -b -n1 > top_b_n1.txt” to save the output to a file

Resolution 3: Increase swap space

To increase swap space on your system, follow the instructions in these kbase articles: How do I add a swap file to my Red Hat Enterprise Linux system? How do I check for information about the swap on my Red Hat Enterprise Linux system? How can we increase or decrease the size of an LVM-based swap volume on Red Hat Enterprise Linux? The guide for how much swap size the system should have is What is the recommended swap size for Red Hat platforms?.
You might want to add another 4GB of swap to your system to reduce the alerts. To do that the command would be:

Raw

… and follow the rest of kbase How do I check for information about the swap on my Red Hat Enterprise Linux system? to create and activate the swap. Note 7 swapon command accept priority option which swap device should be used at first compared to the other swap devices. The details are written inman 2 swapon.
Quote fromman 2 swapon

Raw

Resolution 4: Flushing the swap

The First option is to flush the swap by utilizing the following commands:

Raw

Warning: Flushing swap in this way will force the entire contents of swap back into main memory. If your system already low on memory this may cause it to go into an Out Of Memory condition (OOM). Due care and consideration should be taken before using this option. If the system is somewhat low on memory instead of very low pages may have to be reclaimed while bringing the contents of swap into memory – this may degrade performance for a short time.

Root Cause

The system has 64GB of physical memory and 2GB of swap, thus it is relatively easy for swap to become 100% full. This caused alerts to be triggered on the system where alerts are generated by a kind of system monitoring tool. It is better to increase the amount of swap space to prevent it from becoming 100% full. This will prevent alerts from being generated.
Also, please remember that in a linux system, swapping, itself is not bad. Any data that is filesystem backed, is never swapped. Only the anonymous pages go into swap memory. The linux kernel’s page frame reclaim algoritm (PFRA) essentially comes into play in three occasions.
1) Out of memory reclaiming – kernel detects a ‘low memory’ condition.
2) Hibernation reclaiming – kernel must free memory because it is entering suspend-to-disk state.
3) Periodic reclaiming – kernel employs a thread to reclaim memory periodically.
Speaking from a wide perspective (not the most scientific statement), the problem description of this article gleans towards the issue of periodic reclaiming. Here, the kswapd kernel thread is periodically awoken to check whether the number of free memory pages are below the pages_high watermark in a particular memory zone. When it detects such a case, it transfers the anonymous pages to swap, based on the scanning priority and access time of the page. Over time, those swapped out pages might gather a substantial size, which is then indicated by the monitoring system. However, this mechanism of swapping is in fact, one of the many wonders that memory management subsystem does to improve performance.
Linux doesn’t have a problem with swap being at 100%. If the processes that are using the swap don’t need that memory and Linux doesn’t have anything else to put into swap it will quite happily leave it at 100%. If this is generating alerts then you can increase the swap space to stop the alerts but you can’t force the system to not use the swap unless you start killing processes or flush the swap.

Diagnostic Steps

  • Check output offreecommand.
点赞

发表评论

电子邮件地址不会被公开。 必填项已用*标注