Sunday, May 10, 2020

Ubuntu Linux High CPU For Swap Process

What do you do if you just installed a fresh Ubuntu 20.04 server, and after installing some packages with "apt" you notice high CPU usage from the swap process?

If "top" shows kswapd0 persistently using high CPU, and "freemem -d" and swap are ok, you can try to adjust the swappiness in file sysctl.conf and reboot.
$ cat /proc/sys/vm/swappiness
60
$ sudo vi /etc/sysctl.conf
$ cat /etc/sysctl.conf | grep vm
vm.swappiness=10


Changing swappiness didn't fix this problem of high CPU usage. Let's dig deep.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal
install@d990 ~ $ uname -a
Linux d990 5.4.0-29-generic #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux


Look closely at "top" output.
$ top
top - 19:03:26 up 7 min,  3 users,  load average: 3.09, 2.72, 1.44
Tasks: 132 total,   1 running, 131 sleeping,   0 stopped,   0 zombie
%Cpu(s): 76.3 us,  0.4 sy,  0.0 ni, 23.2 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15892.2 total,  10206.3 free,   4412.8 used,   1273.1 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.  11199.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1071 kevin     20   0 2435108   2.3g   1480 S 300.0  14.7  20:50.80 kswapd0   1147 minec     20   0 7861400   1.9g  28544 S   6.6  11.9   1:54.40 java
      1 root      20   0  167604  11524   8368 S   0.0   0.1   0:00.98 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par+
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_perc+
      9 root      20   0       0      0      0 S   0.0   0.0   0:00.01 ksoftir+
     10 root      20   0       0      0      0 I   0.0   0.0   0:00.13 rcu_sch+
     11 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migrati+
     12 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_in+
     13 root      20   0       0      0      0 I   0.0   0.0   0:00.01 kworker+
     14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
     15 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1
     16 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_in+
     17 root      rt   0       0      0      0 S   0.0   0.0   0:00.14 migrati+


$ top -u kevin
top - 19:03:59 up 8 min,  3 users,  load average: 3.05, 2.75, 1.49
Tasks: 132 total,   1 running, 131 sleeping,   0 stopped,   0 zombie
%Cpu(s): 76.3 us,  0.3 sy,  0.0 ni, 23.1 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15892.2 total,  10205.5 free,   4413.5 used,   1273.2 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.  11199.0 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1071 kevin     20   0 2435108   2.3g   1480 S 300.3  14.7  22:28.66 kswapd0
   1015 kevin     20   0   14368   6760   2800 S   0.0   0.0   0:00.00 rsync



Why is kevin in charge of swap? Kevin has yet to log in to the system.
$ last kevin

wtmp begins Sat May  9 18:16:21 2020
 

$ groups kevin
kevin : kevin
 

$ sudo grep kevin /etc/sudoers
 

$ ps -fu kevin
UID          PID    PPID  C STIME TTY          TIME CMD
kevin       1015       1  0 18:56 ?        00:00:00 rsync
kevin       1071       1 99 18:56 ?        00:11:53 ./kswapd0



We know kevin has not logged in, is only in his own group, and does not have sudo. This was the most recent account we created on the machine.
$ tail -1 /etc/passwd
kevin:x:1005:1004:,,,,novice tech learner:/home/kevin:/bin/bash



Comment out the entry in the passwd file.
$ tail -1 /etc/passwd
kevin:x:1005:1004:,,,,novice tech learner:/home/kevin:/bin/bash
 

$ sudo vi /etc/passwd
 

$ tail -1 /etc/passwd
#kevin:x:1005:1004:,,,,novice tech learner:/home/kevin:/bin/bash


Run top, and it won't know the "kevin" username for uid 1005. It is still consuming CPU.
$ top
top - 19:08:43 up 13 min,  3 users,  load average: 3.13, 2.96, 1.92
Tasks: 130 total,   1 running, 129 sleeping,   0 stopped,   0 zombie
%Cpu(s): 76.2 us,  0.6 sy,  0.0 ni, 23.0 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15892.2 total,  10198.5 free,   4413.8 used,   1279.9 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.  11198.6 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1071 1005      20   0 2435108   2.3g   1480 S 300.7  14.7  36:40.32 kswapd0
   1147 minec     20   0 7861400   1.9g  28544 S   6.7  11.9   2:14.81 java
    375 root      20   0       0      0      0 S   0.3   0.0   0:00.01 jbd2/sd+
      1 root      20   0  167604  11524   8368 S   0.0   0.1   0:01.00 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par+
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_perc+
 


With the passwd entry for kevin commented out, let's reboot and observe what happens.
$ sudo systemctl reboot

$ top
top - 19:14:04 up 1 min,  1 user,  load average: 1.35, 0.61, 0.23
Tasks: 138 total,   1 running, 137 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.4 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15892.2 total,  12795.1 free,   1850.3 used,   1246.8 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.  13763.3 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    898 minec     20   0 7861400   1.7g  28288 S   6.3  10.7   1:20.07 java
    156 root      20   0       0      0      0 I   0.3   0.0   0:00.16 kworker+
    443 root      19  -1  133560  61216  60108 S   0.3   0.4   0:00.61 systemd+
   1206 root      20   0   13416   8268   7096 S   0.3   0.1   0:00.01 sshd
   1207 sshd      20   0   12160   4616   3708 S   0.3   0.0   0:00.01 sshd
      1 root      20   0  167744  11508   8440 S   0.0   0.1   0:03.25 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par+


Let's remove the kevin account properly. Uncomment the line in /etc/passwd and delete the account.
$ sudo vi /etc/passwd
 

$ sudo userdel -r kevin
$ grep kevin /etc/passwd

$ uptime
 19:15:53 up 3 min,  1 user,  load average: 0.21, 0.42, 0.20


Reboot and look for normal functioning.
$ sudo systemctl reboot



1 comment:

  1. Thank you so much!!
    I followed every single step and I managed to get my ubuntu working again!

    ReplyDelete