What do you do if you just installed a fresh Ubuntu 20.04 server, and after installing some packages with "apt" you notice high CPU usage from the swap process?
If "top" shows kswapd0 persistently using high CPU, and "freemem -d" and
swap are ok, you can try to adjust the swappiness in file sysctl.conf
and reboot.
$ cat /proc/sys/vm/swappiness
60
$ sudo vi /etc/sysctl.conf
$ cat /etc/sysctl.conf | grep vm
vm.swappiness=10
Changing swappiness didn't fix this problem of high CPU usage. Let's dig deep.
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04 LTS
Release: 20.04
Codename: focal
install@d990 ~ $ uname -a
Linux d990 5.4.0-29-generic #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Look closely at "top" output.
$ top
top - 19:03:26 up 7 min, 3 users, load average: 3.09, 2.72, 1.44
Tasks: 132 total, 1 running, 131 sleeping, 0 stopped, 0 zombie
%Cpu(s): 76.3 us, 0.4 sy, 0.0 ni, 23.2 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 15892.2 total, 10206.3 free, 4412.8 used, 1273.1 buff/cache
MiB Swap: 4096.0 total, 4096.0 free, 0.0 used. 11199.7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1071 kevin 20 0 2435108 2.3g 1480 S 300.0 14.7 20:50.80 kswapd0 1147 minec 20 0 7861400 1.9g 28544 S 6.6 11.9 1:54.40 java
1 root 20 0 167604 11524 8368 S 0.0 0.1 0:00.98 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par+
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_perc+
9 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftir+
10 root 20 0 0 0 0 I 0.0 0.0 0:00.13 rcu_sch+
11 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migrati+
12 root -51 0 0 0 0 S 0.0 0.0 0:00.00 idle_in+
13 root 20 0 0 0 0 I 0.0 0.0 0:00.01 kworker+
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/1
16 root -51 0 0 0 0 S 0.0 0.0 0:00.00 idle_in+
17 root rt 0 0 0 0 S 0.0 0.0 0:00.14 migrati+
$ top -u kevin
top - 19:03:59 up 8 min, 3 users, load average: 3.05, 2.75, 1.49
Tasks: 132 total, 1 running, 131 sleeping, 0 stopped, 0 zombie
%Cpu(s): 76.3 us, 0.3 sy, 0.0 ni, 23.1 id, 0.3 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 15892.2 total, 10205.5 free, 4413.5 used, 1273.2 buff/cache
MiB Swap: 4096.0 total, 4096.0 free, 0.0 used. 11199.0 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1071 kevin 20 0 2435108 2.3g 1480 S 300.3 14.7 22:28.66 kswapd0
1015 kevin 20 0 14368 6760 2800 S 0.0 0.0 0:00.00 rsync
Why is kevin in charge of swap? Kevin has yet to log in to the system.
$ last kevin
wtmp begins Sat May 9 18:16:21 2020
$ groups kevin
kevin : kevin
$ sudo grep kevin /etc/sudoers
$ ps -fu kevin
UID PID PPID C STIME TTY TIME CMD
kevin 1015 1 0 18:56 ? 00:00:00 rsync
kevin 1071 1 99 18:56 ? 00:11:53 ./kswapd0
We know kevin has not logged in, is only in his own group, and does not have sudo. This was the most recent account we created on the machine.
$ tail -1 /etc/passwd
kevin:x:1005:1004:,,,,novice tech learner:/home/kevin:/bin/bash
Comment out the entry in the passwd file.
$ tail -1 /etc/passwd
kevin:x:1005:1004:,,,,novice tech learner:/home/kevin:/bin/bash
$ sudo vi /etc/passwd
$ tail -1 /etc/passwd
#kevin:x:1005:1004:,,,,novice tech learner:/home/kevin:/bin/bash
Run top, and it won't know the "kevin" username for uid 1005. It is still consuming CPU.
$ top
top - 19:08:43 up 13 min, 3 users, load average: 3.13, 2.96, 1.92
Tasks: 130 total, 1 running, 129 sleeping, 0 stopped, 0 zombie
%Cpu(s): 76.2 us, 0.6 sy, 0.0 ni, 23.0 id, 0.3 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 15892.2 total, 10198.5 free, 4413.8 used, 1279.9 buff/cache
MiB Swap: 4096.0 total, 4096.0 free, 0.0 used. 11198.6 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1071 1005 20 0 2435108 2.3g 1480 S 300.7 14.7 36:40.32 kswapd0
1147 minec 20 0 7861400 1.9g 28544 S 6.7 11.9 2:14.81 java
375 root 20 0 0 0 0 S 0.3 0.0 0:00.01 jbd2/sd+
1 root 20 0 167604 11524 8368 S 0.0 0.1 0:01.00 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par+
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_perc+
With the passwd entry for kevin commented out, let's reboot and observe what happens.
$ sudo systemctl reboot
$ top
top - 19:14:04 up 1 min, 1 user, load average: 1.35, 0.61, 0.23
Tasks: 138 total, 1 running, 137 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.4 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 15892.2 total, 12795.1 free, 1850.3 used, 1246.8 buff/cache
MiB Swap: 4096.0 total, 4096.0 free, 0.0 used. 13763.3 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
898 minec 20 0 7861400 1.7g 28288 S 6.3 10.7 1:20.07 java
156 root 20 0 0 0 0 I 0.3 0.0 0:00.16 kworker+
443 root 19 -1 133560 61216 60108 S 0.3 0.4 0:00.61 systemd+
1206 root 20 0 13416 8268 7096 S 0.3 0.1 0:00.01 sshd
1207 sshd 20 0 12160 4616 3708 S 0.3 0.0 0:00.01 sshd
1 root 20 0 167744 11508 8440 S 0.0 0.1 0:03.25 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_par+
Let's remove the kevin account properly. Uncomment the line in /etc/passwd and delete the account.
$ sudo vi /etc/passwd
$ sudo userdel -r kevin
$ grep kevin /etc/passwd
$ uptime
19:15:53 up 3 min, 1 user, load average: 0.21, 0.42, 0.20
Reboot and look for normal functioning.
$ sudo systemctl reboot
Thank you so much!!
ReplyDeleteI followed every single step and I managed to get my ubuntu working again!