Sunday, May 10, 2020

Ubuntu Linux High CPU For Swap Process

What do you do if you just installed a fresh Ubuntu 20.04 server, and after installing some packages with "apt" you notice high CPU usage from the swap process?

If "top" shows kswapd0 persistently using high CPU, and "freemem -d" and swap are ok, you can try to adjust the swappiness in file sysctl.conf and reboot.
$ cat /proc/sys/vm/swappiness
60
$ sudo vi /etc/sysctl.conf
$ cat /etc/sysctl.conf | grep vm
vm.swappiness=10


Changing swappiness didn't fix this problem of high CPU usage. Let's dig deep.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal
install@d990 ~ $ uname -a
Linux d990 5.4.0-29-generic #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux


Look closely at "top" output.
$ top
top - 19:03:26 up 7 min,  3 users,  load average: 3.09, 2.72, 1.44
Tasks: 132 total,   1 running, 131 sleeping,   0 stopped,   0 zombie
%Cpu(s): 76.3 us,  0.4 sy,  0.0 ni, 23.2 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15892.2 total,  10206.3 free,   4412.8 used,   1273.1 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.  11199.7 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1071 kevin     20   0 2435108   2.3g   1480 S 300.0  14.7  20:50.80 kswapd0   1147 minec     20   0 7861400   1.9g  28544 S   6.6  11.9   1:54.40 java
      1 root      20   0  167604  11524   8368 S   0.0   0.1   0:00.98 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par+
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_perc+
      9 root      20   0       0      0      0 S   0.0   0.0   0:00.01 ksoftir+
     10 root      20   0       0      0      0 I   0.0   0.0   0:00.13 rcu_sch+
     11 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migrati+
     12 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_in+
     13 root      20   0       0      0      0 I   0.0   0.0   0:00.01 kworker+
     14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
     15 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1
     16 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_in+
     17 root      rt   0       0      0      0 S   0.0   0.0   0:00.14 migrati+


$ top -u kevin
top - 19:03:59 up 8 min,  3 users,  load average: 3.05, 2.75, 1.49
Tasks: 132 total,   1 running, 131 sleeping,   0 stopped,   0 zombie
%Cpu(s): 76.3 us,  0.3 sy,  0.0 ni, 23.1 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15892.2 total,  10205.5 free,   4413.5 used,   1273.2 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.  11199.0 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1071 kevin     20   0 2435108   2.3g   1480 S 300.3  14.7  22:28.66 kswapd0
   1015 kevin     20   0   14368   6760   2800 S   0.0   0.0   0:00.00 rsync



Why is kevin in charge of swap? Kevin has yet to log in to the system.
$ last kevin

wtmp begins Sat May  9 18:16:21 2020
 

$ groups kevin
kevin : kevin
 

$ sudo grep kevin /etc/sudoers
 

$ ps -fu kevin
UID          PID    PPID  C STIME TTY          TIME CMD
kevin       1015       1  0 18:56 ?        00:00:00 rsync
kevin       1071       1 99 18:56 ?        00:11:53 ./kswapd0



We know kevin has not logged in, is only in his own group, and does not have sudo. This was the most recent account we created on the machine.
$ tail -1 /etc/passwd
kevin:x:1005:1004:,,,,novice tech learner:/home/kevin:/bin/bash



Comment out the entry in the passwd file.
$ tail -1 /etc/passwd
kevin:x:1005:1004:,,,,novice tech learner:/home/kevin:/bin/bash
 

$ sudo vi /etc/passwd
 

$ tail -1 /etc/passwd
#kevin:x:1005:1004:,,,,novice tech learner:/home/kevin:/bin/bash


Run top, and it won't know the "kevin" username for uid 1005. It is still consuming CPU.
$ top
top - 19:08:43 up 13 min,  3 users,  load average: 3.13, 2.96, 1.92
Tasks: 130 total,   1 running, 129 sleeping,   0 stopped,   0 zombie
%Cpu(s): 76.2 us,  0.6 sy,  0.0 ni, 23.0 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15892.2 total,  10198.5 free,   4413.8 used,   1279.9 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.  11198.6 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   1071 1005      20   0 2435108   2.3g   1480 S 300.7  14.7  36:40.32 kswapd0
   1147 minec     20   0 7861400   1.9g  28544 S   6.7  11.9   2:14.81 java
    375 root      20   0       0      0      0 S   0.3   0.0   0:00.01 jbd2/sd+
      1 root      20   0  167604  11524   8368 S   0.0   0.1   0:01.00 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par+
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker+
      8 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_perc+
 


With the passwd entry for kevin commented out, let's reboot and observe what happens.
$ sudo systemctl reboot

$ top
top - 19:14:04 up 1 min,  1 user,  load average: 1.35, 0.61, 0.23
Tasks: 138 total,   1 running, 137 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.4 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15892.2 total,  12795.1 free,   1850.3 used,   1246.8 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used.  13763.3 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    898 minec     20   0 7861400   1.7g  28288 S   6.3  10.7   1:20.07 java
    156 root      20   0       0      0      0 I   0.3   0.0   0:00.16 kworker+
    443 root      19  -1  133560  61216  60108 S   0.3   0.4   0:00.61 systemd+
   1206 root      20   0   13416   8268   7096 S   0.3   0.1   0:00.01 sshd
   1207 sshd      20   0   12160   4616   3708 S   0.3   0.0   0:00.01 sshd
      1 root      20   0  167744  11508   8440 S   0.0   0.1   0:03.25 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par+


Let's remove the kevin account properly. Uncomment the line in /etc/passwd and delete the account.
$ sudo vi /etc/passwd
 

$ sudo userdel -r kevin
$ grep kevin /etc/passwd

$ uptime
 19:15:53 up 3 min,  1 user,  load average: 0.21, 0.42, 0.20


Reboot and look for normal functioning.
$ sudo systemctl reboot



Install Ubuntu 20.04 Server

The server version of Ubuntu had previously been tuned for server-oriented workloads. This is reportedly no longer the case, so a primary difference between Ubuntu 20 server and desktop is that server lacks a graphical user interface.

Download an image from the Ubuntu releases page. Most everything is 64 bit. Note that "AMD" means it works on the AMD and Intel instruction sets. You can use the AMD64 image on a modern Intel CPU.

Burn the image to a DVD or other mountable storage. Boot the machine from the storage. This install will use hard-wired Ethernet and a static IP address. If you have a real (typically non-consumer internet service) domain name, use that as the "search domain".







































This is a server install, so maybe you do not want "games" in your search path. Backup the "environment" file then remove the games directory from the search path.
$ sudo mv /etc/environment /etc/environment.orig
$ sudo vi /etc/environment
$ cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"


Modify the shell login files in your home directory.
$ cd ~
$ cp -p .bashrc .bashrc.orig
$ mv .profile .profile.orig
$ mv .bashrc .bash_profile

Remove colorization by setting TERM environment variable in .bash_profile.
$ echo $TERM
xterm-256color
$ export TERM=xterm-mono

Edit .bash_profile and put in a bit of color to the command prompt variable PS1.
$ grep 033 ~/.bash_profile
PS1='\[\033[01;32m\]\u@\h\[\033[00m\] \w \$ '


Put the present working directory at the end of the PATH variable. Add this to file .bash_profile.

export PATH=$PATH:.
Remove shell's suggestions for a mis-typed command. Add this to file .bash_profile.
unset command_not_found_handle

Then "source" the login files or simply log out and log in again.
$ ./.bash_profile


Get familiar with the install and the machine.
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal
 


$ uname -a
Linux d990 5.4
.0-26-generic #30-Ubuntu SMP Mon Apr 20 16:58:30 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
 

$ lspci
00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:16.3 Serial controller: Intel Corporation 6 Series/C200 Series Chipset Family KT Controller (rev 04)
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (Lewisville) (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b4)
00:1c.2 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 3 (rev b4)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4)
00:1f.0 ISA bridge: Intel Corporation Q67 Express Chipset LPC Controller (rev 04)
00:1f.2 RAID bus controller: Intel Corporation SATA Controller [RAID mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 04)
 

$ df -k
Filesystem      1K-blocks    Used  Available Use% Mounted on
udev              8093172       0    8093172   0% /dev
tmpfs             1627360    1204    1626156   1% /run
/dev/sda2      1921800384 9591096 1814517336   1% /
tmpfs             8136796       0    8136796   0% /dev/shm
tmpfs                5120       0       5120   0% /run/lock
tmpfs             8136796       0    8136796   0% /sys/fs/cgroup
/dev/loop0          27776   27776          0 100% /snap/snapd/7264
/dev/loop1          56320   56320          0 100% /snap/core18/1705
/dev/loop2          70656   70656          0 100% /snap/lxd/14804
tmpfs             1627356       0    1627356   0% /run/user/1000


Familiarize yourself with the network configuration.
$ ls -l /etc/netplan
total 4
-rw-r--r-- 1 root root 280 May 10 00:03 00-installer-config.yaml

$ cat /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    enp0s25:
      addresses:
      - 192.168.0.9/24
      gateway4: 192.168.0.1
      nameservers:
        addresses:
        - 1.1.1.1
        - 8.8.8.8
        search:
        - duckdns.org
  version: 2



Look at the syslog.
$ sudo tail /var/log/syslog

Look at the running processes, then look at running services.
$ ps -ef | more
$ systemctl list-units --all --type=service --no-pager

Let's remove a service we don't want automatically started, and one we don't need.
$ sudo systemctl stop rsync
$ sudo systemctl disable rsync

$ systemctl status vgauth
● vgauth.service - Authentication service for virtual machines hosted on VMware
     Loaded: loaded (/lib/systemd/system/vgauth.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Sun 2020-05-10 00:16:27 UTC; 2h 30min ago
       Docs: http://github.com/vmware/open-vm-tools

May 10 00:16:27 d990 systemd[1]: Condition check resulted in Authentication service for virtual machines hosted on VMware being skipped.


$ sudo systemctl stop vgauth
$ sudo systemctl disable vgauth

$ systemctl status open-vm-tools
● open-vm-tools.service - Service for virtual machines hosted on VMware
     Loaded: loaded (/lib/systemd/system/open-vm-tools.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Sun 2020-05-10 02:56:23 UTC; 3min 54s ago
       Docs: http://open-vm-tools.sourceforge.net/about.php

May 10 02:56:23 d990 systemd[1]: Condition check resulted in Service for virtual machines hosted on VMware being skipped.
 

$ sudo systemctl stop open-vm-tools
[sudo] password for install:
 

$ sudo systemctl disable open-vm-tools
Synchronizing state of open-vm-tools.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable open-vm-tools
Removed /etc/systemd/system/multi-user.target.wants/open-vm-tools.service.
 

$ systemctl status open-vm-tools
● open-vm-tools.service - Service for virtual machines hosted on VMware
     Loaded: loaded (/lib/systemd/system/open-vm-tools.service; indirect; vendor preset: enabled)
     Active: inactive (dead)
       Docs: http://open-vm-tools.sourceforge.net/about.php

May 10 02:56:23 d990 systemd[1]: Condition check resulted in Service for virtual machines hosted on VMware being skipped.


This is a server machine, so we don't need this process attaching to a sound card.
$ apt list pulseaudio
Listing... Done
pulseaudio/focal-updates 1:13.99.1-1ubuntu3.5 amd64 [upgradable from: 1:13.99.1-1ubuntu3.3]
N: There are 3 additional versions. Please use the '-a' switch to see them.
$ sudo apt remove pulseaudio

 
Since this is a server install, the majority of the time it will be accessed remotely. A person using an X display may want to work with images, so install an image viewer such as eog.
$ sudo apt install eog


Disable printing of Ubuntu newswire during shell logon.
$ sudo chmod -x /etc/update-motd.d/50-motd-news

Note the firewall is not active.
$ sudo ufw status
Status: inactive


Install software updates. You may need to reboot the machine to apply all software updates.
$ sudo apt update
$ sudo apt upgrade
$ sudo systemctl reboot

Familiarize yourself with users and groups.
$ cat /etc/passwd
$ cat /etc/group

Put in users and groups.
$ sudo addgroup minecrft
Adding group `minecrft' (GID 1001) ...
Done.
$ sudo adduser minec --ingroup minecrft
Adding user `minec' ...
Adding new user `minec' (1001) with group `minecrft' ...


On a consumer-type internet connection, you may want to configure a dynamic DNS service such as DuckDNS. Create the user, get your information from duckdns.org, then configure software.
$ sudo addgroup duckdns
$ sudo adduser duckdns --ingroup duckdns
Read this to configure the software and crontab entry for duckdns.


Let's change the time zone to Amsterdam.
$ cat /etc/timezone
Etc/UTC
$ timedatectl
               Local time: Sun 2020-05-10 19:12:56 UTC
           Universal time: Sun 2020-05-10 19:12:56 UTC
                 RTC time: Sun 2020-05-10 19:12:56
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

$ timedatectl list-timezones | grep -i ams
Europe/Amsterdam
 

$ sudo timedatectl set-timezone Europe/Amsterdam
 

$ cat /etc/timezone
Europe/Amsterdam

$ timedatectl
               Local time: Sun 2020-05-10 21:14:02 CEST
           Universal time: Sun 2020-05-10 19:14:02 UTC
                 RTC time: Sun 2020-05-10 19:14:02
                Time zone: Europe/Amsterdam (CEST, +0200)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no


 

While we could try to disable the cloud initialization with
$ touch /etc/cloud/cloud-init-disabled
Let's remove it.
$ sudo apt remove cloud-init


Optionally, install X server.
$ sudo apt install tightvncserver
$ sudo apt install xterm
Then configure your .Xresources file.


Optionally, install javascript runtime via apt.
$ sudo apt install nodejs
$  which node
/usr/bin/nodejs
$ nodejs --version
v10.19.0

$ sudo apt install chromium-browser
$ which chromium-browser
/usr/bin/chromium-browser

Optionally, upgrade the node software.
$ curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
$ sudo apt install nodejs
$ which node
/usr/bin/node
$ node -v
v12.16.3
$ npm -v
6.14.4



Anyone editing files with vim (vi is typically vim) may want to learn the basics of the .vimrc startup file.
$ cat ~/.vimrc
syntax off
set showmatch
set hlsearch
set matchpairs+=<:>,(:),{:},[:]

:nmap <F1> <nop>


For a graphical editor, install nedit.
$ sudo apt install nedit


Familiarize yourself with memory and disk space, network interfaces and networking, and how the machine is running.  Review the output from the following commands.
Since ifconfig is deprecated, use the ip command. Instead of traceroute, use the mtr command.
$ free -m
$ df -k
$ sudo lshw


$ landscape-sysinfo
$ top
$ htop


$ ip a
$ mtr wunderground.com 

__________________________________________________________

Update of this blog post with more readable explanation of network settings to be used during install from the console. These examples use IP address 192.168.0.6.
















$ ls -l /etc/netplan
total 4
-rw-r--r-- 1 root root 260 Oct 16 21:13 00-installer-config.yaml

$ cat /etc/netplan/*
# This is the network config written by 'subiquity'
network:
  ethernets:
    enp5s0:
      addresses:
      - 192.168.0.6/24
      gateway4: 192.168.0.1
      nameservers:
        addresses:
        - 1.1.1.1
        - 8.8.8.8
        search: []
  version: 2

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether bc:30:5b:e7:a4:f9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.6/24 brd 192.168.0.255 scope global enp5s0
       valid_lft forever preferred_lft forever
    inet6 fe80::be30:5bff:fee7:a4f9/64 scope link
       valid_lft forever preferred_lft forever