Linux

Systemd - systemctl (including Cheat Sheet)

I promised these articles for quite some time. Let's finally start with some systemd. Hated, loved, ranted about, praised ... Well, let's say systemd is at least controversial. Still, it is built-in in the majority of Linux distributions. But why is it so? How does it work?

Daniel Schier

Jan 23, 2024 — 10 min read

Photo by Pixabay from Pexels

systemd

Before digging into the systemctl command, I want to give a gist of systemd and how it works. The most controversial statements about systemd are made, because it is misunderstood, not because it is a bad software. Yes, it has some bugs here and there, but the concept is different to many known alternatives.

So, systemd was built to replace Init services like SysV-Init or Upstart, right? Well, yes, but also no. Systemd replaces other Init software, but it CAN take care of way more things. Its modular services can make it easy to interact with networks, logging and many more things. To some degree, it is the layer between the kernel and other applications.

The kernel provides abstraction to the hardware with drivers, its own API/ABI, scheduler for CPU, storage and whatnot. Systemd sits on top and makes it easier to use this hardware. A general purpose approach to configure the Linux based system as a whole, one might say. Think of name resolution, time services, IP address configuration and other things that make use of the network hardware. In the past, you needed to configure different services for each of these. Each having their own configuration file format, API, and coding standards.

With systemd, you get the basics out of the way and can focus on higher applications that make an impact on productivity. Commands like systemctl, journalctl, hostnamectl, timedatectl and many more provide the user interface to systemd.

systemctl

Now, that this is out of the way, let's dig into the first, important command: systemctl. The systemctl command controls the systemd system and systemd service manager. This means, you can start and stop services, choose the targets for boot ups, but also get a lot of information about these services. Additionally, you can manage the service manager itself, configure the systemd environment and even more.

For the sake of this article, I will give a brief overview of some essential commands and background. In addition, I will add a cheat sheet at the end or on the "Cheat Sheets" page.

Units

Before using the systemctl command, one needs to get an idea of systemd units and systemd unit files. In general, these are configuration files for devices, services, boot targets or timers. A unit, therefore, represents a component of your Linux system that can be controlled with systemd. Too abstract? Ok, let's simplify it and give a proper example.

If you want to start the Apache httpd web server on a Linux system, you need to start the binary including some configuration, environment variables and whatnot. The location of the binary, the command that should be used, and much more must be known to you. Furthermore, you need a way to monitor the lifetime of the executed binary. Is it still running? Do you want to restart it automatically after a crash? Are there commands that must be executed before running the web server? What about after a crash or graceful shutdown? Are there other services that need to be handled, or you depend on?

Well, in the past, these parameters, commands, and options were put in a script and this script was executed. Depending on the author of the script, this worked well, nice or not. Systemd on the other hand describes these things in unit files and handles them as a unit. In case of Apache httpd on Fedora, you can check the content of the unit file and its location with:

$ systemctl cat httpd

# /usr/lib/systemd/system/httpd.service
# See httpd.service(8) for more information on using the httpd service.

# Modifying this file in-place is not recommended, because changes
# will be overwritten during package upgrades.  To customize the
# behaviour, run "systemctl edit httpd" to create an override unit.

# For example, to pass additional options (such as -D definitions) to
# the httpd binary at startup, create an override unit (as is done by
# systemctl edit) and enter the following:

#       [Service]
#       Environment=OPTIONS=-DMY_DEFINE

[Unit]
Description=The Apache HTTP Server
Wants=httpd-init.service
After=network.target remote-fs.target nss-lookup.target httpd-init.service
Documentation=man:httpd.service(8)

[Service]
Type=notify
Environment=LANG=C

ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
# Send SIGWINCH for graceful stop
KillSignal=SIGWINCH
KillMode=mixed
PrivateTmp=true
OOMPolicy=continue

[Install]
WantedBy=multi-user.target

# /usr/lib/systemd/system/service.d/10-timeout-abort.conf
# This file is part of the systemd package.
# See https://fedoraproject.org/wiki/Changes/Shorter_Shutdown_Timer.
#
# To facilitate debugging when a service fails to stop cleanly,
# TimeoutStopFailureMode=abort is set to "crash" services that fail to stop in
# the time allotted. This will cause the service to be terminated with SIGABRT
# and a coredump to be generated.
#
# To undo this configuration change, create a mask file:
#   sudo mkdir -p /etc/systemd/system/service.d
#   sudo ln -sv /dev/null /etc/systemd/system/service.d/10-timeout-abort.conf

[Service]
TimeoutStopFailureMode=abort

As you can see, there is a lot of stuff that needs to be taken care of. And yes, these files look different for each and every service. But, they are wayyyy easier to read and understand than shell scripts.

Now, systemd units can be a lot of things. Here is a short list of possible unit files.

.service: a service that should run in the background
.timer: a scheduler to manage a service on a regular basis (similar to cron jobs)
.target: a state, that can be reached from the system, often used to synchronize other units
.socket: a socket description for socket based activation
.device: devices that demand systemd management
.mount: mount points on a system; content of /etc/fstab will be autogenerated
.automount: configures a mount point that will be mounted automatically
.swap: describes the swap space of a system
.snapshot: is generated by running systemctl snapshot to reconstruct a given state of a system
.slice: slices relate to cgroups/Linux Control Groups and restrict access to resources
.scope: used to manage sets of systemd processes; automatically created

Now, you might know why I am always appending the kind of unit in my Ansible articles when I am interacting with systemd. Managing these units is part of the systemctl command. And, in case you want to stop here and read the manual instead, please run man systemctl.

Unit Commands

Still here? Awesome. Now that we know what a unit is, we can start working with them. Let's stick with the above-mentioned Apache httpd, when it comes to concrete commands and output of the same.

Help(!)

In case you need help, the following commands might be helpful.

# Read the man page
$ man systemctl

# Get help for systemctl
$ systemctl --help

# Get help for a unit
$ systemctl help UNIT

Using these command will help you, for sure, in my opinion even more than using the search engine of your choice.

Inspect

Before running a systemd unit, you might want to inspect it. This is pretty easy and already shown above.

# Pattern
systemctl cat UNIT

# Example
systemctl cat httpd.service

The output will show the content and path of a given unit file.

Status

Next, you can check the status, in case you wonder what's going on.

# Pattern
$ systemctl status UNIT

# Example
$ systemctl status httpd.service 

● httpd.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Thu 2024-01-18 08:55:17 CET; 7s ago
       Docs: man:httpd.service(8)
   Main PID: 43940 (httpd)
     Status: "Started, listening on: port 80"
      Tasks: 177 (limit: 18624)
     Memory: 19.9M
        CPU: 65ms
     CGroup: /system.slice/httpd.service
             ├─43940 /usr/sbin/httpd -DFOREGROUND
             ├─43954 /usr/sbin/httpd -DFOREGROUND
             ├─43956 /usr/sbin/httpd -DFOREGROUND
             ├─43957 /usr/sbin/httpd -DFOREGROUND
             └─43958 /usr/sbin/httpd -DFOREGROUND

Jan 18 08:55:17 nb01 systemd[1]: Starting httpd.service - The Apache HTTP Server...
Jan 18 08:55:17 nb01 (httpd)[43940]: httpd.service: Referenced but unset environment variable evaluates to an empty string: OPTIONS
Jan 18 08:55:17 nb01 httpd[43940]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using fe80::b16a:26c5:33ca:14b4%wlp0s20f3. Set the 'ServerName' directive globally to supp>
Jan 18 08:55:17 nb01 httpd[43940]: Server configured, listening on: port 80
Jan 18 08:55:17 nb01 systemd[1]: Started httpd.service - The Apache HTTP Server.

And, in case you want the status of the entire system:

$ systemctl status

● 192.168.122.144
    State: running
    Units: 333 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Thu 2024-01-18 08:27:06 UTC; 40min ago
  systemd: 252-18.el9
   CGroup: /
           ├─init.scope
           │ └─1 /usr/lib/systemd/systemd rhgb --switched-root --system --deserialize 31
           ├─system.slice
           │ ├─NetworkManager.service
           │ │ └─807 /usr/sbin/NetworkManager --no-daemon
           │ ├─atd.service
           │ │ └─1132 /usr/sbin/atd -f
           │ ├─auditd.service
           │ │ ├─750 /sbin/auditd
           │ │ └─752 /usr/sbin/sedispatch
           │ ├─avahi-daemon.service
           │ │ ├─784 "avahi-daemon: running [192.local]"
           │ │ └─797 "avahi-daemon: chroot helper"
           │ ├─chronyd.service
           │ │ └─802 /usr/sbin/chronyd -F 2
           │ ├─crond.service
           │ │ ├─1135 /usr/sbin/crond -n
           │ │ └─5238 /usr/sbin/anacron -s
           │ ├─dbus-broker.service
           │ │ ├─778 /usr/bin/dbus-broker-launch --scope system --audit
           │ │ └─783 dbus-broker --log 4 --controller 9 --machine-id 8eb463806b0848319d9af65912d9fe18 --max-bytes 536870912 --max-fds 4096 --max-matches 131072 --audit
           │ ├─fail2ban.service
           │ │ └─838 /usr/bin/python3 -s /usr/bin/fail2ban-server -xf start

Pretty awesome, we can get an overview of all of our units. Here you can also see some slices, scopes and sockets in use.

Start/Stop

Starting services and timers is straight forward, too.

# Pattern
systemctl start UNIT

# Example
systemctl start httpd.service

The command will not give any output if everything was working as expected. And stopping? You guessed it, it's also possible.

# Pattern
systemctl stop UNIT

# Example
systemctl stop httpd.service

Again, there will be no output, if everything went well.

Enable/Disable

In case you want to have a unit loaded after boot, you might want to enable it. This will ensure that the service is loaded or present by default.

# Pattern
$ systemctl enable UNIT

# Example
$ systemctl enable httpd.service
Created symlink /etc/systemd/system/multi-user.target.wants/httpd.service → /usr/lib/systemd/system/httpd.service.

Yes, you can also disable a service, if it was enabled beforehand.

# Pattern
$ systemctl disable UNIT

# Example
$ systemctl disable httpd.service
Removed "/etc/systemd/system/multi-user.target.wants/httpd.service".

If you read the output carefully, you will see that enable and disable are just creating symlinks. This symlink relates to a given target, in our case the default-target.

Mask/Unmask

Stopping and disabling services does not prevent them from being started, either by a user or as a dependency of other services. If you really want to make a service or timer unusable, you need to mask the unit.

# Pattern
$ systemctl mask UNIT

# Example
$ systemctl mask httpd.service
Created symlink /etc/systemd/system/httpd.service → /dev/null.

And in case you want to make it available again:

# Pattern
$ systemctl unmask UNIT

# Example
$ systemctl unmask http.service
Removed "/etc/systemd/system/httpd.service".

Edit

There is one more thing, I want to talk about. I have seen countless articles, Stack Overflow answers and whatnot where people edit files in the /usr/lib path or copy files over /etc/systemd, for the mere sake of adjusting a service to their liking. Also, very often files are placed in /etc/systemd without an actual explanation of what's going on there.

The preferred way to edit existing services is systemctl edit UNIT. It will open an editor where you can place your adjustments and automatically place the correct file in /etc/systemd with only the content that is needed.

The result can be shown with systemctl cat UNIT afterward.

Cheat Sheet

As promised, I have started a dedicated page for cheat sheets. You will find the below cheat sheet and more (over time) on this page.

cheat-sheet-systemctl

Cheat Sheet about the systemctl command

cheat-sheet-systemctl.pdf

66 KB

Docs & Links

Below, you can find some links that might be helpful to get started with systemd and systemctl, in general.

System and Service Manager

Conclusion

This was a long one. Wasn't it? Now, what can I say about systemd? It made my life easier. Administration became straight forward. Services, timers, boot targets — all follow the same idea of unit files and are easier to maintain and edit.

There is some kind of nostalgic "it was simpler with scripts", but was it really? I don't think so. Yet, I am looking back to the old times and when running on BSD, it's often with the idea of "good old days". Yet, systemd promises a bright future, including simpler name resolving, time management and even home directory encryption in the future.

What about you? Systemd or no-systemd? How do you handle other things, that are not Init based/replacements?