# Understanding NetworkManager logs

It is probable that you have encountered problems when configuring networking in your laptop, server, VM or even container. There are high changes that NetworkManager has been involved there. This article aims to clarify how to debug and read logs from NetworkManager.

## What is NetworkManager?

NetworkManager is a daemon that attempts to make networking configuration
and operation as painless and automatic as possible by managing the
primary network connection and other network interfaces, like Ethernet,
Wi-Fi, and Mobile Broadband devices.

This definitions is provided by the NetworkManager manpage and it is quite complete and correct but if you are not familiar with networking or system administration you might find this a little bit useless.

In essence, NetworkManager is a software that is running as a service in the system so it will be always running in the background performing multiple operations. For a common user these operations are basically two.

* Wi-Fi Access Point scanning/connecting
* Dynamic IP address configuration through DHCP

Usually, NetworkManager is controlled by a client. The clients are the software that the user runs actively to configure all needed. e.g nmcli, nmtui, nm-applet. When debugging and reading the logs, we will focus on NetworkManager logs so this article is going to be useful no matter which client you use.

## Logs and proper debug configuration

By default NetworkManager does not log all the domains, it only logs essential operations and all warnings/errors. For a common use that is enough but when debugging an issue you might want to enable TRACE logs.

Adding the following block to your NetworkManager configuration logging file will enable TRACE logs for all the domains.

[logging]
level=TRACE
domains=ALL

The logging configuration file can be found at `/etc/NetworkManager/conf.d/95-logging.conf`

Please, notice that restarting NetworkManager service is mandatory after modifying the global configuration.

# systemctl restart NetworkManager

You can find more information about logging configuration at `LOGGING` section in the manpage.

## Where are the logs?

You can find the logs in syslog using `journalctl`. The most common command is `journalctl -u NetworkManager`. If NetworkManager has been running for a while this command is not going to be really useful as you will find thousands of logging lines.

I recommend filtering by time according to your needs.

journalctl --since "2023-07-10 15:10:00" -u NetworkManager

## Directly to the issue, how to find out relevant information

I would like to take an descending perspective of the multiple domains you might find in the NetworkManager logs.

### Settings-connection

These will be the log lines closer to the user side. On these lines you will fine what is the client configuring on the NetworkManager daemon. It is the first place to spot a wrong configuration.

...: <trace> [1698315996.9760] settings-connection[d641522308f4faf9,ac16258d-c663-3eb4-a24d-aaf61bbac537]: update settings-connection flags to unsaved,nm-generated,visible (was none)
...: <trace> [1698315996.9760] dbus-object[d641522308f4faf9]: export: "/org/freedesktop/NetworkManager/Settings/46"
...: <debug> [1698315996.9769] ++ connection 'new connection' (0x2aa4cb0/NMSimpleConnection/"802-3-ethernet"): [/org/freedesktop/NetworkManager/Settings/46]
...: <debug> [1698315996.9770] ++ connection                [ 0x2a2ccb0 ]
...: <debug> [1698315996.9770] ++ connection.autoconnect-priority = -999
...: <debug> [1698315996.9770] ++ connection.id             = 'Wired connection 5'
...: <debug> [1698315996.9771] ++ connection.interface-name = 'eth1.ep'
...: <debug> [1698315996.9771] ++ connection.permissions    = []
...: <debug> [1698315996.9771] ++ connection.timestamp      = 1698315996
...: <debug> [1698315996.9772] ++ connection.type           = '802-3-ethernet'
...: <debug> [1698315996.9772] ++ connection.uuid           = 'ac16258d-c663-3eb4-a24d-aaf61bbac537'

As you can notice on this snippet, these lines will reflect what is configured on the connection. If a field is misconfigured here it is indicating that the data passed from the client is wrong. This could be caused due to a misuse of the client or a bug.

If that is your case, please re-check the configuration you are trying to set on the client and if that doesn't match the logs, feel free to open an issue on NetworkManager gitlab repository.

### Manager

Is kind of the core of NetworkManager. It manages the state of devices, triggers activation of connections and general state of NetworkManager daemon.

manager: block-autoconnect: device[ef59e48eb5faea69,eth2p]-profile[307286e7c9684e41,Wired connection 4]: retries set 2

Currently, NetworkManager has a mechanism to autoconnect connections. This mechanism can be configured to retry a fixed number of times and block the autoconnection if reached. When that happens you will notice lines similar to this. If the connection is not autoconnecting as expected these lines will be helpful to understand what is happening.

manager: (br0): assume: don't assume because device is not managed

Assumed devices are a confusing concept for the users. In essence, when a device is created and it is not unmanaged by NetworkManager, NetworkManager will try to "assume" the configuration and manage the device honoring the existing configuration. These is something common when investigating reboot issues.

Anyway, this is a concept that is planned to be dropped at some point.

### Policy

The policy module as the name indicates is the one that manages several policies for connections. As mentioned in the past one of this policy is the autoconnect state.

policy: auto-activating connection 'Office Wi-Fi' (1447da44-a51e-4387-a016-f6793cec4e88)

In this line you can notice which connection is being auto-activated.

policy: block-autoconnect: unblocking port profiles for controller ifname="wlp0s20f3", uuid="1447da44-a51e-4387-a016-f6793cec4e88"

Another example is this line that tells you which port is being unblocked for autoconnection.

### Device

The device is the interface representation in NetworkManager. These log lines shows all the operations done at interface level.

device (wlp0s20f3): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')

The state is indicating in which phase is the device. This will provide hints of what happened to the device. A common state that might be interesting to check is `config`. At this phase, if something goes wrong it can go to deactivating.

Understanding the multiple states is important when debugging an issue.

device[4c10b9d3a3aa1f8c] (wlp0s20f3): ip6: check-state: state pending => pending, is_failed=0, is_pending=1, is_started=1 temp_na=0, may-fail-4=1, may-fail-6=1

During the ip-check phase, the device will check the IP configuration. A common problem is that a connection is configured to hold a DHCP IPv4 address but there is no DHCP server configured. When that happens, the device will hold on ip-check state until there is a timeout. If an interface is activating but failing on IP configuration, this is a really nice line to spot.

device[b89c9b9cec81ecb5] (eth1p): ip:dhcp4: DHCP failing: timeout getting lease
device[b89c9b9cec81ecb5] (eth1p): ip:dhcp4: set state fail (was pending)
device (eth1p): state change: ip-config -> failed (reason 'ip-config-unavailable', sys-iface-state: 'managed')

These lines are an example of what was described above.

device[4c10b9d3a3aa1f8c] (wlp0s20f3): mtu: device-mtu: 1460 (was 1500), ipv6-mtu: 1410, ifindex: 3

Other device properties and changes can be found on the device logs like the MTU or MAC address.

### l3cfg

The l3cfg or Layer-3-configuration module is the one in charge of configuring and managing IP addresses and routes for the device.

l3cfg[07759e4bfa8fc334,ifindex=25]: obj-state: track: [548ba95452f95975, ip6-address, fe80::94f9:74ad:d4fd:437b/64 lft forever pref forever lifetime 112999-0[0,0] (...)

Here you can notice how a new IPv6 address is being tracked and configured by NetworkManager on the interface with ifindex 25.

l3cfg[07759e4bfa8fc334,ifindex=25]: obj-state: track: [6bd82d63498852f3, ip6-route, type unicast fe80::/64 dev 25 metric 1024 mss 0 rt-src ipv6ll]

And a similar line for an IPv6 route. In essence, if an address/route is missing or misconfigured l3cfg lines are going to help you finding out what is happening.

### DHCP

Going more into details for Layer 3 configuration, you will find `dhcp4` lines. These lines are from the DHCP client used. They describe the events happening on the client side.

dhcp4 (eth2): event: send DISCOVER to 255.255.255.255

Here you can notice how the DISCOVER was sent to broadcast addresss.

### The platform lines

The platform module is the one in charge of kernel configuration. It sets the netlink attributes according to what the user configured.