# Logging Guidelines

This document describes the basic principles of logging which should be followed when working with the code base. Following these principles will not only make the logging output more coherent, but can significantly reduce time spent searching for bugs and make it easier for node operators to navigate the state of their actively running node.

## Log Levels

The log messages are divided into four basic levels:

- `Error` - Errors in the node. Although the node operation may continue, the error indicates that it should be addressed.
- `Warning` - Warnings should be checked in case the problem recurs to avoid potential damage.
- `Info` - Informational messages useful for node operators that do not indicate any fault or error.
- `Debug` - Information concerning program logic decisions, diagnostic information, internal state, etc. which is primarily useful for developers.

There is a notion of `V-level` attached to the `Debug` level. `V-levels` provide a simple way of changing the verbosity of debug messages. `V-levels` provide a way for a given package to distinguish the relative importance or verbosity of a given log message. Then, if a particular logger or package logs too many messages, the package can simply change the `V` level for that logger.

## Log Messages

The key-value pairs (after a log message) should add more context to log messages. Keys are quite flexible and can contain more or less any string value. For consistency with existing code, there are a few conventions that should be followed:

- Make your keys human-readable and be specific as much as possible (choose "peer_address" instead of "address" or "peer", "batch_id" instead of "id" or "batch", etc.)
- Be consistent throughout the code base.
- Keys should naturally correspond to parts of the message string.
- Use lower case for simple keys and lower_snake_case for more complex keys.

Although key names are mostly unrestricted, it is generally a good idea to stick to printable ASCII characters, or at least to match the general character set of your log lines.

`Error` and `Warning` log messages should provide enough information to identify the problem in the code base without the need to record the file name and line number alongside them, although this information can be added using the logging library.

If you are writing/changing logs, it is recommended to look at the output. Answer questions like: is this what you expected? Is anything missing? Are they understandable? Do you need more/less context?

## Users

Two types of application users are identified: node operators and developers. Node operators should not be presented with confusing technical implementation details in log messages, but only with meaningful information regarding the functionality of the application in the form of meaningful statements. Developers are the users who know the implementation details and the technical details will help them in debugging the problematic state of the application.

In general, the goal for developers is to create log messages that are easy to understand, analyze, and reference code so they can troubleshoot issues quickly. In the case of node operators, log messages should be concise and provide all the necessary information without unnecessary details.

That is, the same problem event can have two log lines, but with different severity. This is the case for a combination of `Error/Debug` or `Warning/Debug`, where `Error` or `Warning` is for the node operator and `Debug` is for the developer to help them investigate the problem. Log `Info` and `Debug V-levels` are informative about expected state changes, but `Info`, with operational information, is for node operators and `Debug V-levels`, with technical details, for developers.

> The `Error` level should be used with caution when it comes to node operators. It is recommended to log application code errors at the `Error` level if they require node operator intervention or if the node cannot continue its normal operation (unrecoverable errors). Otherwise, recoverable (application code) errors should be logged at the `Debug` level with an appropriate `V-level`. The `Debug` level should not be used to monitor the node, but metrics should be used for this purpose.

Log lines should not contain the following attributes (remove, mask, sanitize, hash, or encrypt the following):

- application source code
- session identification values
- access tokens
- passwords
- database connection strings
- encryption keys and other primary secrets and sensitive information

## Recommendations

Some useful logging recommendations for cleaner output:

- Do not log a function entry unless it is significant or unless it is logged at the debug level.
- Avoid logging from many iterations of a loop. It is OK to log from iterations of small loops or to periodically log from large loops.
- Truncate or summarize large messages or files in some way that will be useful for debugging.
- Avoid errors that are not really errors and can confuse the node operator. This sometimes happens when an application error is part of a successful execution flow.
- Do not log the same or similar errors repeatedly. It can quickly clutter the log and hide the real cause. The frequency of error types is best addressed by monitoring, and logs need to capture details for only some of these errors.