---
tags: DNF Counting, Fedora
---

# DNF Counting: Sending bus messages

## Topics

### Root Namespace

`logging.stats`

This is so other use cases can utilize the same certificates and RabbitMQ user, and just change the `stats` suffix for topics.

### Message Types and Schemas

#### Basic schema – `logging.stats.*`

All messages in the `logging.stats` topic namespace should contain these fields in their bodies:

- `host`: a string, the full name of the host on which the script/program sending the message is run on

#### Debugging messages – `logging.stats.debug.*`

Any debugging message should have a topic in this namespace. Its contents are free-form, no particular structure should be expected.

#### `logging.stats.sync.*`

Messages in this topic namespace are about syncing log data from the web proxies to the machine where they are processed, i.e. the log server.

- `run_id`: a string containing a unique ID for each run of the script which syncs log data from e.g. the proxies to one place. This is to tell apart different runs which in some cases overlap in time.

#### `logging.stats.sync.start`, `logging.stats.sync.finish`

These messages mark when the script starts or finishes processing one host to sync log data from (including retries)

#### `logging.stats.sync.host.*`

- `synced_host`: a string, the full name of the host from which logs are synced.

#### `logging.stats.sync.host.start`, `logging.stats.sync.host.finish`

These messages mark when syncing log data from one proxy has started or finished.

#### `logging.stats.sync.host.skip`

This message notifies if a host is skipped.

- `reason`: The reason why a host was skipped (e.g. "Synced host unknown")

#### `logging.stats.sync.host.logdate.start`, `logging.stats.sync.host.logdate.finish`

We sync log data from the past three days for each host. These messages mark when the syncing of one date for a host starts and finishes.

- `log_date`: a string, the date for which log data is synced

#### `logging.stats.sync.host.logdate.fail.*`

These messages are sent if syncing log data from a machine to the log server fails for some reason. The right-most part of the topic tells if the host will be retried (`*.retry`) or not (`*.final`).

- `reason`: a string describing what went wrong (e.g. "timed out")

## Shell Script

Here's the proposed script snippet to use in syncHttpLogs.sh (e.a.):

function send_bus_msg {
    local topic="$1"
    shift
    local sent_at="$(TZ=UTC date -Iseconds)"
    local id="$(uuidgen -r)"
    local body_piece
    local body="{"
    local sep=""

    for body_piece; do
        local key_type key type value
        key_type="${body_piece%%=*}"
        key="${key_type%%:*}"
        type="${key_type#${key}}"
        type="${type#:}"
        value="${body_piece#*=}"

        if [ "$type" != "int" ]; then
            # quote strings
            value="${value//\\/\\\\}"
            value="${value//\"/\\\"}"
            value="\"${value}\""
        fi
        body="${body}${sep}\"${key}\": ${value}"
        sep=", "
    done
    body="${body}}"
 
    fedora-messaging publish - << EOF >/dev/null
{"body": ${body}, "headers": {"fedora_messaging_schema": "base.message", "fedora_messaging_severity": 20, "sent-at": "${sent_at}"}, "id": "${id}", "queue": "queue", "topic": "${topic}"}
EOF
}

It is used like this: `send_bus_msg <topic> <var1>=<value1> <var2>=<value2> ...`

## Notes

* `fedora-messaging publish` expects:
* One line of JSON per message (i.e. no pretty formatting).
* As of now, a queue name—even though it isn't needed it will throw a schema verification error otherwise. Can be anything.
* The message id has to be set in the JSON, it isn't autogenerated. Also, it has to be unique (i.e. probably needs the `uuid` tool if the message is generated in a shell script).
* Datagrepper still runs on `fedmsg`, so some topics/topic namespaces are off-limits or verified harder regarding who's allowed to send what. That's why I test messages with the topic `buildsys.test` didn't get through from the host I was testing (koji01.stg because it has a working fedora-messaging configuration). Anything else I tested not starting with `buildsys.` worked, e.g. [this message](https://apps.stg.fedoraproject.org/datagrepper/id?id=2021-6f4dfec9-80d7-49e6-80e0-f2357b63422c&is_raw=true&size=extra-large) which used the topic `test` (that got expanded to `org.fedoraproject.stg.test`).

## Test Message

A minimal message which I could send over the bus in staging:

{"body": {"test_key1": "test_value1"}, "headers": {"fedora_messaging_schema": "base.message", "fedora_messaging_severity": 20, "sent-at": "2021-07-21T12:23:12+00:00"}, "id": "6f4dfec9-80d7-49e6-80e0-f2357b63422c", "queue": "this shouldn't matter", "topic": "test"}

Prerequisites:

- Host configured for fedora-messaging with certificates recognized by the AMQP broker. E.g. `koji01.stg.iad2.fedoraproject.org`
- Permissions to read the host-side certificate files.
- A unique `id` value, i.e. updated every time.