FRR Management Framwork
====
###### tags: `SONiC`
# Agenda 2021/05/27
How to use the .j2 file
How to use protocols interact with FRR daemon
How a config_db.json be load into the FRR daemons.

[FRR Management Framework HLD](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_Design_Doc_Unified_FRR_Mgmt_Interface.md)
[FRRouting](/4D69f2fjR6GrCaTuhrRJ9A) my note

# FRR System Architecture
FRR is a suite of daemons that work together to build the routing table. Each major protocol is implemented in its own daemon, and these daemons talk to a middleman daemon (zebra), which is responsible for coordinating routing decisions and talking to the dataplane.
FRR daemons can be managed through a single integrated user interface shell called vtysh. vtysh connects to each daemon through a UNIX domain socket and then works as a proxy for user input.

Daemon start up config in SONiC

dockers/docker-fpm-frr/frr/supervisord/supervisord.conf.j2
platform/vs/docker-sonic-vs/supervisord.conf

FRR Daemon start up parameter

src/sonic-frr/frr/tools/etc/frr/daemons

# Communication Protocol to FRR daemons
VTYSH communicates with FRR daemons by way of domain socket. Each daemon creates its own socket, typically in `/var/run/frr/<daemon>.vty`.
We could check in GDB like following:

b vtysh_client_run
(gdb) p *vclient
$2 = {fd = 7, name = 0x55ce74661299 "staticd", flag = 32768, path = "/var/run/frr/staticd.vty", '\000' <repeats 4071 times>, next = 0x0}

In the VTYSH to daemon direction, messages are simply NUL-terminated strings whose content are CLI commands. Here is a typical message from VTYSH to a daemon:

Request
ip route 10.77.77.0/24 10.44.44.2\0

The response format has some more data in it. First is a NUL-terminated string containing the **plaintext response, which is just the output of the command that was sent in the request**. The plaintext response is followed by 3 null marker bytes, followed by a 1-byte status code that indicates whether the command was successful or not.

Response

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Plaintext Response                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                 Marker (0x00)                 |  Status Code  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Our `frrcfgd` will use same protocol to communicate with FRR daemons.

# Advance Introduction of frrcfgd.py
- New FRR config deamon (frrcfgd) fully based on config-DB events, the legacy bgpcfgd daemon(src/sonic-bgpcfgd/bgpcfgd/main.py) will be replace.

# FRR Daemon startup config file

    graph TD
        docker_init.sh([docker_init.sh])
        sonic-cfggen[sonic-cfggen]
        jinja2["staticd.conf.j2/staticd.db.conf.j2"]
        config["/etc/frr/staticd.conf"]
        docker_init.sh-->sonic-cfggen
        jinja2-->sonic-cfggen
        sonic-cfggen-->config

## Templates
- if `frr_mgmt_framework_config` enable, sonic-cfggen will use `templates/frr/staticd.conf.j2` to generate `/etc/sonic/frr/staticd.conf` (individual config file for staticd daemon).

templates/
├── bfdd
│   └── bfdd.conf.j2
├── bgpd
│   ├── bgpd.conf.db.addr_family.evpn.j2
│   ├── bgpd.conf.db.addr_family.j2
│   ├── bgpd.conf.db.comm_list.j2
│   ├── bgpd.conf.db.j2
│   ├── bgpd.conf.db.nbr_af.j2
│   ├── bgpd.conf.db.nbr_or_peer.j2
│   ├── bgpd.conf.db.pref_list.j2
│   ├── bgpd.conf.db.route_map.j2
│   └── bgpd.conf.j2
├── frr
│   └── frr.conf.j2
├── ospfd
│   ├── ospfd.conf.db.area.j2
│   ├── ospfd.conf.db.comm_list.j2
│   ├── ospfd.conf.db.distributeroute.j2
│   ├── ospfd.conf.db.interface.j2
│   ├── ospfd.conf.db.policyrange.j2
│   ├── ospfd.conf.db.vlink.j2
│   └── ospfd.conf.j2
└── staticd
    ├── staticd.conf.j2
    ├── staticd.db.conf.j2
    └── staticd.db.default_route.conf.j2

`*.conf.j2` are responsible include `*.db.conf.j2` which is template file of FRR daemon config file format.
`*.db.conf.j2` describe how the CONFIG_DB data translate to FRR daemon config.

`sonic-cfggen` use CONFIG_DB and `*.db.conf.j2` as input, then generate the FRR daemons config.

## sonic-cfggen

- Render template with json:
`sonic-cfggen -d -t /usr/share/template/bgpd.conf.j2`
- Dump config DB content into json file:
`sonic-cfggen -d --print-data > db_dump.json`
- Load content of json file into config DB:
`sonic-cfggen -j db_dump.json --write-to-db`
### docker_init.sh trigger sonic-cfggen
When the bgp container startup, the docker_init.sh execuate.

CFGGEN_PARAMS=" \
    -d \
    -y /etc/sonic/constants.yml \
    -t /usr/share/sonic/templates/frr_vars.j2 \
    -t /usr/share/sonic/templates/supervisord/supervisord.conf.j2,/etc/supervisor/conf.d/supervisord.conf \
    -t /usr/share/sonic/templates/bgpd/gen_bgpd.conf.j2,/etc/frr/bgpd.conf \
    -t /usr/share/sonic/templates/supervisord/critical_processes.j2,/etc/supervisor/critical_processes \
    -t /usr/share/sonic/templates/zebra/zebra.conf.j2,/etc/frr/zebra.conf \
    -t /usr/share/sonic/templates/staticd/gen_staticd.conf.j2,/etc/frr/staticd.conf \
    -t /usr/share/sonic/templates/gen_frr.conf.j2,/etc/frr/frr.conf \
    -t /usr/share/sonic/templates/isolate.j2,/usr/sbin/bgp-isolate \
    -t /usr/share/sonic/templates/unisolate.j2,/usr/sbin/bgp-unisolate \
    -t /usr/local/sonic/frrcfgd/bfdd.conf.j2,/etc/frr/bfdd.conf \
    -t /usr/local/sonic/frrcfgd/ospfd.conf.j2,/etc/frr/ospfd.conf \
"

## jinja2

[jinja Document](https://jinja.palletsprojects.com/en/3.0.x/)

- sonic-cfggen handle jinja2

paths: `['/', '/usr/share/sonic/templates', '/usr/local/sonic/frrcfgd']`

def _get_jinja2_env(paths):
    """
    Retreive Jinj2 env used to render configuration templates
    """
    loader = jinja2.FileSystemLoader(paths)
    redis_bcc = RedisBytecodeCache(SonicV2Connector(host='127.0.0.1'))
    env = jinja2.Environment(loader=loader, trim_blocks=True, bytecode_cache=redis_bcc)
    env.filters['sort_by_port_index'] = sort_by_port_index
    env.filters['ipv4'] = is_ipv4
    env.filters['ipv6'] = is_ipv6
    env.filters['unique_name'] = unique_name
    env.filters['pfx_filter'] = pfx_filter
    env.filters['ip_network'] = ip_network
    for attr in ['ip', 'network', 'prefixlen', 'netmask', 'broadcast']:
        env.filters[attr] = partial(prefix_attr, attr)
    # Pass the is_multi_asic function as global
    env.globals['multi_asic'] = is_multi_asic

    return env

- Custom Filters
templates/staticd/staticd.db.conf.j2

{%- set ip_addr = ip_prefix.split('/')[0] %}
{%- if ip_addr|ipv4 %}
    {%- set ns.str = 'ip route' %}
{%- else %}
    {%- set ns.str = 'ipv6 route' %}
{%- endif %}

Filter python implementation

env.filters['ipv4'] = is_ipv4
def is_ipv4(value):
    if not value:
        return False
    if isinstance(value, netaddr.IPNetwork):
        addr = value
    else:
        try:
            addr = netaddr.IPNetwork(str(value))
        except:
            return False
    return addr.version == 4

[build-in-filter](https://jinja.palletsprojects.com/en/3.0.x/templates/#builtin-filters)
- macro
It is like a function implement.

macro iproute(ip_prefix, nh_blackhole, nh_ip, nh_intf, nh_dist, nh_tag, nh_vrf)

Render the data

env = _get_jinja2_env(paths)
for template_file, dest_file in args.template:
    template = env.get_template(os.path.basename(template_file))
    template_data = template.render(data)
    if dest_file == "config-db":
        deep_update(data, FormatConverter.to_deserialized(json.loads(template_data)))
    else:
        with smart_open(dest_file, 'w') as df:
            print(template_data, file=df)

- FRR daemon config Format output

docker exec -it bgp bash
root@sonic:/# sonic-cfggen -d -t /usr/local/sonic/frrcfgd/staticd.db.conf.j2
! template: staticd/staticd.db.conf.j2
!
! Static Route configuration using config DB static route table
!

ip route 10.1.1.0/24 192.168.3.2 Ethernet4
ipv6 route FC00:0:1:cc::/64 FC00:0:1:bb:0:0:0:2 Ethernet4
!
vrf Vrf01
  ip route 10.1.1.0/24 192.168.3.2 Ethernet4
  exit-vrf
!

### /etc/frr/staticd.conf
Set the log or FRR options appear here.

! =========== Managed by sonic-cfggen DO NOT edit manually! ====================
! generated by templates/frr/staticd.conf.j2 using config DB data
! file: staticd.conf
!
!
! template: common/daemons.common.conf.j2
!
hostname sonic
password zebra
enable password zebra
!
log syslog informational
log facility local4
!
! end of template: common/daemons.common.conf.j2!
!
! set static default route to mgmt gateway as a backup to learned default
!!
! template: staticd/staticd.db.conf.j2
!
! Static Route configuration using config DB static route table
!
!

# `BgpdClientMgr`: Interact with FRR daemon
- TABLE_DAEMON:
- Bind the config_db table name with FRR dameon.

'STATIC_ROUTE': ['staticd']
'DEVICE_METADATA': ['bgpd']

- ALL_DAEMONS: all running FRR daemons.

ALL_DAEMONS = ['bgpd', 'zebra', 'staticd', 'bfdd', 'ospfd', 'pimd']

- __create_frr_client:
Create the client socket by ALL_DAEMONS and connect by address `/run/frr/staticd.vty (/var/run/frr/<daemon>.vty)`
- VTYSH_CMD_DAEMON:
- Define which daemon will handle the corespond command

bgpd_client.run_vtysh_command(table, command, daemons)
self.__send_data(sock, command + '\0')
self.__get_reply(sock)

# `BGPConfigDaemon`: Handle with CONFIG_DB

Get the table from config db.

nbr_table = self.config_db.get_table('BGP_NEIGHBOR')
sroute_table = self.config_db.get_table('STATIC_ROUTE')

### Subscribe and Listen

def subscribe_all(self):
        for table, hdlr in self.table_handler_list:
            self.config_db.subscribe(table, hdlr)

Start listen Redis keyspace events and will trigger corresponding handlers when content of a table changes.

def listen(self):
    """Start listen Redis keyspace events and will trigger corresponding handlers when content of a table changes.
    """
    self.pubsub = self.get_redis_client(self.db_name).pubsub()
    self.pubsub.psubscribe(**{"__keyspace@{}__:*".format(self.get_dbid(self.db_name)): self.sub_msg_handler})
    self.sub_thread = self.pubsub.run_in_thread(sleep_time = 0.01)

### Binding table and handler list:

self.table_handler_list = 
('BGP_NEIGHBOR', self.bgp_neighbor_handler),
('STATIC_ROUTE', self.bgp_table_handler_common)

#### bgp_table_handler_common handler

May 24 13:52:15.139524 sonic DEBUG bgp#/frrcfgd: ----------------------------------
May 24 13:52:15.139524 sonic DEBUG bgp#/frrcfgd:  BGP table handling
May 24 13:52:15.139524 sonic DEBUG bgp#/frrcfgd: ----------------------------------
May 24 13:52:15.139524 sonic DEBUG bgp#/frrcfgd: table : STATIC_ROUTE
May 24 13:52:15.139524 sonic DEBUG bgp#/frrcfgd: key   : 10.1.1.0/24
May 24 13:52:15.139524 sonic DEBUG bgp#/frrcfgd: op    : SET
May 24 13:52:15.139524 sonic DEBUG bgp#/frrcfgd: data  :
May 24 13:52:15.139524 sonic DEBUG bgp#/frrcfgd:         blackhole  - False
May 24 13:52:15.139524 sonic DEBUG bgp#/frrcfgd:         ifname     - Ethernet4
May 24 13:52:15.139524 sonic DEBUG bgp#/frrcfgd:         nexthop    - 192.168.3.2
May 24 13:52:15.139524 sonic DEBUG bgp#/frrcfgd:         vrf        - default
May 24 13:52:15.139524 sonic DEBUG bgp#/frrcfgd:

### self.__add_op_to_data
Prepare the CONFIG_DB data:

{'blackhole': (False, ADD), 
 'ifname': (Ethernet4, ADD), 
 'nexthop': (192.168.3.2, ADD), 
 'vrf': (default, ADD)}

### self.__update_bgp:
Data get from Redis DB:

STATIC_ROUTE, 
{'blackhole': (False, ADD), 'ifname': (Ethernet4, ADD), 'nexthop': (192.168.3.2, ADD), 'vrf': (default, ADD), 'ip_prefix': (10.1.1.0/24, UPDATE)},
['configure terminal', 'vrf default'], 
default

-

### Define global keymap data:
- Input data field value:
- ++ or +: optional value
- no prefix: Mandatory field
- field&field: Key list
- Output command print format

static_route_map = [
(['ip_prefix|ipv4', '++blackhole', '++nexthop', '++ifname', '++track', '++tag', '++distance', '++nexthop-vrf'],
'{no:no-prefix}ip route {} {:blackhole} {} {} {:track} {:nh-tag} {} {:nh-vrf}',
hdl_static_route, socket.AF_INET),

(['ip_prefix|ipv6', '++blackhole', '++nexthop', '++ifname', '++track', '++tag', '++distance', '++nexthop-vrf'],
'{no:no-prefix}ipv6 route {} {:blackhole} {} {} {:track} {:nh-tag} {} {:nh-vrf} ',
hdl_static_route, socket.AF_INET6)
]

- Table to key map

tbl_to_key_map = {
'STATIC_ROUTE':     static_route_map,
'BGP_NEIGHBOR':     cmn_key_map[0:2] + nbr_key_map + cmn_key_map[2:],
}

### Find the keymap/db_field by table and tbl_key

key_map = BGPKeyMapList(self.tbl_to_key_map[table], table, tbl_key)

- table: STATIC_ROUT
- tbl_key: {'ip_prefix': 'ipv6'}
- self.tbl_to_key_map[table]

[(['ip_prefix|ipv4', '++blackhole', '++nexthop', '++ifname', '++track', '++tag', '++distance', '++nexthop-vrf'], '{no:no-prefix}ip route {} {:blackhole} {} {} {:track} {:nh-tag} {} {:nh-vrf}', <function hdl_static_route at 0x7f2d9fe69048>, <AddressFamily.AF_INET: 2>), (['ip_prefix|ipv6', '++blackhole', '++nexthop', '++ifname', '++track', '++tag', '++distance', '++nexthop-vrf'], '{no:no-prefix}ipv6 route {} {:blackhole} {} {} {:track} {:nh-tag} {} {:nh-vrf} ', <function hdl_static_route at 0x7f2d9fe69048>, <AddressFamily.AF_INET6: 10>)]

Looks Like:

db_field: ['ip_prefix', '++blackhole', '++nexthop', '++ifname', '++track', '++tag', '++distance', '++nexthop-vrf'], 
key_map: [CMD: {no:no-prefix}ip route {} {:blackhole} {} {} {:track} {:nh-tag} {} {:nh-vrf}]

### key_map.run_command(self, table, data, cmd_prefix, vrf)
Take key_map and data, then construct the real vtysh command.

Parameter value

data: {'blackhole': (False, ADD), 
'ifname': (Ethernet4, ADD), 
'nexthop': (FC00:0:1:bb:0:0:0:2, ADD), 
'vrf': (default, ADD), 
'ip_prefix': (fc00:0:1:cc::/64, UPDATE)}
cmd_prefix: ['configure terminal', 'vrf default']
vrf: default

1. classfied the db_field to following catgory
- opt_idx_list
- req_idx_list
- upd_id_list
- del_id_list
- no_chg_id_list

2. Get the field value we care:
- **self.get_cmd_data**(key_list, req_idx_list, opt_idx_list, data, del_id_list, no_chg_id_list, merge_vals, True)
3. Construct the vtysh command by keymap
- **key_map.get_command**(daemon, data_val_op[1], start_idx, *(upper_vals + data_val_op[0]))

### Last step: g_run_command

g_run_command(table, cmd_prefix + "-c '%s'" % cmd, True, key_map.daemons, ignore_fail)

May 24 13:52:15.139524 sonic DEBUG bgp#/frrcfgd: VTYSH CMD: configure terminal daemons: ['staticd']
May 24 13:52:15.139524 sonic DEBUG bgp#/frrcfgd: VTYSH CMD: vrf default daemons: ['staticd']
May 24 13:52:15.146124 sonic DEBUG bgp#/frrcfgd: VTYSH CMD: ip route 10.1.1.0/24 False 192.168.3.2 Ethernet4 daemons: ['staticd']

Finally, it send to the `BgpdClientMgr` class.

# Appendix
----
BGP_NEIGHBOR table:

"BGP_NEIGHBOR": {
        "10.0.0.1": {
            "rrclient": 0,
            "name": "ARISTA01T2",
            "local_addr": "10.0.0.0",
            "nhopself": 0,
            "holdtime": "180",
            "asn": "65200",
            "keepalive": "60"
        },
 }

STATIC_ROUTE table:

# config_db.json
"STATIC_ROUTE":{
    "10.1.1.0/24": {
        "vrf":"default",
        "nexthop":"192.168.3.2",
        "blackhole":"False",
        "ifname":"Ethernet4"
    },
    "FC00:0:1:cc::/64": {
        "vrf":"default",
        "nexthop":"FC00:0:1:bb:0:0:0:2",
        "blackhole":"False",
        "ifname":"Ethernet4"
    }
},

# [FRR Management HLD from SONiC](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_Design_Doc_Unified_FRR_Mgmt_Interface.md)

:::info
- Notation: docker-fpm-frr (bgp container, sonic-frr)
:::
# Configuration to CONFIG_DB
SONiC have many configuration method to `CONFIG_DB(4)`
- [SONiC Configuration Database Manual](https://github.com/Azure/sonic-swss/blob/master/doc/Configuration.md)
- [SONiC Configuration Setup Service](https://github.com/Azure/SONiC/blob/b16791f3ca16398153556f40b11831cfbb6aba79/doc/ztp/SONiC-config-setup.md)

For now, it is using the `FRR vtysh` to set the `ROUTE_TABLE` which is in the `APPL_DB`. There is no method will sync the configuration to `CONFIG_DB`.

There are few major method to setting the `CONFIG_DB`

| Terms | Definition | comment |
| -------- | -------- | -------- |
| config-setup | config-setup service (/usr/bin/config-setup) | `files/image_config/config-setup/config-setup` |
| /etc/sonic/config_db.json | startup-config | Text |
| minigraph | use on testbed | Text |
| ztp | Zero Touch Provisioning | Text |
| sonic-mgmt-framework | RESTFUL/Openapi/swagger/yang | Text |
| sonic-utilities | CLI interfaces | src/sonic-utilities/config/main.py |

Each type have difference purpose.

On the other hand, SONiC also have 3 different management frameworks and 1 management framework for testbed.
They are all make people confuse and using on different function.

### sonic-buildimage (for config_db)
- [src/sonic-mgmt-framework](https://github.com/Azure/sonic-mgmt-framework)
- **[src/sonic-frr-mgmt-framework](https://github.com/Azure/sonic-buildimage/tree/master/src/sonic-frr-mgmt-framework)**
- It is for convert the CONFIG_DB ROUTE_TABLE setting to APPL_DB
:::danger
Now, we setting the route table `APPL_DB` from `FRR vtysh`.
In the future it maybe different
:::
- [src/sonic-utilities](https://github.com/Azure/sonic-utilities)
- CLI method, such as command `config interface ip add Ethernet0 192.168.112.254/24 `
### Testbed (management the testbed)
- [sonic-mgmt](https://github.com/Azure/sonic-mgmt)
This is another story.

# Redis DB choice
[SONiC Configuration Database Manual](https://github.com/Azure/sonic-swss/blob/master/doc/Configuration.md)
:::success
Chose the right ddatabase to find the keys.

redis-cli
127.0.0.1:6379[4]> select 1
# or 
redis-cli -n <database index>

The database index mapping is in `src/sonic-swss-common/common/schema.h`
:::

#define APPL_DB         0
#define ASIC_DB         1
#define COUNTERS_DB     2
#define LOGLEVEL_DB     3
#define CONFIG_DB       4
#define PFC_WD_DB       5
#define FLEX_COUNTER_DB 5
#define STATE_DB        6
#define SNMP_OVERLAY_DB 7
#define RESTAPI_DB      8
#define GB_ASIC_DB      9
#define GB_COUNTERS_DB  10
#define GB_FLEX_COUNTER_DB  11
#define CHASSIS_APP_DB      12
#define CHASSIS_STATE_DB    13

# sonic-utilities
source code : `src/sonic-utilities/config/main.py`

# Open `frr_mgmt_framework_config`
14:06
src/sonic-mgmt-framework

clue1: /usr/bin/docker_init.sh
It will load the flag value from config_db.json

clue2: dockers/docker-fpm-frr/frr/frr_vars.j2

{
    "frr_mgmt_framework_config":
        {% if "frr_mgmt_framework_config" in DEVICE_METADATA["localhost"].keys() %}
            "{{ DEVICE_METADATA["localhost"]["frr_mgmt_framework_config"] }}"
        {% else %}
            ""
}

The DEVICE_METADATA is a key.

So, we could add `"frr_mgmt_framework_config": "true"` to config_db.json like following.


config reload

docker exec -it bgp bash
root        82  1.0  0.8 171612 16924 pts/0    Sl   08:10   0:00 /usr/bin/python3 /usr/local/bin/frrcfgd

#or use it without enter containr bash
docker exec -i bgp ps -aux

root        59  0.6  0.4 171612 17260 pts/0    Sl   08:36   0:02 /usr/bin/python3 /usr/local/bin/frrcfgd

[FRR_Mgmt_Interface](https://github.com/Azure/SONiC/blob/7747e2f77ca34a7e7539f4f572d5657c42a957dc/doc/mgmt/SONiC_Design_Doc_Unified_FRR_Mgmt_Interface.md)

# Create client socket and connect with damon server
`BgpdClientMgr.__create_frr_client`
Connect to frr daemon:
following are server address.

root@sonic:/# ls /run/frr/
bgpd.pid  bgpd.vty  staticd.pid  staticd.vty  zebra.pid  zebra.vty  zserv.api

Debug Log: `/var/log/frr`

docker cp sonic-frr-mgmt-framework.tgz bgp:/
gdb python3
r -m pdb frrcfgd.py