# The Tagion Developer's Onboarding Guide (Draft 1)

# Internal project's architecture

Tagion core project consists of many units, categorized as `bin`, `lib`, and `wrap`.

- **Library** (`lib`) contains business logic covered by unit tests, compiles to the static or shared library;
- **Binary** (`bin`) contain CLI interface to libraries, compiles to executable;
- **Wrapper** (`wrap`) contains external libraries integrated in Tagion build system compiles to the static or shared library.

Any Tagion unit is compiled by a set of GNU Make scripts, called **Tub** (Tagion Unit Builder). Every unit must comply with Tub structure (more details in Tub project).

### The bin units overview

##### `bin-wave`
The main binary file starts the 'main-node.' The 'main node' connects to the network, runs the hachgraph, and synchronizes the data. It is a full-fledged network node that can be used for operations with tagions, balance checking, etc.

##### `bin-wallet`
(cnl - command line) Wallet - this full-fledged network wallet generates invoices and contracts, pays for invoices, updates the user's plan, sends requests to network nodes, etc.

##### `bin-hibonutil`
The project uses the HIBON (Hashinvariant Binary Object Notation) specification. So, the 'bin-hibonutil' file can read the HIBON data format and convert it to JSON and vice versa.

##### `bin-dartutil`
Is used for work with the DART (Distributed Archive of Random Transactions) database. It can read and modify DART.

##### `bin-boot`
(cnl) - Allows adding a DART file with the original balance to the database to start the developers' or the test versions of networks, not to start the main network.

##### `bin-wasmutil`

##### `bin-subscription`

### The lib units overview

Libs can depend on each other (`utils` -> `basic` -> `options`). Dependencies are specified in the `context.mk`.

#### The libs short description
**`lib-basic`**
The basic library that stores loggers, Tagion exceptions, messages, keywords, and error classes.

**`lib-utils`**
The extra functions' storage.

**`lib-communication`**
Uses HRPC (Hash Remote Procedure Call) similar to RPC (Remote Procedure Call) when a computer program causes a procedure (subroutine) to execute in a different address space (commonly on another computer on a shared network), which is coded as if it was a normal (local) procedure call without the programmer explicitly coding the details for the remote interaction.
This protocol sends a data structure to a remote server with a method name and arguments. And then, this server returns the answer, as if calling a function on a remote server.
HRPC is the same but uses the HIBON (*Hashinvariant Binary Object Notation, which Binary JSON inspires (BSON). But the two formats are not compatible. In HiBON, the keys are sorted according to the ordering rules. By ordering the keys, the data is hash invariant for the same collection.) data format instead of the usual one, which the industry accepts. Here are all the necessary structures to make RPC requests.
We often use such requests to query a node, update the balance, or retrieve information. The query for the node is made using the HRPC request.

**`lib-crypto`**
The wrapper for crypto functions like hash, encryption, and decryption.
lib-crypto depends on the external module secp256k1, located in wrapper 'wrap-secp256k1'.
This encryption algorithm (used in bitcoin) is the industry standard for encryption and electronic signatures.
The interface to this wrapper is the specific standard implementation of the SecureNet, which represents the related methods like a public key, private key, derive, etc.

**`lib-dart`**
Stores the [DART](https://github.com/tagion/space-content/blob/master/website-home/Tagion%20Technical%20Paper/distributed-database.md) (Distributed Archive of Random Transactions) database logic. DART is built to store and keep track of transactions in the Tagion network. The database efficiently handles the removal and addition of transactions in a secure and distributed manner. Each transaction is stored in a distributed hash-table using a cryptographic hash of the transaction T data. Each transaction is identified by a unique hash value, h. The transaction is put into a table ordered by the numerical value of the hash.

> [not sure if we need this section here while having a reference for DART in the techpaper]**The DART database structure**

**The DART database structure**

* The hash-table is distributed between the nodes in the network, where each node manages a sample of sections.
* A section must be handled by more than Q nodes to keep redundancy and security of the data.
* Each node must maintain the database sections within the nodes section angle. This means adding and removing the transaction and updating the Merkle-tree root of the section hash.
* The DART is updated according to the transaction list in an epoch generated by the network.
* The scripting engine will evaluate actions in the epoch and decide if an archive should be added, removed, or selected.
* Selection of an archive means that the archive is sent back to the network and deleted from the DART. When a node updates a section, it must calculate the Section Merkle-root, sign it, and send it to the network.
* The signed section and the selected archive are distributed via the gossip protocol to the network. Each node will collect all the signed roots of the updated section when the majority has been reached for all updated sections.

**The DART rules**:
* The hash counts for all the data stored in DART.
* The data is stored in a tree according to the hash.
* The data hash is counted first, then the hash of each Merkle Tree node.
* The hash is the same for the whole database. If to change the minor data there, it will affect the hash of the entire database. We can understand which section of the database has changed and which one needs to be synchronized from this tree.
* Data in DART is stored using archives and recorders (sets of archives).
* The archive is a wrapper that stores the data and the data operation type: (NONE, REMOVE, ADD).
* The recorder is the array of archives containing data and operations on them.
* All operations with DART (adding, deleting) always go through archives and recorders. There is a chain of recorders to roll back or forth any operation.

**`lib-funnel`**
Stores the scripting engine, set of its basic structures, DART-related facilities (like bills), etc.
The scripting engine allows the creation of some instructions in a particular language. These instructions modify DART and become archives and recorders.
The scripting engine is located between the wallet and the database and transforms the wallet's language into the database one.

***Example**: if the wallet sends some money to someone, its final aim is to change the DART content. That's like deleting one banknote and creating another one with another owner. This operation is modifying DART and is stored in the archive and recorder.

**`lib-gossip`**
The network works with the hashgraph. The hashgraph works with a gossip protocol. So, we randomly send some events to different active nodes in the network, and at some point, we come to a consensus. The random protocol, we send some events to random nodes, is stored here.

**`lib-hashgraph`**
The hashgraph logic is stored here. It generates events, comes to a consensus, etc. The hashgraph is a subtype of DAG (Directed Acyclic Graph - directed graph with no directed cycles. It consists of vertices and edges (arcs), with each edge directed from one vertex to another, such that following those directions will never form a closed A directed graph is a DAG if and only if it can be topologically ordered by arranging the vertices as a linear ordering that is consistent with all edge directions).

**`lib-hibon`**
Contains the HIBON specification and all its structures (HIBON, document).
The document is a buffer that can extract the necessary binary data set from itself to other structures. A HIBON structure wrapper over the buffer extracts the needed data and pulls them out on the HIBON structure. The buffer is immutable data. We can pass it between streams and nodes. HIBON can be converted to a document and vice versa to create a new HIBON. (?)

**`lib-logger`**
The primary staging for all the service loggers.

**`lib-mobile`**
The lib for the mobile wallet we use on IOS and Android. That is just a wallet for mobile devices.

**`lib-network`**
It is a set of network functions that allow listening to external sockets and communicating with those who use Tagion as a node. For example, any wallet that wants to send a transaction to a node sends it here. According to the classic client-server architecture, the node receives it thanks to Tagion Network, which listens for incoming connections.

**`lib-options`**
All the high-level services use options. We can configure the Tatgon way (network) with these options available for the Tagion internal services.

**`p2pgowraper`**
It is used for hashgraph and communication inside the network. This wrapper is over the lib golibp2p (outside lib) module, allowing nodes to communicate via hashgraph & gossip protocols.

**`lib-services`**
The system works due to concurrency programming. The whole set of services runs as separate threads of tasks. All work is due to communication. We're still talking about the same system running many services, many threads, and these services communicate with each other.
The parent service **'p2pTagionService'** - starts all other ones.

**`lib-wallet`**
Contains the Tagion wallet logic.

#### The Tagion services

A Tagion Node is divided into units, and each unit handles a service function as follows:
* A smart contract is sent to the **Transaction-service-unit**, fetching the inputs from the **DART unit** and verifying their signatures.
* The **DART unit** connects to other DARTs via the **P2P unit**. The transaction unit forwards the smart contract, including the inputs, to the **Coordinator-unit**, which adds to an event gossiped to the network via the **P2P unit**.
* When the Coordinator receives an event with a smart contract, it is executed via the **ScriptingEngine-unit**, and the result of outputs is verified.
* When the Coordinator finds an epoch, this epoch is forwarded to the **Transcript-service-unit** that evaluates the correct order and requests the **DART-unit** to erase the inputs and add the newly generated outputs.

The Tagion Node service structure:
Each of the services is running as independent tasks and communication via commutation channels. The different services modules perform the service as described in the list below.

**`P2P `**
This service handles the peer-to-peer communication protocol used to communicate between the nodes.

**`Transaction`**
This service receives the incoming transaction script, validates, verifies, and fetches the data from the DART, and sends the information to the Coordinator.
The **`Transaction service`** usees the lib-network to:
* listen to specific port requests like transactions of search in DART;
* receive and execute a contract;
* receive a list of public keys to check if there are any bills in the database;
* send the event with the contract to all network nodes to reach a consensus and make an [epoch](https://github.com/tagion/space-content/blob/master/website-home/Tagion%20Technical%20Paper/hashgraph-consensus-mechanism.md).

**`Coordinator`**
This service manages the hashgraph-consensus and controls another related service for the node. The **`Coordinator`** generates and receives events and relays to the network. This service generates the epoch and sends the information to the **`ScriptingEngine services`**.

**`DART`**
Services to the Distributed database.
The **`ScriptingEngine`** - handles the executions of the scripts.
The **`transcript service`** (internal) is used for creating the [epoch](https://github.com/tagion/space-content/blob/master/website-home/Tagion%) with the recorder for DART inside, and for ordering of the script executuin.
An epoch is a structure containing a recorder and the epoch number and means that the node has reached a consensus according to this data recorder.
When the **`transcript service`** gets this epoch, it checks all the data in this recorder whether they can be added and whether they are not added or deleted twice, for example. If the data checking is correct, the service sends a request to the **`DARTService`**.
The **`DARTSynchronizeServise`** contains a database object and performs all the operations like the data modifying according to the request sent (add, delete, return the DART data by the appropriate hash).

**`Logger`**
The service handles the information logging for the different services.

**`Monitor`**
The Monitor service is used to monitor the activities locally.

**`HeartBeat`**
This service is only used in test mode. This service enables the nodes to execute sequentially, simplifying network debugging.

#### The modes for the network running
When the node starts, it has to:
* Find other nodes (with their address) in the network.
* Synchronize the database to the actual state.

There are the following modes for the network running:

* **Internal** - It runs as one program, where each node is a separate thread communicating with other ones as between threads simulating a network.
* **Local** - It runs the network on several nodes, but the service searches for nodes only in the local network. It will not detect itself or someone else in the public network.
* **Public** - fully distributed.

#### Services used depending on the mode started
* **Internal** - `MdnsDiscoveryService` - scans the local network to find nodes.
* **Local** - `FileDiscoveryService` - the path to one file where all nodes should communicate their addresses, and through some delay, all nodes will read all list of addresses.
* **Public** - `ServerFileDiscoveryService` - (similar to the`FileDiscoveryService`) sends data to Tagion shared server.
* **DART synchronization** uses the `DARTSynchronizeService` and has the following stages:
1. the network nodes search before the local node starts and becomes online;
2. connection to different network nodes, and synchronization the missing data to receive some actual DART state;
3. Get online and start the hashgraph.