# Tagion Developer's Onboarding Guide

# Internal proect architecture
We can start after the setup of the project.
The first step is to open the folder 'src' with the appropriate modules like:

* bin-boot
* bin-dartutil
* bin-hibonutil
* bin-logservicetestmoved
* bin-subscription
* bin-subscriptiondead
* bin-subscriptionmoved
* bin-walet
* bin-wasmutil
* bin-wave

*The prefix 'bin' here always means the 'exe' file without unit tests.
*The unit tests are only set in libs (libraries) and wrappers.

### Binary files
* 'bin-wave' - the main binary file starting the 'main-node'. It connects to the network, runs the hachgraph, and synchronizes the data. Such a file is a full-fledged network node that can be used for operations with tagions, checking the balance, etc.
* 'cnl (command line-wallet' - this full-fledged network wallet is used to generate invoices and contracts, pay for invoices, update the user's plan, send requests to network nodes, etc.
* 'bin-hibonutil' - The project uses the HIBON (Hashinvariant Binary Object Notation) specification. So, the 'bin-hibonutil' file can read the HIBON data format and convert it to JSON and back.
* 'bin-dartutil' - is used for work with the DART database. It can read and modify DART.
* 'bin-boot (cnl)' - allows adding a dart file with the original balance to the database to continue developing or testing versions of networks, not to start the main network.
*(Hashinvariant Binary Object Notation) specification. So, the 'bin-hibonutil' file can read the HIBON data format and convert it to JSON and back.
* 'bin-dartutil' - is used for work with the DART database. It can read and modify DART.
* 'bin-boot (cnl)' - allows adding a dart file with the original balance to the database to continue developing or testing versions of networks, not to start the main network.

### Libs
* Libs (libraries) - the set of structures, classes, and functions containing the whole project's logic. If we need to use such logic outside of the project, we have to create the binary file containing the appropriate function that accepts some end users' arguments in a specific context.
* Libs can be codependent (like 'basic' or 'utils' and 'options'). We can use all the functions of each, but it's necessary to compile all the dependent libs before.
* The dependency of each module is specified in the 'context. mk' file, indicating the reference of the current lib or binary file to some other libs.
* The 'wraps' are the third-party libs used in the project. Usually, we avoid modifying them.

There are the following libs used in the Tagion project:
* lib-basic
* lib-utils
* lib-communication
* lib-crypto
* lib-dart
* lib-funnel
* Lib -gossip
* lib-hibon
* lib-logger
* lib-mobile
* lib-network
* lib-options
* lib-p2pgowrapper
* lib-services
* lib-wallet

#### Libs short description
**lib-basic** - the basic library stores logs and error classes.
lib-utils - the basic functions' storage. These functions are not needed everywhere, but, for example, two modules need some extra function, and we store it here.
**lib-communication** - uses HRPC (Hash Remote Procedure Protocol) similar to RPC (Remote Procedure Call) when a computer program causes a procedure (subroutine) to execute in a different address space (commonly on another computer on a shared network), which is coded as if it were a normal (local) procedure call without the programmer explicitly coding the details for the remote interaction.
In other words, we use this protocol to send a data structure to a remote server with a method name and arguments. And then, this server returns the answer as if calling a function on a remote server.
HRPC is the same but uses the HIBON (*Hashinvariant Binary Object Notation, which Binary JSON inspires (BSON). But the two formats are not compatible. In HiBON, the keys are sorted according to the ordering rules. By ordering the keys, the data is hash invariant for the same collection.) data format instead of the usual one, which the industry accepts. Here are all the necessary structures to make RPC requests.
We often use such requests to query a node, update the balance, or retrieve information. The query for the node is made using the HRPC request.

**lib-crypto** - the wrapper for crypto functions like hash, encryption, and decryption.
lib-crypto depends on the external module secp256k1, located in wrapper 'wrap-secp256k1'.
The encryption algorithm used in bitcoin is the industry standard for encryption and electronic signatures.
The interface to this wrapper is SecureNet to a specific standard implementation that represents our private key and related methods like a public key, private key, etc.

**lib-dart** - stores the DART (Distributed Archive of Random Transactions) database logic. DART is built to store and keep track of transactions in the Tagion network. The database efficiently handles the removal and addition of transactions securely and distributedly. Each transaction is stored in a distributed hash table using a cryptographic hash of the transaction T data. A unique hash value identifies each transaction, h. The transaction is put into a table ordered by the numerical value of the hash.

**The structure of the DART database**


The hash table is distributed between the nodes in the network, where each node manages a sample of sections. A section must be managed by more than Q nodes to keep the redundancy and security of the data. Each node must maintain the database sections within the nodes section angle. This means adding and removing the transaction and updating the Merkle-tree root of the section hash. The DART is updated according to the transaction list in an epoch generated by the network. The scripting engine will evaluate actions in the epoch and decide if an archive should be added, removed, or selected. The selection of an archive means that the archive is sent back to the network and deleted from the DART. When a node updates a section, it must calculate the Section Merkle-root, sign it, and send it to the network. The signed section and selected archive are distributed to the network via gossip. Each node will collect all the signed roots of the updated section when the majority has been reached for all updated sections.
The node must calculate the Bull's eye (Merkle root) of the DART and sign and distribute the information via the gossip protocol. When most of the nodes in the network have reached a consensus of the Bull's eye value, the DART is considered updated. If no consensus has been reached for DART, the current transaction in the epoch must be dropped, and the DART must revert to the previous state. The Bull's eye value is stored in a hash-linked chain of blocks where each block points to the previous block's hash. Each block contains the Bull's eye pointer and a block number. It ensures data integrity, and the state of the database. The concept is shown below.


The data in a section is mapped using a Sparse Merkle Tree (SMT), making it efficient to add and remove archives into the Merkle tree.
The hash point into the DART is divided by rims. Rim zero is the most significant byte (MSB) of the hash fingerprint of the archive. Rim one is the next byte in the hash etc.
In the current implementation of the DART, the first two rims are used as the section index.
Thus the DART has a total number of indices $2^{16} = 65536$ which is equivalent to the two bytes unsigned number.
Each section stores the archive in a hash table. An SMT is used as a look-up table, and each rim is sectioned into a sub-sparse Merkle tree. It means that rim two is the first SMT and rim three is the second SMT etc.
As a comparison, a traditional Merkle tree with $2^{24} = 8^3 \approx 16 \cdot 10^{6}$ Archives. To calculate a full Merkle tree, it requires the calculation of around $32 \cdot 10^6$ hashes. By contrast, using an SMT with $16\cdot 10^6$ archives means just around 2000 hashes have to be calculated.

**The DART rules**:
* The hash counts for all the data stored in DART.
* The data is stored in a tree according to the hash.
* The data hash is counted first, then the hash of each Merkle Tree node.
* The hash is the same for the whole database. If to change minor data there, it will affect the hash of the entire database. From this tree, we can understand which section of the database has changed and which one needs to be synchronized.
* Data in DART is stored using recorders and archives.
* An archive is a wrapper over any data with the following type: (NONE, REMOVE, ADD).
* In addition to the data itself, the archive also stores the type of operation on this data.
* There is a recorder (a set of archives), which is an array of archives, data, and operations on this data. When we add something to DART, we add an archive for adding to DART.
* All operations with DART always go through archives and recorders. When we delete something from DART, we have an archive to remove it. There is a chain of these recorders to roll back or forth any operation.

**lib-funnel** - here, the scripting engine and set of basic structures are stored. If the wallet sends some money to someone, its final aim is to change the DART content.
That's like deleting your banknote and creating another one with another owner. This operation modifies DART and is stored in the archive and recorder.
In this case, we need a scripting engine - the engine that allows the creation of some instructions in a particular language. These instructions modify DART and become archives and recorders. This engine stands between the wallet and the database and transforms the language the wallet communicates into the language the database speaks. Here are the structures that we store in DART, such as bills.

**Lib-gossip** - the network works with a hashgraph, and the hashgraph works with a gossip protocol. So, we randomly send some events to different active network nodes, and at some point, we reach a consensus. The random protocol, we send some events to random nodes, is described here.

**lib-hashgraph**- the logic of the hashgraph is stored here. It generates events, comes to a consensus, etc. Hashgraph is a subtype of DAG (Directed Acyclic Graph - directed graph with no directed cycles. It consists of vertices and edges (arcs), with each edge directed from one vertex to another, such that following those directions will never form a closed A directed graph is a DAG if and only if it can be topologically ordered by arranging the vertices as a linear ordering that is consistent with all edge directions).

**lib-hibon** - contains the hibon specification and all its structures. In Tagion we have the hibon and document.
The document is a structure with a buffer and can extract the necessary binary data set from this buffer. It is a HIBON structure wrapper over the buffer to extract the needed data. It pulls them out on the HIBON structure. The buffer is immutable data. We can pass it between streams and nodes. Hibon can be converted to a document and vice versa to create a new hibon.

**lib-logger** - the primary staging for all the services' loggers.

**lib-mobile** - lib for the mobile wallet we use on IOS and Android. That is just a wallet for mobile devices.

**lib-network** - a set of network functions that allow us to listen to external sockets and communicate with those who use us as a node. For example, any wallet that wants to send a transaction to a node sends it here. According to the classic client-server architecture, the node receives it thanks to Tagion Network, which listens for incoming connections.

**lib-option**s - all our high-level services use options. With these options, we can configure our network Tagion way, which is available for our internal services.
**p2pgowraper** - for hashgraph and communication inside the network. This wrapper over the lib golibp2p (external lib) module allows nodes to communicate via hashgraph & gossip protocols for communication between network nodes.

**lib-services** - is the primary storage for all the series loggers (a set of network functions that allow us to listen to external sockets and communicate with those who use us as a node) to a node that sends it to the node. According to the classic client-server architecture, the node receives it thanks to Tagion Network, which listens for incoming connections.
The system works due to concurrency programming. The whole set of services runs as separate threads of tasks. All work is due to communication. We're still talking about the same system running many services and threads that communicate.
The parent service **'p2pTagionService'** - starts all other ones.

#### The Node Stack

A Tagion Node is divided into units, and each unit handles a service function in the following manner: A smart contract is sent to the Transaction-service unit, fetching the inputs from the DART unit and verifying their signatures. The DART unit connects to other DARTs via the P2P unit. The transaction unit forwards the smart contract, including the inputs, to the Coordinator-unit, which adds to an event that is gossiped to the network via the P2P unit. When the Coordinator receives an event with a smart contract, it is executed via the ScriptingEngine-unit, and the result of outputs is verified. When the Coordinator finds an epoch, this epoch is forwarded to the Transcript-service-unit, which evaluates the correct order and requests the DART unit to erase the inputs and add the newly generated outputs. The Tagion Node service structure:
* Each service is running as independent tasks and communication between each other via commutation channels. The different services modules perform the service as described in the list below.
* Coordinator - This service manages the hashgraph-consensus and controls another related service for the node. The Coordinator generates and receives events and relays to the network. This service also generates the epoch and sends the information to the ScriptingEngine services.
* Transaction - This service receives the incoming transaction script, validates, verifies, and fetches the data from the DART, and sends the information to the Coordinator.
* DART - Services to the Distributed database.
* P2P - This service handles the peer-to-peer communication protocol used to communicate between the nodes.
* ScriptingEngine - Handles the executions of the scripts.
* Transcript - Services the Epoch and orders the script execution.
* Logger - The service handles the information logging for the different services.
* Monitor - The Monitor service is used to monitor the activities locally.
* HeartBeat - This service is only used in test mode. This service enables the nodes to execute sequentially, simplifying network debugging.

#### How it works
This structure uses the lib-network (so-called server). This service can listen to specific port requests: transactions or search in darts on hashes. We can send a contract to it, execute it, or send a list of public keys, which we want to check if there are any bills in the database. The Tagion Service sends this event with this contract to all network nodes, and it comes to a consensus and makes an epoch.
The internal 'transcript service is used for creating the epoch with the recorder for DART inside.
An **epoch** is a structure with a recorder and the number of the epoch. In this case, the node has reached a consensus on this data recorder.
Accordingly, this service gets this epoch; it checks all the data in this recorder, whether it can be added, and whether they are not added or deleted twice, for example. If the data checking is correct, the service sends a request to the dart service (right now, it's DARTSynchronizeServise, in the future DARTServise).

The **DART SynchronizeServise** has a database object responsible for all operations with this database. It can modify the data according to the request sent (add, delete, and return the DART data by appropriate hash).

##### Discovery services
When the node starts, it has to:
* Find other nodes (with their address) in the network.
* Synchronize the database to the current state.

There are the following modes for the network running:

* **Internal** - It runs as one program, where each node runs with five nodes (nodes). Each node in this program will be a separate thread, and they will communicate between threads simulating a network.
* **Local** - It runs the network on several nodes, but the service searches for nodes only in the local network. It will not find itself or find someone in the public network.
* **Public** - fully distributed.

Services depending to the start mode:

* **Internal** - MdnsDiscoveryService - scans the local network to find some nodes.
* **Local** - FileDiscoveryService The path to one file where all nodes should communicate the address, and through some delay, all nodes will read all lists of addresses.
* **Public** - ServerFileDiscoveryService -It is similar to FileDiscoveryService but sends data to one of our shared servers. There is a DART synchronization before starting the node when it becomes online. It must first find the network nodes and then synchronize the DART. We connect to different network nodes, and with them, we synchronize the missing data to receive some current DART state.

When the synchronization is finished, the hashgraph starts.