owned this note
owned this note
Published
Linked with GitHub
# What are we measuring
Codex: v0.1.6
Log message "Received blocks from peer" indicates that a node received a block. It identified the peer from which the block was received and which block was received. (dataset + index)
We are counting per node, how many times a specific block is received. We are disregarding which peer it was sent from.
# Test
In this test we create a swarm in which 1 node contains a file. The other nodes simultaneously begin downloading it.
File size: 1, 5, 10, 20 MB
Number of nodes: 3, 5, 10, 20
# Data

# Data (csv)
filesize,numNodes,recv1x,recv2x,recv3x,recv4x,recv5x,recv6x,recv7x,recv8x,recv9x,recv10x,recv11x,recv12x,recv13x,recv14x,recv15x,recv16x,recv17x,recv18x,recv19x
1,3,34,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
1,5,60,6,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
1,10,126,13,4,3,1,2,3,0,1,0,0,0,0,0,0,0,0,0,0
1,20,202,28,13,12,12,10,5,8,12,12,5,1,1,1,0,0,1,0,0
5,3,141,21,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
5,5,208,82,29,5,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
5,10,397,154,86,40,36,12,2,2,0,0,0,0,0,0,0,0,0,0,0
5,20,433,286,185,130,104,83,62,53,46,30,21,10,13,20,7,5,10,4,37
10,5,391,183,59,11,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
10,10,761,321,187,93,45,26,11,4,1,0,0,0,0,0,0,0,0,0,0
10,20,906,553,430,283,217,131,105,77,58,43,38,19,21,28,28,35,63,20,4
20,3,555,87,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
20,5,813,298,140,33,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
20,10,1490,631,361,208,96,55,36,11,1,0,0,0,0,0,0,0,0,0,0
20,20,1584,1225,857,517,371,267,212,157,138,92,71,86,80,83,91,180,59,24,5
# Download Flow
Inspecting the v0.1.6 implementation of Codex, we find the following sequence of events when download is initiated.
## 1. Fetch manifest
Manifest encoded as a single block is fetched using the download-CID. We proceed only after the manifest is received and decoded successfully. We will not consider erasure-coding because the tests have been performed without it.
## 2. Initialize stream
A stream object is initialized. This stream will transmit data to the downloading application.
## 3. Read loop
While the bytes streamed is less than the total bytes in the dataset, we use the bytes-sent counter to determine the block-index in the dataset. We ask for this block from the network-block-store. When received, we write the bytes from the block to the stream and advance the current position. Only after the block has been copied to the stream entirely do we ask the network-block-store for the next block.
# Requesting a block
We will take a closer look at the sequence of events occuring in the block exchange engine of Codex when a block is requested. This section considers only the retrieval of a single block.
## 1. Pending block
A pending-block handle is created to represent the block retrieval process. If one already exists, the existing one is used instead.
## 2. Cheapest, known peer
From already-connected peers, we select the ones we know have the block, and sort them by the price for the block. If none are available, DHT discovery is started asynchronously.
## 3. Peer selection
If we have any peers from the previous step, we select one semi-randomly. If we don't, we select one from *all* the peers known to the block exchange engine. If we don't have any of those either, we exit the process assuming that DHT discovery will yield peers some time in the future.
## 4. Once peer
Once a peer has been selected, we send a wantBlock to that peer. And we send a wantHave message to all other peers.
## 5. Response to the wantBlock message
When receiving a wantBlock message, the peer will send the block back if it has the block in local storage.
## 6. Response to the wantHave message
When receiving a wantHave message, the peer will send a blockPresence response. It contains for the blocks in the wantHave message an indicator if the block is present or not. Additionally, it contains all the block CIDs currently wanted by this peer.