Short intro to IPFS

What is IPFS?

IPFS is a decentralized file distributing system based on peer-to-peer protocol, adding any content on it is like broadcast a unique self-describe torrent file to every peer connected and start to seed the file.
Files will remain on the internet until there's no node seeding it.
IPFS could be seen as a single BitTorrent swarm, exchanging objects within one Git repository.

Easy instruction

Useful command

ipfs init //initialize
ipfs id //view node id
ipfs add <path to Files/Folders> [-r] [-w] //add file to ipfs //-r for recursive -w for keeping directory structure
ipfs cat <ipfs-path> //view file 
ipfs get <ipfs-path> //download file
ipfs ls <ipfs-path-of-directory> //list file in directory

Going online

ipfs daemon //run ipfs online
ipfs name publish <ipfs-path> //using ipns feature, public files to your nodeid

after running daemon, you can use web console
http://localhost:5001/webui

Using official gateway to view files

https://ipfs.io/ipfs/<ipfs-path>
https://ipfs.io/ipns/<nodeid>/[path-to-file]

Official Demo

IPFS design

The IPFS Protocol is divided into a stack of sub-protocols:

• Identities
• Network
• Routing
• Exchange
• Objects
• Files
• Naming

reference:
https://github.com/ipfs/specs

Identities

Nodes are identified by NodeId, which is a hashed public key(default RSA2048),

key generation 的計算

type PrivateKey []byte
// self-describing keys
type Node struct {
    NodeId NodeID
    PubKey PublicKey
    PriKey PrivateKey
}

and required PoW while establishing it.(In order to prevent sybil attack)

https://www.twcert.org.tw/assets/pdf/SybiAttack.pdf

difficulty = <integer parameter>
n = Node{}
do {
    n.PubKey, n.PrivKey = PKI.genKeyPair()
    n.NodeId = hash(n.PubKey)
    p = count_preceding_zero_bits(hash(n.NodeId))
} while (p < difficulty)

p = count_preceding_zero_bits(hash(n.NodeId))
why?7/12討論

Upon first connecting, peers exchange public keys, and check:
hash(other.PublicKey) equals other.NodeId.
If not, the connection is terminated.

Network

firewall 可以過嗎? ->多分おｋ
7/12討論

12:42:06.468 DEBUG     engine: wants QmaBCkhaeaaCzEKK1K9xzL39vMaXjA6QU1aERqSoXfVJmm - 2147483617 engine.go:241                         
12:42:06.468 DEBUG     engine: peer <peer.ID SoLju6> wants QmaBCkhaeaaCzEKK1K9xzL39vMaXjA6QU1aERqSoXfVJmm ledger.go:80                 
12:42:19.996  INFO    bitswap: Sending block [Block QmaBCkhaeaaCzEKK1K9xzL39vMaXjA6QU1aERqSoXfVJmm] to <peer.ID SoLju6> wantmanager.go:117                              
12:42:23.881 DEBUG     engine: <peer.ID SoLju6> cancel QmaBCkhaeaaCzEKK1K9xzL39vMaXjA6QU1aERqSoXfVJmm engine.go:237                    
12:42:39.681 DEBUG        dht: <peer.ID boMjby> adding <peer.ID SoLju6> as a provider for 'QmaBCkhaeaaCzEKK1K9xzL39vMaXjA6QU1aERqSoXfVJmm'                              
12:42:39.681  INFO        dht: received provider <peer.ID SoLju6> for QmaBCkhaeaaCzEKK1K9xzL39vMaXjA6QU1aERqSoXfVJmm (addrs: [/ip4/127.0.0.1/tcp/4001 /ip4/104.236.151.122/tcp/4001 /ip4/10.12.0.5/tcp/4001 /ip4/172.17.0.1/tcp/4001 /ip6/::1/tcp/4001 /ip6/2604:a880:1:20::1d9:6001/tcp/4001 /ip6/fc3d:9a4e:3c96:2fd2:1afa:18fe:8dd2:b602/tcp/4001 /ip4/127.0.0.1/tcp/8081/ws /ip4/104.236.151.122/tcp/4001]) handlers.go:291

IPFS can use any network, it stores addresses as multiaddr formatted byte strings for the underlying network to use.
It can also connect to local network, and have local swarm peers.
For example :

SCTP/IPv4 connection
/ip4/10.20.30.40/sctp/1234/

SCTP/IPv4 connection proxied over TCP/IPv4
/ip4/5.6.7.8/tcp/5678/ip4/1.2.3.4/sctp/1234/

Routing

IPFS use a DSHT(Distributed Sloppy Hash Table) based on S/Kademlia and Coral.

Coral 的特性？ V.S. bitTorrent DHT
routing 的意義？ => Searching for node or data
7/12討論

DSHT:

A sloppy insert is akin to an append in which a replica pointer appended to
a “full” node spills over to the previous node in the lookup path.
A sloppy retrieve only returns some randomized subset
of the pointers stored under a given key.

http://iptps03.cs.berkeley.edu/final-papers/coral.pdf
IPFS DHT makes a distinction for values stored based on their size,
small values (equal to or less than 1KB) are stored directly on the DHT,
and for values larger, the DHT stores references, which are the NodeIds of
peers who can serve the block.
useful refs:https://github.com/ipfs/examples/blob/master/examples/data/readme.md
(1KB的小檔案快取目前並沒有實做)

1KB 是檔案大小還是 block? -> 檔案大小
一剛開始決定的 block 大小？
var BlockSizeLimit = 1048576 //1 MB
7/19會議

type IPFSRouting interface {
    FindPeer(node NodeId)
    // gets a particular peer's network address
    SetValue(key []bytes, value []bytes)
    // stores a small metadata value in DHT
    GetValue(key []bytes)
    // retrieves small metadata value from DHT
    ProvideValue(key Multihash)
    // announces this node can serve a large value
    FindValuePeers(key Multihash, min int)
    // gets a number of peers serving a large value
}

Block Exchange

清楚的示意圖
7/12討論

IPFS use a Bittorrent inspired protocal: BitSwap to exchange blocks.

It use a credit-like system to incentivice nodes to seed and punish the leechers.
Let the debt ratio r between a node and its peer be:

$$r=\frac{bytes\mathrm{\_}sent}{bytes\mathrm{\_}recv+1}$$
and the probability of sending to a debtor be:

$$P(send|r)=1-\frac{1}{1+exp(6-3r)}$$
The relation between

$P$ and

$r$:

So, the more bytes a peer send to you, the more probability you send bytes to him.

In the case that a node has nothing that its peers want (or nothing at all),
it seeks the pieces its peers want, with lower priority (cache) than what the node wants itself.
This incentivizes nodes to cache and disseminate rare pieces, even if they are not interested in them directly.
BitSwap nodes also keep ledgers accounting the transfers with other nodes.

懲罰機制?
7/12討論

type Ledger struct {
    owner NodeId
    partner NodeId
    bytes_sent int
    bytes_recv int
    timestamp Timestamp
}

Bitswap protocol specification :

// Additional state kept
type BitSwap struct {
    ledgers map[NodeId]Ledger
    // Ledgers known to this node, inc inactive
    active map[NodeId]Peer
    // currently open connections to other nodes
    need_list []Multihash
    // checksums of blocks this node needs
    have_list []Multihash
    // checksums of blocks this node has
}

type Peer struct {
    nodeid NodeId
    ledger Ledger
    // Ledger between the node and this peer
    last_seen Timestamp
    // timestamp of last received message
    want_list []Multihash
    // checksums of all blocks wanted by peer
    // includes blocks wanted by peer's peers
}

// Protocol interface:
interface Peer {
    open (nodeid :NodeId, ledger :Ledger);
    send_want_list (want_list :WantList);
    send_block (block :Block) -> (complete :Bool);
    close (final :Bool);
}

上方 Peer 程式碼 interface & struct？ -> 僅為示意,語法是錯的
7/19會議

Sketch of the lifetime of a peer connection:

1. Open: peers send ledgers until they agree.
2. Sending: peers exchange want_lists and blocks.
3. Close: peers deactivate a connection.
4. Ignored: (special) a peer is ignored (for the duration of a timeout)
   if a node's strategy avoids sending

Objects

IPFS builds a Merkle DAG^[1], a directed acyclic graph where links between
objects are cryptographic hashes of the targets embedded in the sources.
This is a generalization of the Git data structure.

It provides some useful propoties, including:

1. Content Addressing: all content is uniquely identied
   by its multihash checksum, including links.
2. Tamper resistance: all content is verified with its
   checksum. If data is tampered with or corrupted, IPFS
   detects it.
3. Deduplication: all objects that hold the exact same
   content are equal, and only stored once. This is par-
   ticularly useful with index objects, such as git trees
   and commits, or common portions of data.

Path to Files

After add file into IPFS, one can view the content by browsing gateway with

Format

/ipfs/<hash-of-object>/<name-path-to-object>

Example

/ipfs/XLYkgq61DYaQ8NhkcqyU7rLcnSa7dSHQ16x/foo.txt
/ipfs/<hash-of-foo>/bar/baz.txt
/ipfs/<hash-of-bar>/baz.txt
/ipfs/<hash-of-baz>

Naming

By using IPNS, one can access a mutable file with same link.
Example:

/ipns/XLF2ipQ4jD3UdeX5xp1KBgeHRhemUtaA8Vm/
/ipns/XLF2ipQ4jD3UdeX5xp1KBgeHRhemUtaA8Vm/docs
/ipns/XLF2ipQ4jD3UdeX5xp1KBgeHRhemUtaA8Vm/docs/ipfs

It also provide Peer link feature let users can link other users' objects directly into their own objects. [NOT implement]

Alice links to bob Bob
ipfs link /<alice-pk-hash>/friends/bob /<bob-pk-hash>

Eve links to Alice
ipfs link /<eve-pk-hash/friends/alice /<alice-pk-hash>

Eve also has access to Bob
/<eve-pk-hash/friends/alice/friends/bob

access Verisign certified domains
/<verisign-pk-hash>/foo.com

上傳的文件會存在多久？
有 url，有無可能完全存取不到？
7/19討論

---

1. 確保交易的順序 ↩︎