# Slides
<!-- Put the link to this slide here so people can follow -->
slide: https://hackmd.io/@adin/H18k1e7OK#/
---
### Infra Network Infrastructure
- Help the browsers :newspaper:
- Public HTTP Gateways
- Preload Nodes
- Delegated Routers
- Help the network :spider_web:
- DHT Bootstrappers
- DHT Hydra Nodes
---
## Help the browsers
- Restricted transports
- Limited resources
- Requires HTTP foothold
---
### Background: HTTP Gateways
- IPFS :bridge_at_night: HTTP
- Interoperability Layer
- http://mygateway/ipfs/bafy-some-cid
- http://localhost:8080/ipfs/bafy-some-cid
---
### Public HTTP Gateways
- Help users without IPFS available in their environment
- Many public gateways https://ipfs.github.io/public-gateway-checker/
- PL runs ipfs.io and dweb.link - same machines
- dweb.link (newer)
- supports bafy-some-cid.ipfs.dweb.link
- *.dweb.link public suffix list
---
### Preload Nodes
- Help users without good data transports available in their environment
- Restricted remote control of a fuller node
- Difference from Public Gateways
- Will cache + advertise data for you
- Incrementally verifiable downloads of data
- More controlled downloading of data
- not just file, folder, full DAG
---
### Delegated Routing
- Help users without good data transports, or limited resources
- Can do routing requests on your behalf
- At the moment these are the preload nodes
- In the middle of building an alternative protocol for this
---
## Help the network
- Critical to the network that someone runs it
- Bonus helpers
---
### DHT Bootstrappers
- Just a DHT node many people know about
- Need a foothold into the system to learn about others
- Browsers/HTTP have CA's, DNS resolvers, etc.
- Nothing special about them, but they're in go/js-ipfs by default
- Anyone can run one
- We're asking more ecosystem partners to
---
### DHT Hydra Nodes
- Optimized group of DHT server nodes
- Previously important to network stability
- No longer required
---
### DHT Hydra Nodes Cont'd
- Why do we still have them around?
- Can help optimize network behavior
- Helpful for measurements
- Want to donate resources to the network
- Ongoing work on an alternative to the IPFS Public DHT designed for this
- Does/should other groups be running these?
- Not really, if people want to donate to the network they should use ^
---
### Questions ❓
---
# IPFS debug 101
- "It's slow" 🦥
- "It won't load" :skull_and_crossbones:
---
## Baseline Questions ❓
- What's "it"?
- Resolution via the IPFS public gateways, your go-ipfs node, etc.
- What makes you think it should be faster?
- Data is on my laptop which is performing well
- Data is in go-ipfs node in the cloud
- Data is on pinning service
- If you can't answer these it's generally a bad sign
---
## Some common pitfalls
- You think IPFS is a magic free storage machine in the cloud
- You are unreachable over the public internet
- Your data isn't advertised in the DHT
- Your upload speed is very low
---
## IPFS Intuition
- go/js-ipfs just want to get you your data
- Sometimes they'll cheat to get it for you
- Check worst-case paths
---
## I know where the data lives
- If it's my machine let's see if I can grab the CID from it
- IPFS Check (https://ipfs-check.on.fleek.co/)
---
## I know I paid someone, but that's it
- `ipfs dht findprovs`
- Is the data in the DHT
- `ipfs id <peerID>`
- For each of those peers what's it's addresses
- `vole bitswap check <cid> <multiaddr>`
- Let's take a good looking address and try it out
---
## Resource Reminders
- http://ipfs-check.on.fleek.co/
- https://github.com/aschmahmann/vole
---
### Questions ❓
---
- Let's try some baseline questions:
- What's "it"?
- Resolution via the IPFS public gateways, your go-ipfs node, etc.
- What makes you think it should be faster?
- I'm hosting a small file on my home laptop
- If you have 1mbps upload from your laptop and you're the only one with the content how could the data download from gateways any faster than that?
- I've paid a service to hold my data
- running go-ipfs in some cloud infra with plenty of bandwidth
- a pinning service
- If you can't answer this question it's generally a bad sign
- e.g. you think IPFS is a magic free storage machine in the cloud
- Ok, so now you've passed the basic sanity checks now it's onto some technical gotchas:
- If it's my machine let's see if I can grab the CID from it
- IPFS Check
- Specifically made tool because with go-ipfs any tooling that isn't explicitly meant for debugging tends to be smarter than you'd probably want for debugging (e.g. get data from anywhere, peers can dial you instead of you dialing them, etc.)
- Let's do a run through
- Ok, it's not on my machine (e.g. it's Infura's) well this can be a bit annoying with existing tooling, but let's try:
- `ipfs dht findprovs`
- Is the data in the DHT
- `ipfs id <peerID>`
- For each of those peers what's it's addresses
- `vole bitswap check <cid> <multiaddr>`
- Let's take a good looking address and try it out
- Almost everyone who reports "It's slow" has their problem resolved if all the boxes are checked (and their upload bandwidth isn't terrible)
- IPNS is never resolving
- There's no IPNS Check tool yet, but it's easier
- `ipfs dht get /ipns/<record>` if it's missing then probably the author is offline
- While it's not prohibited at all by the protocol go-ipfs happens to not have a way to indicate "I'd like to keep this IPNS record published by someone else alive"
- Unfortunately there are no public IPNS pinning services yet, although a basic one would be trivial to implement
- IPNS is slow
- Unfortunately, the DHT client implementation is slow for IPNS operations. For now the answer is use the accelerated DHT client, and/or IPNS over PubSub
- If you're using the gateways then it's extremely likely that really IPNS is fast but the data it's pointing to is "slow" (see above)
- If you use the accelerated DHT client then `ipfs name resolve` should be fast
---
## Who am I?
- Front-end developer
- VSCode :heart:
- I use tabs. :cat:
---
### 70% of our users are developers. Developers :heart: GitHub.
---
{%youtube E8Nj7RwXf0s %}
---
### Usage flow
---
```graphviz
digraph {
compound=true
rankdir=RL
graph [ fontname="Source Sans Pro", fontsize=20 ];
node [ fontname="Source Sans Pro", fontsize=18];
edge [ fontname="Source Sans Pro", fontsize=12 ];
subgraph core {
c [label="Hackmd-it \ncore"] [shape=box]
}
c -> sync [ltail=session lhead=session]
subgraph cluster1 {
concentrate=true
a [label="Text source\nGithub, Gitlab, ..."] [shape=box]
b [label="HackMD Editor"] [shape=box]
sync [label="sync" shape=plaintext ]
b -> sync [dir="both"]
sync -> a [dir="both"]
label="An edit session"
}
}
```
---
### Architecture of extension
---
---
## Content script
- Bind with each page
- Manipulate DOM
- Add event listeners
- Isolated JavaScript environment
- It doesn't break things
---
# :fork_and_knife:
---
<style>
code.blue {
color: #337AB7 !important;
}
code.orange {
color: #F7A004 !important;
}
</style>
- <code class="orange">onMessage('event')</code>: Register event listener
- <code class="blue">sendMessage('event')</code>: Trigger event
---
# :bulb:
---
- Dead simple API
- Only cares about application logic
---
```typescript
import * as Channeru from 'channeru'
// setup channel in different page environment, once
const channel = Channeru.create()
```
---
```typescript
// in background script
const fakeLogin = async () => true
channel.answer('isLogin', async () => {
return await fakeLogin()
})
```
<br>
```typescript
// in inject script
const isLogin = await channel.callBackground('isLogin')
console.log(isLogin) //-> true
```
---
- Public HTTP Gateways
- People have built lots of tooling around HTTP so an HTTP gateway is effectively an interoperability layer that lets you send a request `http://mygateway/ipfs/bafy-some-cid` and have it resolve for you
- Anyone who is running go-ipfs will by default have a local gateway accessible at 127.0.0.1:8080 (e.g. http://127.0.0.1:8080/ipfs/bafy-some-cid will load that CID just fine)
- What's an HTTP Gateway?
- People have built lots of tooling around HTTP so an HTTP gateway is effectively an interoperability layer that lets you send a request `http://mygateway/ipfs/bafy-some-cid` and have it resolve for you
- Anyone who is running go-ipfs will by default have a local gateway accessible at 127.0.0.1:8080 (e.g. http://127.0.0.1:8080/ipfs/bafy-some-cid will load that CID just fine)
- The point of these is to help users who do not have access to IPFS software in their environment to still be able to retrieve data
- Many people run public gateways https://ipfs.github.io/public-gateway-checker/
- We run two:
- ipfs.io (older)
- dweb.link (newer)
- Is nice because it supports bafy-some-cid.ipfs.dweb.link which provides origin isolation. Also, *.dweb.link is in the public suffix list so we have announced to everyone are intentions as to how we're using it
- Preload nodes
- These are effectively nodes that the community is allowed to ask to do some work for them
- The point of these is to help users who do not have access to a sufficient set of capabilities to build compatible IPFS software (e.g. web browsers which have very limited data transports) gain greater access to the network
- They allow you to ask the node to cache and advertise your data for a while, unlike gateways which make no such guarantees
- They allow for verifiable downloads of data which isn't fully available over gateways
- They allow for more controlled downloading of data (gateways currently only allow download files, tar's of entire folders, or CAR's of entire DAGs)
- Delegating routing
- These are nodes that can be asked to do routing requests on behalf of other nodes
- The point of these is roughly the same as the preload nodes
- At the moment these are the preload nodes
- DHT Bootstrappers
- Any system needs some foothold to learn about the network (CA's built into browsers/OS's, DNS resolvers, etc.), bootstrappers are these footholds for the IPFS public DHT
- When a go-ipfs node starts up it connects to the bootstrappers and then does some queries to learn about the rest of the network
- There's nothing special about the PL bootstrappers, if you don't like them or can't reach them you can configure any that you like, even a standard go-ipfs node would do
- We're trying to get more folks in the ecosystem to run these to prevent any centralized infra issues from disrupting use of the p2p network
- Hydra DHT nodes
- These are DHT server nodes that have been optimized to work together (Sybils for good)
- These (and their less optimized predecessors) were previously needed for network stability, although the network is fine without them. We've had outages of these in the past and the network has been fine
- So why do we still have them around?
- They can optimize network queries by making information available earlier than it's supposed to
- They can be a useful tool for network measurements, although less blunt tools could be used to greater effect here
- We don't currently have a good way for some benevolent actors (e.g. PL) to donate content routing resources to the network
- There is work on a routing system that is particularly designed for this in mind
- Should other groups be running these?
- Not at the moment. It's currently helpful that the hydras occupy ~5% of the network space and can therefore help out on almost every query. If more groups started doing this we'd end up competing for network space.
- PL isn't getting much benefit out of the hydras, it's mostly a benevolent donation and the areas where it is useful to us we are actively moving towards the hydras not being needed
---
# :100: :muscle: :tada:
---
### Wrap up
- Cross envornment commnication
- A small library to solve messaging pain
- TypeScript Rocks :tada:
---
### Thank you! :sheep:
You can find me on
- GitHub
- Twitter
- or email me
{"metaMigratedAt":"2023-06-16T14:45:36.327Z","metaMigratedFrom":"YAML","title":"IPFS Network Infra","breaks":true,"description":"What types of public IPFS infrastructure is run and why?","contributors":"[{\"id\":\"6c027841-8e73-4a4d-8ec7-e99a5271fad7\",\"add\":21708,\"del\":8969}]"}