# Waku 2023-H1 Progress Blogpost

As we move forward into Q3 of 2023, we're excited to share some of the progress the Waku team has achieved in the first half of the year. Although we've been busy on many fronts, our main focus was on the scalability of Waku protocols, specifically with a focus on Status Communities. In fact, we explicitly set ourselves the ambitious goal of providing a Waku network that can scale to ~10K _active_ users per Status Community by the end of Q2 2023 and provide the minimum number of network services to make such a community viable. Our belief was that the protocols, tools and solutions we design in the process, could then be applied to multiple Status Communities, allowing the overall Waku network to scale to very large numbers (1 million or more nodes). This post will explore some of the highlights and challenges we encountered on this journey.

## Laying the theoretical groundwork

We started by doing a [theoretical analysis (mathematical modelling)](https://hackmd.io/IYmyyUIdSEuRZ7WbYd6dng?view) of GossipSub scaling under certain conditions. GossipSub is the underlying libp2p protocol on which the backbone of the Waku network, Waku Relay, is built. The numbers for message rates and sizes chosen for the model, comes from the telemetric analyses of typical traffic on very large Discord servers and existing Status Communities. From this we concluded, that the Waku Relay network can scale to 10,000 nodes per shard. If we make the further assumption that each Status Community operates on a separate shard, each community can scale to 10,000 _active_ relay-only users while bandwidth usage remains within a reasonable range for each user. Realistically most community nodes aren’t active simultaneously, so this approach could scale to ~100,000 nodes per community if roughly 1/10th of users are active at any time.

## Simulations: bridging theory to practical implementation

To test our theoretical conclusions, the Distributed Systems Testing (DST) team ran extensive simulations using a Waku simulation tool built on the Kurtosis framework, [Wakurtosis](https://github.com/vacp2p/wakurtosis). While achieving a full-scale simulation of 10,000 nodes poses challenges, initial simulations of up to 1000 nodes aligned well with our mathematical models, giving us confidence in the viability of our approach. In fact, the DST team has recently published a [summary report](https://docs.google.com/document/d/1U3bzlbk_Z3ZxN9tPAnORfYdPRWyskMuShXbdxCj4xOM/edit) detailing conclusions about Waku scalability, matching our theoretical expectations while showing that the nwaku client can deal with the resource overhead in a growing network.

## Telemetry and application measurements

Having drawn some theoretical conclusions under which a Waku Relay network will scale and with some practical experimentation to back these up, we also did a [telemetric analysis](https://github.com/vacp2p/vac-book/blob/main/scratch/status-scaling/status-telemetry-analysis.md) of current Status Community traffic to determine if Status Community usage of Waku is scalable. This allowed us to discover potential bottlenecks when sending large messages over Waku and propose some strategies to help Communities use Waku in a reasonable way without compromising on its scalability.

## Scaling through static sharding

Based on the analyses above, we published [an RFC](https://rfc.vac.dev/spec/57/) bringing together an integrated strategy to scale Status Communities based on static sharding. This requires manually selecting and configuring static shard(s) per Community, with no sharing of shards between communities. Static sharding provides a simple way to achieve scalability with reasonable DoS protection (see below), but also requires some trade-offs in decentralization and anonymity. For that reason, our longer term strategy is to provide a way to auto-scale the Waku network with "autosharding" - a way for multiple applications and communities to share the same set of shards, without having to intervene manually in the underlying message routing strategies. [This Github issue](https://github.com/waku-org/research/issues/3) provides more information on our network strategy going forward.

## DoS protection

One of the most important considerations of any Waku network deployment, is protecting the network against spam and Denial of Service (DoS) attacks. While we are still working on productionizing a [privacy-preserving spam protection mechanism based on rate-limiting nullifiers](https://rfc.vac.dev/spec/17/), we focused in H1 on developing a simple DoS mitigation mechanism for Status Communities. This requires all messages belonging to a community to be signed with a key distributed to all community members. Since static sharding ensures messages are only routed within a community, each relayer in the Waku network can validate that each routed message truly originated from within the community and discard those that didn't.

## Improving services: store and light protocols

To cater to the needs of a growing network of users on diverse platforms, the team improved (and in some cases completely redesigned) our existing protocols based on the lessons we've learned from dogfooding the Waku network. Of these, the changes to the store and filter protocols were the most significant. The store protocol enables users to retrieve historical messages from service nodes in the network. To serve a large number of such history queries, we've transitioned from a simple SQLite backend to a more robust PostgreSQL archive. This allows store nodes to serve multiple queries in parallel, while also improving message reliability through redundancy. The filter protocol allows resource-restricted devices to receive only a subset of messages being routed in the network. We've redesigned [the entire protocol](https://rfc.vac.dev/spec/12/) to be more robust and provide features requested by Waku-based applications, such as refreshing existing subscriptions. The new version of the filter protocol is currently being dogfooded in a production environment and initial feedback is great. We're also productionizing a discovery mechanism suitable for resource-restricted nodes, called [Waku peer exchange](https://rfc.vac.dev/spec/34/). Work on all these protocols empowers nodes, including resource-restricted devices, to seamlessly participate, ensuring that the network remains accessible to all.

## Improving connectivity: discovering and managing peers

A Waku network can only work well, if we make it easy for nodes to connect and interact effectively. In H1, we spent a significant amount of effort improving how a node discovers peers and manages its connectivity to those peers. We've boosted a node's ability to differentiate between "good" and "bad" connections, including integrating [gossipsub peer scoring](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md#peer-scoring). Nodes behind restrictive NATs can now make themselves discoverable, thanks to the introduction of [libp2p rendezvous discovery](https://rfc.vac.dev/spec/57/#libp2p-rendezvous-and-circuit-relay), and connect to other nodes by coordinating a NAT hole punch via a [circuit-relay connection](https://rfc.vac.dev/spec/57/#libp2p-rendezvous-and-circuit-relay). We've also modified other discovery methods, such as discv5, to consider static sharding when filtering discovered nodes.

## Making it easier to build on Waku

We're continuing to make building on Waku more accessible than ever before. The nwaku client has recently seen the introduction of [a C API](https://github.com/waku-org/nwaku/tree/b9d5d28af5afcc2589357a132c9b5beb57202828/examples/cbindings) that can be seamlessly wrapped in multiple programming languages (see the [Python wrapper](https://github.com/waku-org/nwaku/tree/b9d5d28af5afcc2589357a132c9b5beb57202828/examples/python) as an example). By simplifying the process of integrating nwaku as a service node into other applications, we're fostering an environment where more projects can actively build on Waku and contribute to the Waku network. We've also redesigned and improved our [docs website](https://docs.waku.org/), making it easier than ever to get acquainted with Waku.

## Setting the Stage for Growth

This post provides just a glimpse into the progress the Waku Team has made in the first half of 2023. Our sights are now set on taking what we've learned and building a [viable public Waku Network](https://github.com/waku-org/research/issues/1)that can scale to even larger numbers, while maintaining our core principles of open-access, privacy-preservation, decentralization and transparency about trade-offs. As a team we remain dedicated to this mandate, while acknowledging the huge involvement of community members and collaborators who contribute to our journey. Stay tuned for more exciting updates and innovations as we chart the course for the future of Waku!