# Data Anywhere and the Serverless Bazaar

"The network is eating the cloud." — This statement is not true yet, but it will be.

Paradigm shifts in computing come when the new ways make the old abstractions seem like a waste of developer attention. The rollout of a new paradigm is measured in the half life of technical debt. Serverless was how the cloud won over the previous hardware-first paradigm. The

What does it mean for the network to eat the cloud? When we say [software is eating the world](https://a16z.com/2011/08/20/why-software-is-eating-the-world/) we mean every business is turning into a software business. As the network eats the cloud, the attention developers put into cloud templates and deployment cycles will move to another abstraction, where applications are built in terms of data and compute, not microservices and VPNs.

One of the key drivers of epochal shifts in computing is the [commoditization of complements](https://www.gwern.net/Complement). By making complementary products and services more affordable and accessible, companies can tap into new markets and drive growth. For example, the commoditization of complementary products and services such as storage, networking, and software has helped fuel the growth of the cloud computing market.

As the network commoditizes the cloud, it will do so by offering new abstractions that might be as hard to imagine today, as mature serverless was at the dawn of EC2. Commodity servers brought us the cloud. The mature cloud offers serverless interfaces. The first generation of serverless interfaces run in the cathedral, in the walled gardens of cloud service providers. But as serverless APIs mature they lay the groundwork for the next level of commoditization, from the server to the cloud. Commodity cloud accessed via cryptographically verifiabiable invocations (in the style of UCAN) will bring us to a post-cloud model that is driven by the concepts of data anywhere and usher in the serverless bazaar.

As hard as serverless was to imagine in the pre-devops days of the original cloud, that is how hard we should expect it to be to imagine what a mature data anywhere world looks like.

The protocols supporting this network of function execution engines and data storage providers are becoming an important locus of innovation, while at the same time the obvious choice for certain business use cases, because these abstractions mean lower operational burden. As protocols like IPFS and web auth mature, they are getting cheaper, easier, faster, and more reliable. At some time they will cross a threshold where they become the logical choice for application developers. I think that threshold is now -- platform reach has hit a critical mass with IPFS in browsers, Filecoin storage capacity, and an expanding list of protocol implementations and users. Ecosystem players are moving up the stack, coordinating around application protocols like UCAN. This is the beginning of a golden era, and it is a great time to get involved as an app dev.

What does this golden era look like? Data anywhere means code doesn't reflect where the data or application is hosted, it just asks for the data, and the fastest or cheapest provider wins. This gives tremendous fleixbility when it comes to migrating workloads, building remix apps, or collaborating with data scientists. In the IPFS world, data anywhere is primarily a consequence of the way CIDs (hash-based content identifiers) reverberate into application code.

In this environment, the provider of the compute may not be the provider of the database, and the provider of the database likely doesn't manage the user's keyring. The provider of the storage wont' be the provider of the messaging. This encourages specialized service providers to emerge, who can compete by offering cost-effective, innovative solutions to a wide range of problems. This all depends on verifiable services enabled by cryptographic proofs. This world of trustless UCAN invocations has the power to move computation from the cathedrals of large cloud providers, to the serverless bazaar, where anyone can add functions to the network, and anyone can call them.

This world has appeal not just because of it's flexiblity, but because networks win over clouds for operational reasons. Imagine cost-competitive alternatives to S3 or Lambda, but with the unstoppable characteristics of global distributed blockchains etc. Managing uptime is a lot easier when your data centers are run by a worldwide network of independent providers, and your workloads are written to take advantage of data anywhere and verifiable compute. Protocols like these will be the preferred launchpad of the next generation of viral apps, precisely because they are "already scaled" and don't require deep infrastructure knowledge to leverage.

This shift, away from heavily attended one-off implementations, to self-running network protocols, is part of a broader trend that has been going on for a long time. The shift from mainframes to workstations to personal computing and then the web is well known. The shift from the cloud to the nework is similar. When systems don't have a central administrator their operational burden is commoditized, so these applications are even simpler to maintain and administer than the serverless applications that came before them. Just as the cloud removed the burden of the datacenter, these protocols remove the burden of the cloud. As protocols start to offer competitive features, price, and performance, it will start to seem an unwieldy distraction to pamper centralized cloud templates, function runtimes, and container clusters. As universal run time protocols like WASM mature, why would you choose architectures that depend on running in a particular cloud?

Data anywhere and the serverless bazaar

---

This is the beginning of a new era of computing, where verifiable data and verifiable compute become ubiquitous, and a wide range of innovative service providers emerge to offer low-cost, high-performance solutions to their customers. In this world, there are no centralized data stores, no centralized compute engines, no centralized key management services, identity providers, storage. No centralized anything.

This is a world where the cloud doesn't exist, because it's not necessary. This is a world of data and compute, accessed in a location independent manner. This is a world of decentralized data and decentralized compute, accessed in a decentralized manner. This is a world where applications move from the cathedral of cloud providers to the bazaar of innovative verifiable service providers. In this world, there is no such thing as "the cloud". There is only a network of invocations, proofs, and verifiable reciepts.

This environment offers social media and web builders flexible business models that can blend user-paid and app-paid services. As verifiable data and compute become mainstream, applications can move from the cathedral of cloud providers to the bazaar of innovative verifiable service providers. These applications written in terms of data and compute, which is accessed in a location independent-manner. With a small extension to our programming paradigm to allow for more flexible addressing, the cloud dissolves into a network of invocations, proofs, and verifiable reciepts.

Any technology that relies primarily on a location-based paradigm is on the chopping block. Centralized data silos and access control mechanisms smell the ripest, as they bake non-functional implementation, maintenance, operations, and user experience costs into their architecture.

The history of computing is full of examples of new technologies that made location-based approaches obsolete, such as:
* the PC and LAN (vs. mainframe and dumb terminal)
* the Internet (vs. closed network)
* the web (vs. walled gardens)
* the mobile (vs. desktop)

We are in the midst of a similar technology shift. Serverless is built on the principle that service provisioning, monitoring, maintenance, and operations can be managed separately from the application codem, allowing developers to focus their attention on application code and business goals.

The habit of using serverless APIs opens the door for portable code that runs in a location independent manner. CIDs and UCANs (a self-sovereign authorization standard) allow for another layer of abstraction, where code can have logical service dependencies without depending on who runs those services.

When systems don't have a central administrator their operational burden is commoditized, so these applications are even simpler to maintain and adminster than the serverless applications that came before them. Just as the cloud removed the burden of the datacenter, these protocols remove the burden of the cloud. As protocols start to offer competitive features, price, and performance, pampering centralized cloud templates, function runtimes, and container clusters will start to seem an unwieldy distraction.

This shift, away from heavily attended one-off implementations, to self-running network protocols, is part of a broader trend that has been going on for a long time. The shift from mainframes to workstations to personal computing and then the web is well known. As universal run time protocols like WASM mature, why would you choose architectures that depend on running in a particular cloud?

# Part 2

One of the key drivers of epochal shifts in computing is the [commoditization of complements](https://www.gwern.net/Complement). By making complementary products and services more affordable and accessible, companies can tap into new markets and drive growth. For example, the commoditization of complementary products and services such as storage, networking, and software has helped fuel the growth of the cloud computing market.

Commodity servers brought us the cloud. The mature cloud offers serverless interfaces. The first generation of serverless interfaces run in the cathedral, in the walled gardens of cloud service provider. But as serverless APIs mature, they become more similar, enabling a free for all as maybe players join the bazaar of serverless operators. This enables the next level of commoditization, from the server to the data center. Commodity data centers accessed via cryptographically verifiabiable invocations (in the style of UCAN) will bring us to a post-cloud model that is driven by the concept of data anywhere. We are only at the very beginning of this model, but we can do our best to see where it is going.

As hard as serverless was to imagine in the pre-devops days of the original cloud, that is how hard we should expect it to be to imagine what a mature data anywhere world looks like. (Luckily, yours truly wrote one of the very first serverless implementation, without knowing that's what it was. So maybe there is hope to stumble onto the right course.) Let's try to imagine this world.

What would be the performance implications of a data anywhere world? If data is always stored on a local device, and data processing is done on that same device, then the data is only as fast as the device. If data is stored on a central server, then it is only as fast as the server. If data is stored on a distributed set of devices, and data processing is done on a set of devices that are close to the data, then the data is as fast as the fastest nearby device. This would be more secure, because it would be more difficult to hack all of the devices in the system. It would also be more reliable, because if one device failed, the others could still access the data.

How would the relationship between applications and users change in a data anywhere world? The application would be made of of invocations, proofs, and verifiable reciepts, all of which could be authored and managed by different services. The user could own the accounts that data resides in, granting access for applications to provide adittional value. To read one way to do this today, [check out our blog on UCAN delegation.](https://blog.web3.storage/posts/ucan-delegation-with-w3up)

So how to do we get from here to there?

1. Address data and compute in a location-independent manner.
2. Commoditize the operational burden of data and compute.
3. Secure data and compute in a location-independent manner.
4. Make distributed data and compute easier to use.

The first two points are handled by the foundation on which web3.storage is built: IPFS and Filecoin. Points three and four are the focus of our current work. Our work with teams like Fission and BlueSky on the UCAN protocol ... Our application component library, w3ui, puts developer experience first.... Mention socket supply....

## Technologies used in the prototype

The prototype is built using a combination of existing technologies and experiments. It's based on the following open standards and protocols:

* [OpenID Connect](https://openid.net/specs/openid-connect-core-1_0.html) for identity provider implementation
* [Delegated Authorization](https://tools.ietf.org/html/draft-ietf-oauth-dyn-reg-24) for client registration protocol
* [JSON Web Token (JWT)](https://tools.ietf.org/html/rfc7519) for identity, delegation, and claims
* [JSON Web Signature (JWS)](https://tools.ietf.org/html/rfc7515) for cryptographic signatures
* [JSON Web Key (JWK)](https://tools.ietf.org/html/rfc7517) for key management and exchange
* [JSON Web Encryption (JWE)](https://tools.ietf.org/html/rfc7516) for encryption and decryption
* [Key Management Interoperability Protocol (KMIP)](https://www.oasis-open.org/committees/

Outline:

* I. Introduction
* A. “The network is eating the cloud”
* B. Paradigm shifts in computing
* C. Commoditization of complements
* II. What Does It Mean for the Network to Eat the Cloud?
* A. Every business is turning into a software business
* B. Applications built in terms of data and compute, not microservices and VPNs
* III. Commoditization of the Cloud
* A. New abstractions that are hard to imagine today
* B. Commodity cloud accessed via cryptographically verifiable invocations (UCAN)
* IV. Data Anywhere and the Serverless Bazaar
* A. Flexibility and lower operational burden
* B. Protocols like IPFS, web auth, UCAN, WASM, etc. becoming cheaper, easier, faster, more reliable
* C. Social media and web builders flexible business models
* D. Decentralized data and compute accessed in a decentralized manner
* E. No centralized anything
* V. Technology Shift from Location-Based Paradigms
* A. PC/LAN vs mainframe/dumb terminal
* B. Internet vs closed network
* C Web vs walled gardens
* D Mobile vs desktop
* VI Serverless APIs Open Door for Portable Code
* A Serverless applications simpler to maintain and administer than before
* B Cloud removed burden of datacenter; protocols remove burden of cloud
* C Protocols start to offer competitive features, price, performance
* D Shift away from one-off implementations to self-running network protocols
* VII Performance Implications of Data Anywhere World
* A Data as fast as local device or server depending on where stored
* B More secure & reliable if stored on distributed set of devices
* VIII Relationship Between Applications & Users Change
* A Invocations, proofs & verifiable reciepts authored & managed by different services
* B User owns accounts that data resides in granting access for applications to provide value
* IX How Do We Get From Here To There?
* A Address data & compute in location-independent manner
* B Commoditize operational burden of data & compute
* C Secure data & compute in location-independent manner
* D Make distributed data & compute easier to use
* X Conclusion
* 1. Web3.storage is built on two powerful foundations: IPFS and Filecoin.
* 2. We're currently focusing on points three and four, working with teams like Fission and BlueSky on the UCAN protocol.
* 3. We also have an application component library, w3ui, that puts developer experience first.
* 4. And don't forget about socket supply! It's an important part of the web3.storage ecosystem.
* 5. Together, these components create a powerful platform for developers to build on top of web3.storage. 6. With web3.storage, developers can create secure, decentralized applications quickly and easily!