Decentralized compute platform

💭

In today's digital age, the amount of generated data is growing exponentially, and so to the amount of corresponding compute required for processing. Intensive numerical computing tasks run mostly in the cloud (AWS, Azure, GCP, …) or on in-house servers.

Distributed computing projects like Folding@home pool together the computational power of idle personal computers around the world to tackle key, compute-intensive scientific challenges such as screening new drug therapies. The combined compute created has been measured to surpass even the most powerful supercomputer available.

However, the success of these projects wax and wane with the enthusiasm and participation of volunteers, which is difficult to maintain if there's no clear research progress or common challenge. Can we provide economic incentives for volunteering compute?

IPFS / Filecoin provide marketplace for decentralized storage. Similarly, is it possible to provide compute in a decentralized and secure manner?

I contributed a bit to ConstitutionDAO last week, and even though it didn't work out as intended, it was mindblowing to see how all the contributions from everyone added up to so much (>40M).

In the spirit of "together > alone", I'm interested in what an experimental, proof-of-concept decentralized compute platform might look like.

The high-level idea is that large-scale compute tasks are split into smaller units, which are then submitted to a network of computers around the world. Successfully completed units are rewarded (probably with some token) proportional to the difficulty of the task. Some sort of marketplace could be created. Or an automated exchange for matching work units to available compute.

❓

How do you make make it easy to convert regular, compute-intensive programs into ones that can be easily divided up? Can we interleave bits of parallel computation (sent to the network) and sequential computation?

How do the incentives work? How much do I pay if I want to submit compute to be performed?

How do we verify that a work unit was successfully and correctly completed? With volunteer projects like F@h, there's no reason to game the system since there's no reward for completing a task. But if there is a reward, then people will probably want to game the system for their own profit (by submitting bogus results).

I'm curious if zk-SNARKs can be applied here in a general enough way to verify that an arbitrary computation (like calculating a gradient, perhaps written in some language like Python) was actually successfully completed.

How do you ensure that data is kept private and secure? For public datasets, it might not matter as much, but how do we perform computation on private data?

Compute requests will probably rely on external libraries, but not everyone will have them. How do we bundle / containerize code?

…Another thing to worry about later: besides being technically interesting, does this actually provide a better experience than using a regular cloud provider?

Following up (12/30)

I initially wrote this a month ago at the end of November (11/24). A few things I noticed later:

• I discovered STARKS / StarkWare and what they've built with Cairo, which enables them to use SHARP to prove that a Cairo program ran successfully. To me, this is pretty incredible. I'm not sure of the exact details of how that works, I think zkEVM aims to achieve the same thing but for Solidity (putting the EVM into a ZKP).