# General Grant Proposal: Zero Knowledge Photograph - **Project Name:** Zero Knowledge Photograph ## Project Overview :page_facing_up: ### Overview AI-generated content is everywhere. While the advancement in generative AI helps us unlock new possibilities, malicious uses of such technology may put us at risk of being flooded by disinformation. Being able to detect such disinformation will certainly be one of the critical problems to solve in the distributed and trustless world of web3. Blockchain technology, especially zero knowledge protocols, will be playing an integral part to stop the spreading, or even the creation of distorted information. In this project, we will be focusing on proving the trustworthiness of digital photographs (photos afterwards), one of the most widely-used types of media in this era of digital information. Utilizing ZK-SNARKs and account abstraction, our app enables end-users to prove and verify the legitimacy of photos while protecting user privacy on a public blockchain. ### Motivations / Problems When it comes to digital photos, it is said to be extremely difficult to digitally detect deep fakes, or even edits applied to the original photo. There's no other way but to manually look for a hint of artificiality in a photo to convince ourselves that the photo is taken by an actual camera or that only trivial (non-disinformed) edits are applied to the original photo. More specifically, there exist at least the following major problems with respect to proving the legitimacy of digital photos. - Easy image forgery - We should be assuming that end-users have access to software that is capable of modifying image itself and its metadata - There is no practical and trustless ways to prevent end-users from using the distorted photos on the internet (referring to the centralized web2) - Easy impersonation on social media - There is no guarantee that uploaded photos are either 1) taken by an end-user themselves, 2) shared and approved to be posted by someone an end-user has connection with, or 3) created by an end-user themselves (**NOT completely-generated by some generative AI model**) That being said, it is becoming systematically difficult to discern what is true or false on the internet, resulting in lowering the quality of services, or relying more on centralization for censorship. A public blockchain as well as some cryptographic techniques come into play here, realizing private proving of trustworthiness of digital, human-taken photos. ### Solution Ideas A photo is stored in a computer as digital data, often as a matrix of numeric values. Edits to such digital photo, whatever tools are used, are thus some computation on the underlying matrix. We are able to prove the execution of computation on matrices with the appropriate arithmetic circuits, because ZK-SNARKs use circuits to prove the validity of the execution of some computation from one state to another state. In other words, with ZK-SNARKs, we are able to prove the transition from private state "A" to public state "B", without disclosing the private state. In the case of photo editing, inputs to the circuit would be 1) the original (non-edited) photo (private input) and 2) type of edit applied to the original photo (public input), while output of the circuit would be the edited photo. By whitelisting the types of photo editing that are considered to be non-distorting, we are able to narrow down the ways of possible photo editing. However, in order to prevent photo-based disinformation, it is not enough to verify the validity of photo edits. This is because we are not yet proving the **originality** of the photo given to a circuit as a private input, meaning we should additionally be able to ensure that the given photo 1) is taken by a real camera and not generated by generative AI and 2) is owned by some end-user(s). For the first point, the simplest approach is to develop a photo taking & uploading web app that disables the typical "upload-from-pictures" functionality. For the second one, we should be using smart contract based backend, which contains metadata similar to what's defined in [ERC-721](https://eips.ethereum.org/EIPS/eip-721), but also including owner's signature over the original image. In this way, it is possible to prove the execution of the whitelisted edits on the original image, while keeping track of its provenance, and more importantly, shielding it from the public. ### Deliverable Details The expected functionalities of the deliverables are as follows. To achieve them, we are planning to implement arithmetic circuits (in [circom](https://github.com/iden3/circom) or [halo2](https://github.com/zcash/halo2)), smart contracts (in [foundry](https://github.com/foundry-rs/foundry) or [hardhat](https://github.com/NomicFoundation/hardhat) & Solidity), and demo client web app (in React & TypeScript). #### 1. Photo Taking ![grant-proposal-photo-taking](https://hackmd.io/_uploads/S177TMQB6.png) We are planning to implement the following to prove the originality and ownership of a taken photo. - Client App - connects wallet (or uses account abstraction technique to hide user info from the public) - takes a photo - generates signatures over the taken photo and each of its metadata - Smart Contract - stores user info - stores signatures of the taken photo #### 2. Proof of Photo Edits ![grant-proposal-proof-of-photo-edits](https://hackmd.io/_uploads/SysXaMQSa.png) We are planning to implement the following to prove the validity of edits applied to the original photo. - Client App - sends edited photo with signature of the original photo to smart contract (note that an end-user must provide valid signature of the original photo in order to prove the validity of the edits) - uploads edited photo to some of the p2p database (e.g. [IPFS](https://ipfs.tech/)) - Smart Contract - verifies signature of the original photo - generates a ZKP that proves the validity of execution of the whitelisted photo edits - stores verification results of ZKPs - Arithmetic Circuit - checks the validity of execution of some computation on an image (the following are some of the non-distorting image computation) - unedited (no edits) - crop - rotate - resize - white balance - contrast - grayscale / saturation - blur / sharpen #### 3. Proof of Photo Metadata ![grant-proposal-proof-of-metadata](https://hackmd.io/_uploads/B12aXQQrT.png) We are planning to implement the following to prove the validity of metadata of the original photo. - Client App - sends metadata with signature of the original photo to smart contract - Smart Contract - verifies signature of the original photo - generates a ZKP that proves the validity of metadata of the original photo - stores verification results of ZKPs - Arithmetic Circuit - checks the validity of metadata of an image *Note that there are some previous works regarding zero knowledge photo/image such as [zk-img](https://medium.com/@danieldkang/zk-img-fighting-deepfakes-with-zero-knowledge-proofs-9b76c23e3789), yet there exists neither open-source implementation of it nor its application. This project would be the first open source project that will help the advancement of zero knowledge technology in image/video processing. ## Team :busts_in_silhouette: ### Team members * Names of team members: **Seiya Kobayashi** * Email: **work@seiya.tech** * Telegram handle: **@seiya_kobayashi** ### Team's experience - A former grantee from the PSE team at Ethereum Foundation ([blogpost](https://hackmd.io/@seiyakobayashi/rku9i9fN3)) - A seasoned full-stack web2 engineer - Solid experience of web3 (DApps) development - Solid understanding of blockchain-related technologies including zero knowledge systems - Basic understanding of AI-related technologies - B.A. degree in Computer Science from Boston University (2020) ### Team Code Repos * https://github.com/SeiyaKobayashi/zero-knowledge-photo ## Development Roadmap :nut_and_bolt: ### Overview * **Total Estimated Duration:** 3 months & 2 weeks (2023/12/01 - 2024/03/18, longer than 3 months because this duration contains year-end holidays) * **Full-time equivalent (FTE):** 1 * **Total Costs:** $36,000 (calculated as $67.5 / hour) ### Milestone #1: Design & Documentation * **Estimated Duration:** 2 weeks (14 days) * **FTE:** 1 * **Costs:** $5,400 * **Estimated duration**: 2023/12/01 - 2023/12/14 | Number | Deliverable | Specification | | ------------- | ------------- | ------------- | | 1. | Repo Setup | We will set up our GitHub repo determining directory structure, and provide minimum documentation in `README`. | | 2. | Design & Documentation | We will design the detailed structure of each component of our demo application, providing architecture diagrams as well as the list of necessary functionalities. Catch-up with the concept of image processing is also necessary. | ### Milestone #2: Circuit Implementation * **Estimated Duration:** 3 weeks (21 days) * **FTE:** 1 * **Costs:** $8,100 * **Estimated duration**: 2023/12/15 - 2024/01/15 (on vacation 2023/12/24 - 2024/01/03) | Number | Deliverable | Specification | | ------------- | ------------- | ------------- | | 1. | circuit #1: `unedited` | We will implement an arithmetic circuit that proves `unedited` operation on an image. | | 2. | circuit #2: `resize` | We will implement an arithmetic circuit that proves `resize` operation on an image. | | 3. | circuit #3: `crop` | We will implement an arithmetic circuit that proves `crop` operation on an image. | | 4. | circuit #4: `rotate` | We will implement an arithmetic circuit that proves `rotate` operation on an image. | | 5. | circuit #5: `white balance` | We will implement an arithmetic circuit that proves `white balance` operation on an image. | | 6. | (*optional; if time permits*) circuit #6: `contrast` | We will implement an arithmetic circuit that proves `contrast` operation on an image. | ### Milestone #3: Smart Contract Implementation * **Estimated Duration:** 3 weeks (21 days) * **FTE:** 1 * **Costs:** $8,100 * **Estimated duration**: 2024/01/16 - 2024/02/06 | Number | Deliverable | Specification | | ------------- | ------------- | ------------- | | 1. | functionality #1: `user operations` | We will write a piece of code that stores the end-user's info (e.g. wallet address). We will also provide unit tests of functions with appropriate coverage. | | 2. | functionality #2: `signature operations` | We will write a piece of code that stores and verifies the signatures of the image and its metadata. We will also provide unit tests of functions with appropriate coverage. | | 3. | functionality #3: `zkp operations` | We will write a piece of code that generates/verifies ZKPs, and stores the results of ZKP verification. We will also provide unit tests of functions with appropriate coverage. | ### Milestone #4: Client App Implementation * **Estimated Duration:** 1 month (30 days) * **FTE:** 1 * **Costs:** $10,800 * **Estimated duration**: 2024/02/07 - 2024/03/08 | Number | Deliverable | Specification | | ------------- | ------------- | ------------- | | 1. | functionality #1: `wallet connection` | We will write a piece of code that connects the end-user's wallet to our smart contract. | | 2. | functionality #2: `photo taking` | We will write a piece of code that enables end-users to take photos directly on a web browser, and sends signatures of the taken image as well as each metadata to our smart contract. | | 3. | functionality #3: `edited photo upload` | We will write a piece of code that enables end-users to upload edited photos along with the signature of the original photo to our smart contract for proof generation & verification. | | 4. | functionality #4: `edited photo sharing` | We will write a piece of code that enables end-users to upload edited and verified photos to IPFS for sharing. | | 5. | (*optional; if time permits*) functionality #5 : `metadata upload` | We will write a piece of code that enables end-users to upload photo metadata along with the signature of the original photo to our smart contract for proof generation & verification. | | 6. | Tests & Documentation | We will make sure that our application works as expected by writing tests. We will also update documentation (e.g. `README`s) as needed. | ### Milestone #5: Written Publication * **Estimated Duration:** 10 days * **FTE:** 1 * **Costs:** $3,600 * **Estimated duration**: 2024/03/08 - 2024/03/18 | Number | Deliverable | Specification | | ------------- | ------------- | ------------- | | 1. | Publication | We will write a blogpost to document the development process, highlighting the significance of the project and technical difficulties faced during the research and development. | ## Additional Information :heavy_plus_sign: I am also eager to give presentations about this project after its completion at some of the Ethereum-related global conferences (I did it for my last grant project at [EthTaipei](https://ethtaipei.org/)).