# Copy to R2

## Goals
- Copy historical content from S3 into R2 using Carpark format `carCid/carCid.car`
- Enable freeway and dag API to serve content from R2
- Enable E-IPFS in CF

## Buckets and namespaces

- `dotstorage-staging-0`
- `us-east-2`
- namespaces:
- `raw` => DONE
- `complete`
- [info-staging-0]:
- size: 18.5G
- number: 6807
- `dotstorage-prod-0`
- `us-east-2`
- namespaces:
- `raw` => DONE
- `complete`
- [info-prod-0]:
- size: 358T
- number: 118M
- `dotstorage-prod-1`
- `us-west-2`
- namespaces:
- `raw` => DONE
- `complete`
- [info-prod-1]:
- size: 936T
- number: 445M

- others buckets to do
- `w3up-cars`
- other account bucket (web3.storage)

Note:
- Info as of 10th May

## Bucket history

- `Aug 1, 2022`
- started writes into `dotstorage-prod-1` bucket
- [`Sep 19, 2022`](https://github.com/web3-storage/web3.storage/pull/1785) `carpark-prod-0` R2 bucket writes added
- [`Sep 21, 2022`](https://github.com/web3-storage/web3.storage/pull/1884) released API
- [`Oct 17, 2022`](https://github.com/web3-storage/web3.storage/pull/2035) DUDEWHERE + SATNAV
- [`Oct 17, 2022`](https://github.com/web3-storage/web3.storage/pull/2025) reelased API

## Processes

### Get Files with content to copy over

- RAW [`dotstorage-prod-0-raw.json`](https://bafybeidxsyhzorwa2e2lbt3u6jp4akb2dy574y2n4dlipuqsafl7u7ozme.ipfs.w3s.link/dotstorage-prod-0-raw.json)
- RAW [`dotstorage-prod-1-raw.json`](https://bafybeibyuannrlmveiccjaesrpu5zlwuf22u2u3xdcbgr5p4unpr6qpqqm.ipfs.w3s.link/dotstorage-prod-1-raw.json)

- larger complete CAR files

## Other notes

1. `carpark-prod-0` in R2 got initial writes without `DUDEWHERE` and `SATNAV`

## Complete

1. Get files with list of Complete CARs with "inKey (`complete/${contentCid}.car`), size"
1.1. 10.000 entries create new file
2. Iterate list, read from S3 and write to disk, while calculating car CID and md5 on the go
2.1. write record to queue with rootCid, carCid and md5
2.2. write carCid and md5 to object metadata (`complete/${contentCid}.car`)
3. Read record from queue, read rootCid item from S3 and pipe to R2

Consider filter size smaller than 100mb and use lambda as consumer from queue to copy small stuff.

Timelines:
- `dotstorage-prod-0` is already available
- `dotstorage-prod-1` MUST wait until cluster migration complete

## Others

### TSV to JSON

jq -rRs 'split("\n")[1:-1] |
         map([split("\t")[]|split(",")] | {
                 "clusterOnly":.[1],
                 "contentCid":.[4]
             }
    )' w3_cids_uploads_blocked_users.txt > w3_cids_uploads_blocked_users.json

[info-staging-0]: https://ap-southeast-1.console.aws.amazon.com/cloudwatch/home?region=ap-southeast-1#metricsV2:graph=~(timezone~'UTC~region~'ap-southeast-1~view~'timeSeries~stacked~false~start~'-PT2160H~end~'P0D~metrics~(~(~'AWS*2fS3~'NumberOfObjects~'StorageType~'AllStorageTypes~'BucketName~'dotstorage-staging-0~(period~86400~region~'us-east-2))~(~'.~'BucketSizeBytes~'.~'StandardStorage~'.~'.~(period~86400~region~'us-east-2))~(~'...~'IntelligentTieringFAStorage~'.~'.~(period~86400~region~'us-east-2))));query=~'*7bAWS*2fS3*2cBucketName*2cStorageType*7d*20dotstorage;region=us-east-2;forwardXA=no
[info-prod-0]: https://ap-southeast-1.console.aws.amazon.com/cloudwatch/home?region=ap-southeast-1#metricsV2:graph=~(timezone~'UTC~region~'ap-southeast-1~view~'timeSeries~stacked~false~metrics~(~(~'AWS*2fS3~'NumberOfObjects~'StorageType~'AllStorageTypes~'BucketName~'dotstorage-prod-0~(period~86400~region~'us-east-2))~(~'.~'BucketSizeBytes~'.~'StandardStorage~'.~'.~(period~86400~region~'us-east-2)))~start~'-PT2160H~end~'P0D);query=~'*7bAWS*2fS3*2cBucketName*2cStorageType*7d*20dotstorage;region=us-east-2;forwardXA=no
[info-prod-1]: https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#metricsV2:graph=~(timezone~'UTC~region~'us-west-2~view~'timeSeries~stacked~false~start~'-PT2160H~end~'P0D~stat~'Average~period~86400~metrics~(~(~'AWS*2fS3~'NumberOfObjects~'StorageType~'AllStorageTypes~'BucketName~'dotstorage-prod-1~(period~86400))~(~'.~'BucketSizeBytes~'.~'StandardStorage~'.~'.~(period~86400))));query=~'*7bAWS*2fS3*2cBucketName*2cStorageType*7d*20dotstorage;region=us-west-2;forwardXA=no