# Uploading to S3

1. Create buckets with proper names. For example:
* dr9.legacysurvey.org
* dr9-north.legacysurvey.org
* dr9-south.legacysurvey.org

us-west-2 region is recommended here because it's cheap, but feel free to pick any.

2. Set bucket permission to public readable:

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"PublicRead",
      "Effect":"Allow",
      "Principal": "*",
      "Action":["s3:GetObject","s3:GetObjectVersion"],
      "Resource":["arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"]
    }
  ]
}

3. Use `boto` or `boto3` to upload to the buckets.
Optionally, use `s3-parallel-put` because `boto` by default does not have concurrency.

FOLDER=0
~/s3-parallel-put --bucket=dr9-north.legacysurvey.org --bucket_region=us-west-2 --host=s3-us-west-2.amazonaws.com --put=stupid --secure --prefix $FOLDER --processes=20 --log-filename=/global/cscratch1/sd/ziyaoz/$FOLDER $FOLDER

4. In the case that you need to upload lots of files very quickly, you can launch a few EC2 instances, transfer tarred files from NERSC to them, and upload from there.