Python / boto3
===
###### tags: `Python`
###### tags: `Python`, `boto3`, `AWS`, `S3`, `session`

<br>

[TOC]

<br>

## Boto3 Docs 1.26.44 documentation
> https://boto3.amazonaws.com/v1/documentation/api/latest/index.html

### [Session](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/session.html)
- Default session
- Custom session

<br>

## Custom session

import boto3
session = boto3.Session(
    aws_access_key_id=settings.AWS_SERVER_PUBLIC_KEY,
    aws_secret_access_key=settings.AWS_SERVER_SECRET_KEY,
)

- get an S3 resource

    s3 = session.resource('s3')

- 參考資料
- [How to specify credentials when connecting to boto3 S3?](https://stackoverflow.com/questions/45981950)
- get a low-level S3 client

    s3_client = session.client('s3')

- 參考資料
- [How to save S3 object to a file using boto3](https://stackoverflow.com/questions/29378763)

<br>

## 底下測試有成功
### 使用 custom session

AWS_ACCESS_KEY_ID = "X7T******PSZ"
AWS_SECRET_ACCESS_KEY = "vLs******yuJ"

import boto3
session = boto3.Session(
    aws_access_key_id=AWS_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
)
s3 = session.client('s3', endpoint_url="https://cloudstorage.oneai.twcc.ai")
s3.list_buckets()

執行結果

{'ResponseMetadata': {
    HTTPStatusCode': 200,
    'HTTPHeaders': {
      'content-type': 'text/xml',
      'request-id': 'db7606f9-fc08-4439-a0de-de5b5feeed98',
      'date': 'Mon, 09 Jan 2023 02:59:38 GMT',
      'transfer-encoding': 'chunked'
    },
    'RetryAttempts': 0
  },
  'Buckets': [
    ...
    {
      'Name': 'mlflow',
      'CreationDate': datetime.datetime(2023, 1, 9, 2, 59, 38, 542000, tzinfo=tzutc())
    },
    ...
  ],
  'Owner': {
    'DisplayName': 'abf29785-7dd8-4c15-a06a-733f124772b3',
    'ID': 'abf29785-7dd8-4c15-a06a-733f124772b3'
  }
}

### mlflow + s3
> 參考範例：
> [](https://i.imgur.com/fIwrcdp.png)

import mlflow
import os
os.environ["AWS_ACCESS_KEY_ID"] = AWS_ACCESS_KEY_ID
os.environ["AWS_SECRET_ACCESS_KEY"] = AWS_SECRET_ACCESS_KEY
os.environ["MLFLOW_S3_ENDPOINT_URL"] = "https://cloudstorage.oneai.twcc.ai"
os.environ["MLFLOW_TRACKING_URI"] = "https://mlflow-token:eyJ******2b3/"

from mlflow.tracking import MlflowClient
client = MlflowClient()
client.get_model_version_download_uri('penguin', '1')

執行結果：

's3://mlflow/154/30ea3245ad944c4f91f9b7bdcb731331/artifacts/model'

default_kwargs = {
    "Bucket": 'mlflow',
    "Prefix": '154/30ea3245ad944c4f91f9b7bdcb731331/artifacts/model'
}

s3.list_objects_v2(**default_kwargs)

執行結果：

{'ResponseMetadata': {'HTTPStatusCode': 200,
  'HTTPHeaders': {'content-type': 'text/xml',
   'request-id': '8bc1ab5e-49c7-45ef-af8f-48c0abc95763',
   'date': 'Mon, 09 Jan 2023 03:23:14 GMT',
   'transfer-encoding': 'chunked'},
  'RetryAttempts': 0},
 'IsTruncated': False,
 'Contents': [{'Key': '154/30ea3245ad944c4f91f9b7bdcb731331/artifacts/model/',
   'LastModified': datetime.datetime(2022, 12, 22, 6, 0, 17, 412000, tzinfo=tzutc()),
   'ETag': 'd41d8cd98f00b204e9800998ecf8427e',
   'Size': 0,
   'StorageClass': 'STANDARD'},
  {'Key': '154/30ea3245ad944c4f91f9b7bdcb731331/artifacts/model/MLmodel',
   'LastModified': datetime.datetime(2022, 12, 22, 6, 0, 12, 428000, tzinfo=tzutc()),
   'ETag': '189f5e240830ed83023275d1cad578ec',
   'Size': 394,
   'StorageClass': 'STANDARD'},
  ...],
 'Name': 'mlflow',
 'Prefix': '154/30ea3245ad944c4f91f9b7bdcb731331/artifacts/model',
 'Delimiter': '',
 'MaxKeys': 1000,
 'EncodingType': 'url'}

<br>

## 參考資料
- [How to use Boto3 to download all files from an S3 Bucket?](https://www.learnaws.org/2022/07/02/boto3-download-files-s3/)