# Ozone Docker Compose 啟動與 OpenDAL S3 介面使用指南
# Ozone Docker Compose Setup and OpenDAL S3 Interface Guide
本文件說明如何使用 Docker Compose 啟動 Apache Ozone，並透過 OpenDAL 的 S3 介面連線本地環境。
This document describes how to start Apache Ozone using Docker Compose and connect it to your local environment through OpenDAL's S3 interface.
---
## 目錄 / Table of Contents
- [前置需求 / Prerequisites](#前置需求--prerequisites)
- [系統需求 / System Requirements](#系統需求--system-requirements)
- [安裝 OpenDAL / Install OpenDAL](#安裝-opendal--install-opendal)
- [啟動 Ozone 叢集 / Starting Ozone Cluster](#啟動-ozone-叢集--starting-ozone-cluster)
- [驗證 Ozone 服務 / Verifying Ozone Services](#驗證-ozone-服務--verifying-ozone-services)
- [設定 S3 存取憑證 / Configuring S3 Access Credentials](#設定-s3-存取憑證--configuring-s3-access-credentials)
- [使用 OpenDAL S3 介面 / Using OpenDAL S3 Interface](#使用-opendal-s3-介面--using-opendal-s3-interface)
- [快速測試連線 / Quick Connection Test](#快速測試連線--quick-connection-test)
- [OpenDAL 傳輸機制與檔案移動條件 / OpenDAL Transfer Mechanism and File Movement Conditions](#opendal-傳輸機制與檔案移動條件--opendal-transfer-mechanism-and-file-movement-conditions)
- [上傳本地檔案到 Ozone / Upload Local Files to Ozone](#上傳本地檔案到-ozone--upload-local-files-to-ozone)
- [常見操作範例 / Common Operations Examples](#常見操作範例--common-operations-examples)
- [故障排除 / Troubleshooting](#故障排除--troubleshooting)
- [完整測試腳本程式碼 / Complete Test Scripts Code](#完整測試腳本程式碼--complete-test-scripts-code)
---
## 前置需求 / Prerequisites
### 系統需求 / System Requirements
- **Docker**: 最新穩定版本 / Latest stable version
- **Docker Compose**: 至少 v2.30 / At least v2.30
- **Python 3** (可選，用於 Python 範例) / Python 3 (Optional, for Python examples)
- **OpenDAL**: 需要安裝的 OpenDAL 庫 / OpenDAL library (needs to be installed)
### 檢查 Docker 環境 / Check Docker Environment
```bash
# 檢查 Docker 版本 / Check Docker version
docker --version
# 檢查 Docker Compose 版本 / Check Docker Compose version
docker compose version
```
### 安裝 OpenDAL / Install OpenDAL
根據您使用的語言，選擇對應的安裝方式：
Depending on your programming language, choose the corresponding installation method:
#### Python / Python
```bash
# 使用 pip 安裝 / Install using pip
pip3 install opendal
# 或使用 pip / Or use pip
pip install opendal
```
#### Rust / Rust
```bash
# 在 Cargo.toml 中添加依賴 / Add dependency in Cargo.toml
cargo add opendal
```
#### Node.js / Node.js
```bash
# 使用 npm 安裝 / Install using npm
npm install opendal
# 或使用 yarn / Or use yarn
yarn add opendal
```
#### Java / Java
在您的 `pom.xml` 或 `build.gradle` 中添加 OpenDAL 依賴。
Add OpenDAL dependency in your `pom.xml` or `build.gradle`.
**驗證安裝 / Verify Installation:**
```bash
# Python / Python
python3 -c "import opendal; print('OpenDAL version:', opendal.__version__)"
# Rust / Rust
cargo --version # 確保 Cargo 已安裝 / Ensure Cargo is installed
# Node.js / Node.js
node -e "console.log(require('opendal').version)"
```
---
## 啟動 Ozone 叢集 / Starting Ozone Cluster
### 方法一：使用官方預建置的 Docker Compose 設定（推薦）/ Method 1: Using Official Pre-built Docker Compose Configuration (Recommended)
**推薦使用此方法**，因為它使用預建置的 Docker 映像檔，無需先建置 Ozone 專案。
**This method is recommended** as it uses pre-built Docker images and doesn't require building the Ozone project first.
#### 步驟 1: 下載官方 Docker Compose 設定 / Step 1: Download Official Docker Compose Configuration
```bash
# 在專案根目錄下載設定檔 / Download configuration file in project root
# 請替換為您的實際專案路徑 / Please replace with your actual project path
cd /path/to/ozone
curl -O https://raw.githubusercontent.com/apache/ozone-docker/refs/heads/latest/docker-compose.yaml
```
#### 步驟 2: 啟動 Ozone 集群 / Step 2: Start Ozone Cluster
**單一 DataNode 啟動 / Single DataNode:**
```bash
docker compose up -d
```
**多個 DataNode 啟動（建議用於測試複製功能）/ Multiple DataNodes (Recommended for testing replication):**
```bash
# 啟動 3 個 DataNode / Start 3 DataNodes
docker compose up -d --scale datanode=3
```
此命令會自動從 Docker Hub 拉取所需的 `apache/ozone:2.0.0` 映像檔並啟動所有服務。
This command will automatically pull the required `apache/ozone:2.0.0` images from Docker Hub and start all services.
### 方法二：使用專案內建的 Docker Compose 設定 / Method 2: Using Built-in Docker Compose Configuration
**注意 / Note:** 此方法需要先建置 Ozone 專案，或者修復 `.env` 檔案中的 Maven 變數。
**Note:** This method requires building the Ozone project first, or fixing Maven variables in the `.env` file.
Ozone 專案已經包含了 Docker Compose 設定檔，位於 `hadoop-ozone/dist/src/main/compose/ozone/` 目錄。
The Ozone project already includes Docker Compose configuration files located in the `hadoop-ozone/dist/src/main/compose/ozone/` directory.
#### 步驟 1: 進入 Docker Compose 目錄 / Step 1: Navigate to Docker Compose Directory
```bash
cd hadoop-ozone/dist/src/main/compose/ozone/
```
#### 步驟 2: 修復 .env 檔案（如果需要）/ Step 2: Fix .env File (If Needed)
如果遇到 `.env` 檔案中的 Maven 變數錯誤，需要手動修復：
If you encounter Maven variable errors in the `.env` file, you need to fix it manually:
```bash
# 備份原始檔案 / Backup original file
cp .env .env.backup
# 建立修復後的 .env 檔案 / Create fixed .env file
cat > .env << 'EOF'
HDDS_VERSION=2.2.0-SNAPSHOT
HADOOP_IMAGE=apache/hadoop:3
OZONE_RUNNER_VERSION=latest
OZONE_RUNNER_IMAGE=apache/ozone-runner
OZONE_OPTS=
EOF
```
#### 步驟 3: 啟動 Ozone 集群 / Step 3: Start Ozone Cluster
**使用專案提供的啟動腳本 / Using Project Startup Script:**
```bash
# 單一 DataNode / Single DataNode
./run.sh -d
# 多個 DataNode / Multiple DataNodes
OZONE_DATANODES=3 ./run.sh -d
```
**或直接使用 docker compose / Or use docker compose directly:**
```bash
# 設定必要的環境變數 / Set required environment variables
export OZONE_RUNNER_IMAGE=apache/ozone-runner
export OZONE_RUNNER_VERSION=latest
# 啟動叢集 / Start cluster
docker compose up -d --scale datanode=1
```
---
## 驗證 Ozone 服務 / Verifying Ozone Services
### 檢查服務狀態 / Check Service Status
```bash
# 查看所有服務狀態 / View all service status
docker compose ps
# 查看特定服務日誌 / View specific service logs
docker compose logs s3g
docker compose logs om
docker compose logs scm
docker compose logs datanode
# 即時追蹤日誌 / Follow logs in real-time
docker compose logs -f s3g
```
### 存取 Web UI / Access Web UIs
啟動成功後，您可以透過以下網址存取 Ozone 的管理介面：
After successful startup, you can access Ozone's management interfaces at the following addresses:
- **Ozone Manager UI**: http://localhost:9874
- **Storage Container Manager (SCM) UI**: http://localhost:9876
- **Recon UI**: http://localhost:9888
- **S3 Gateway**: http://localhost:9878
### 測試 S3 Gateway 連線 / Test S3 Gateway Connection
```bash
# 測試 S3 Gateway 是否正常執行 / Test if S3 Gateway is running
curl http://localhost:9878
```
**預期結果 / Expected Result:**
- 如果返回 403 錯誤或 XML 錯誤訊息，這是正常的，表示 S3 Gateway 正在執行，只是需要認證。
- If you get a 403 error or XML error message, this is normal - it means the S3 Gateway is running but requires authentication.
**驗證所有服務的 HTTP 狀態碼 / Verify HTTP Status Codes for All Services:**
```bash
# 檢查所有服務的 HTTP 響應 / Check HTTP responses for all services
curl -s -o /dev/null -w "Ozone Manager: %{http_code}\n" http://localhost:9874
curl -s -o /dev/null -w "SCM: %{http_code}\n" http://localhost:9876
curl -s -o /dev/null -w "Recon: %{http_code}\n" http://localhost:9888
curl -s -o /dev/null -w "S3 Gateway: %{http_code}\n" http://localhost:9878
```
**預期的狀態碼 / Expected Status Codes:**
- Ozone Manager: `200` (正常 / OK)
- SCM: `200` (正常 / OK)
- Recon: `200` (正常 / OK)
- S3 Gateway: `403` 或 `400` (正常，需要認證 / OK, requires authentication)
---
## 設定 S3 存取憑證 / Configuring S3 Access Credentials
### 重要說明 / Important Note
**在非安全模式下，您不需要申請 token 或生成憑證！**
**In non-secure mode, you do NOT need to apply for tokens or generate credentials!**
預設的 Ozone Docker Compose 設定是**非安全模式**（security disabled），這意味著：
The default Ozone Docker Compose configuration is in **non-secure mode** (security disabled), which means:
- ✅ 可以使用**任意的**存取金鑰和秘密金鑰
- ✅ You can use **any arbitrary** access key and secret key
- ❌ 不需要 Kerberos 認證
- ❌ No Kerberos authentication required
- ❌ 不需要執行 `ozone s3 getsecret` 命令
- ❌ No need to run `ozone s3 getsecret` command
**只有在啟用安全模式（Kerberos）時，才需要透過 `ozone s3 getsecret` 命令取得憑證。**
**Only when security mode (Kerberos) is enabled, you need to obtain credentials via the `ozone s3 getsecret` command.**
### 設定環境變數 / Set Environment Variables
在非安全模式下，直接設定任意值即可：
In non-secure mode, you can directly set any values:
```bash
# 設定 AWS 存取憑證（使用文件內推薦的測試憑證）/ Set AWS access credentials (using recommended test credentials)
export AWS_ACCESS_KEY_ID=testuser/scm@EXAMPLE.COM
export AWS_SECRET_ACCESS_KEY=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
# 或使用其他任意值 / Or use any other arbitrary values
export AWS_ACCESS_KEY_ID=ozone
export AWS_SECRET_ACCESS_KEY=ozone123
# 或使用更簡單的值 / Or use simpler values
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test123
```
### 安全模式下的憑證取得（可選）/ Obtaining Credentials in Secure Mode (Optional)
如果您啟用了安全模式，則需要透過以下方式取得憑證：
If you have enabled secure mode, you need to obtain credentials through the following methods:
#### 方法 1: 使用命令行 / Method 1: Using Command Line
```bash
# 在 Ozone Manager 容器中執行 / Execute in Ozone Manager container
docker compose exec om ozone s3 getsecret
# 輸出示例 / Example output:
# awsAccessKey=testuser/scm@EXAMPLE.COM
# awsSecret=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
```
#### 方法 2: 使用 REST API / Method 2: Using REST API
```bash
# 需要先進行 Kerberos 認證 / Requires Kerberos authentication first
curl -X PUT --negotiate -u : https://localhost:9879/secret
```
### 快速測試連線 / Quick Connection Test
我們提供了一個測試腳本來驗證 S3 Gateway 連線：
We provide a test script to verify S3 Gateway connection:
```bash
# 執行測試腳本 / Run test script
./test-s3-connection.sh
```
### 使用 AWS CLI 測試連線 / Test Connection with AWS CLI
如果您已安裝 AWS CLI，可以使用以下命令測試：
If you have AWS CLI installed, you can use the following commands to test:
```bash
# 設定環境變數 / Set environment variables
export AWS_ACCESS_KEY_ID=testuser/scm@EXAMPLE.COM
export AWS_SECRET_ACCESS_KEY=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
# 建立一個測試桶 / Create a test bucket
aws s3api --endpoint http://localhost:9878 create-bucket --bucket test-bucket
# 列出所有桶 / List all buckets
aws s3api --endpoint http://localhost:9878 list-buckets
# 上傳檔案 / Upload a file
echo "Hello Ozone" > /tmp/test.txt
aws s3 --endpoint http://localhost:9878 cp /tmp/test.txt s3://test-bucket/test.txt
# 列出桶中的檔案 / List files in bucket
aws s3 --endpoint http://localhost:9878 ls s3://test-bucket/
```
**注意 / Note:** 如果沒有安裝 AWS CLI，可以直接跳到下一步使用 OpenDAL 進行連線。
**Note:** If AWS CLI is not installed, you can skip to the next step and use OpenDAL to connect.
---
## 使用 OpenDAL S3 介面 / Using OpenDAL S3 Interface
OpenDAL 是一個統一的資料存取層，支援多種後端儲存系統，包括 S3 相容的服務。
OpenDAL is a unified data access layer that supports multiple backend storage systems, including S3-compatible services.
### 快速測試連線 / Quick Connection Test
在開始使用 OpenDAL 之前，我們提供了一個測試腳本來驗證連線：
Before using OpenDAL, we provide a test script to verify the connection:
```bash
# 設定環境變數 / Set environment variables
export AWS_ACCESS_KEY_ID=testuser/scm@EXAMPLE.COM
export AWS_SECRET_ACCESS_KEY=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
# 執行測試腳本 / Run test script
python3 test-opendal-ozone.py
```
**測試腳本功能 / Test Script Features:**
- ✅ 建立 OpenDAL S3 操作符 / Create OpenDAL S3 operator
- ✅ 檢查桶是否存在 / Check if bucket exists
- ✅ 測試寫入操作 / Test write operation
- ✅ 測試讀取操作 / Test read operation
- ✅ 測試列出檔案 / Test list operation
- ✅ 測試刪除操作 / Test delete operation
**重要提示 / Important Note:**
在執行測試腳本之前，請確保桶已建立。如果桶不存在，測試腳本會提供建立桶的說明。
Before running the test script, make sure the bucket is created. If the bucket doesn't exist, the test script will provide instructions on how to create it.
**建立桶的方法 / Methods to Create Bucket:**
```bash
# 方法 1: 使用 Ozone 命令 / Method 1: Using Ozone command
docker compose exec om bash -c "ozone sh bucket create /s3v/test-bucket"
# 方法 2: 使用 Ozone Manager REST API / Method 2: Using Ozone Manager REST API
curl -X PUT http://localhost:9874/api/v1/bucket/s3v/test-bucket
# 方法 3: 使用 AWS CLI (如果已安裝) / Method 3: Using AWS CLI (if installed)
aws s3api --endpoint http://localhost:9878 create-bucket --bucket test-bucket
```
如果測試成功，您將看到所有操作都通過的訊息。
If the test succeeds, you will see a message that all operations passed.
---
## OpenDAL 傳輸機制與檔案移動條件 / OpenDAL Transfer Mechanism and File Movement Conditions
### OpenDAL 與 Ozone 的綁定方式 / How OpenDAL Binds to Ozone
**重要概念 / Important Concept:**
OpenDAL 與 Ozone 的綁定是在**程式執行時動態設定**的，不是預先綁定的。OpenDAL 是一個程式庫（Library），需要在應用程式中匯入並在程式碼中設定連線資訊。
The binding between OpenDAL and Ozone is **dynamically configured at runtime in the program**, not pre-bound. OpenDAL is a library that needs to be imported in the application and configured in code.
#### 綁定過程 / Binding Process
```python
import opendal # 1. 匯入 OpenDAL 庫 / Import OpenDAL library
# 2. 在程式執行時建立操作符並設定連線 / Create operator and configure connection at runtime
op = opendal.Operator(
"s3", # 指定後端類型為 S3 / Specify backend type as S3
endpoint="http://localhost:9878", # Ozone S3 Gateway 端點 / Ozone S3 Gateway endpoint
bucket="test-bucket", # 桶名 / Bucket name
access_key_id="...", # 認證資訊 / Authentication info
secret_access_key="...", # 認證資訊 / Authentication info
region="us-east-1", # 區域 / Region
)
# 3. 此時才建立與 Ozone 的連線 / Connection to Ozone is established at this point
```
#### 綁定特點 / Binding Characteristics
- **動態綁定 / Dynamic Binding**: 每次執行程式時都會重新建立連線
- **設定驅動 / Configuration-Driven**: 所有連線資訊都在程式碼中設定
- **無需預先註冊 / No Pre-registration**: 不需要在 Ozone 或系統層面預先綁定
- **彈性設定 / Flexible Configuration**: 可以透過程式碼、環境變數、設定檔等方式設定
#### 多個連線範例 / Multiple Connections Example
```python
import opendal
# 可以建立多個操作符連線到不同的儲存 / Can create multiple operators for different storage
ozone_op = opendal.Operator("s3", endpoint="http://localhost:9878", bucket="bucket1", ...)
aws_op = opendal.Operator("s3", endpoint="https://s3.amazonaws.com", bucket="bucket2", ...)
# 每個操作符都是獨立的連線 / Each operator is an independent connection
ozone_op.write("file1.txt", b"data1")
aws_op.write("file2.txt", b"data2")
```
### OpenDAL 傳輸機制概述 / OpenDAL Transfer Mechanism Overview
OpenDAL 是一個統一的資料存取層，它透過標準化的 API 來處理不同後端儲存系統的資料傳輸。當使用 OpenDAL 與 Ozone S3 Gateway 進行檔案傳輸時，以下是其工作原理：
OpenDAL is a unified data access layer that handles data transfer for different backend storage systems through standardized APIs. When using OpenDAL with Ozone S3 Gateway for file transfer, here's how it works:
#### 1. 傳輸流程 / Transfer Process
```
本地檔案系統 / Local File System
↓ (讀取 / Read)
應用程式記憶體 / Application Memory
↓ (OpenDAL 操作 / OpenDAL Operation)
OpenDAL S3 操作符 / OpenDAL S3 Operator
↓ (HTTP/HTTPS 請求 / HTTP/HTTPS Request)
Ozone S3 Gateway (Port 9878)
↓ (S3 協定 / S3 Protocol)
Ozone 儲存層 / Ozone Storage Layer
```
#### 2. 檔案上傳流程 / File Upload Process
當您使用 `op.write()` 上傳檔案時，OpenDAL 執行以下步驟：
When you use `op.write()` to upload a file, OpenDAL performs the following steps:
1. **讀取本地檔案到記憶體 / Read Local File to Memory**
- 應用程式先將本地檔案讀取到記憶體中
- The application first reads the local file into memory
- 檔案內容以 `bytes` 或 `memoryview` 形式存在
- File content exists as `bytes` or `memoryview`
2. **建立 S3 API 請求 / Build S3 API Request**
- OpenDAL 根據設定建立 S3 PUT 請求
- OpenDAL builds S3 PUT request based on configuration
- 包含必要的認證資訊（AWS Signature Version 4）
- Includes necessary authentication information (AWS Signature Version 4)
- 設定正確的 HTTP 標頭（Content-Type, Content-Length 等）
- Sets correct HTTP headers (Content-Type, Content-Length, etc.)
3. **發送 HTTP 請求 / Send HTTP Request**
- 透過 HTTP/HTTPS 發送到 Ozone S3 Gateway
- Sent to Ozone S3 Gateway via HTTP/HTTPS
- 端點：`http://localhost:9878`
- Endpoint: `http://localhost:9878`
4. **Ozone 處理 / Ozone Processing**
- S3 Gateway 接收請求並驗證認證
- S3 Gateway receives request and validates authentication
- 將資料儲存到 Ozone 的儲存層
- Stores data to Ozone's storage layer
#### 3. 檔案下載流程 / File Download Process
當您使用 `op.read()` 下載檔案時，OpenDAL 執行以下步驟：
When you use `op.read()` to download a file, OpenDAL performs the following steps:
1. **建立 S3 GET 請求 / Build S3 GET Request**
- OpenDAL 建立 S3 GET 請求
- OpenDAL builds S3 GET request
- 包含認證和目標檔案路徑
- Includes authentication and target file path
2. **發送請求並接收資料 / Send Request and Receive Data**
- 發送 HTTP GET 請求到 Ozone S3 Gateway
- Sends HTTP GET request to Ozone S3 Gateway
- 接收回應資料流
- Receives response data stream
3. **返回資料 / Return Data**
- 將接收到的資料返回給應用程式
- Returns received data to application
- 資料格式可能是 `bytes` 或 `memoryview`
- Data format may be `bytes` or `memoryview`
### 檔案移動的條件與限制 / File Movement Conditions and Limitations
#### 1. 移動檔案的本質 / Nature of File Movement
**重要概念 / Important Concept:**
OpenDAL 和 S3 協定本身**不支援原子性的檔案移動操作**。在 S3 相容的儲存系統中，"移動"檔案實際上是兩個操作的組合：
OpenDAL and the S3 protocol itself **do not support atomic file move operations**. In S3-compatible storage systems, "moving" a file is actually a combination of two operations:
1. **複製檔案 / Copy File**: 從來源路徑複製到目標路徑
2. **刪除來源檔案 / Delete Source File**: 刪除原始檔案
#### 2. 移動檔案的條件 / Conditions for Moving Files
要成功移動檔案，必須滿足以下條件：
To successfully move a file, the following conditions must be met:
##### 條件 1: 來源檔案必須存在 / Condition 1: Source File Must Exist
```python
# 檢查來源檔案是否存在 / Check if source file exists
try:
source_data = op.read("source-file.txt")
# 來源檔案存在，可以繼續移動 / Source file exists, can proceed with move
except Exception as e:
# 來源檔案不存在，無法移動 / Source file does not exist, cannot move
print(f"錯誤：來源檔案不存在 / Error: Source file does not exist")
```
##### 條件 2: 目標路徑不應已存在 / Condition 2: Target Path Should Not Exist
雖然 S3 允許覆蓋，但為了避免意外資料遺失，建議先檢查：
Although S3 allows overwriting, to avoid accidental data loss, it's recommended to check first:
```python
# 檢查目標檔案是否存在 / Check if target file exists
try:
target_data = op.read("target-file.txt")
# 目標檔案已存在，需要決定是否覆蓋 / Target file exists, need to decide whether to overwrite
print("警告：目標檔案已存在 / Warning: Target file already exists")
except:
# 目標檔案不存在，可以安全移動 / Target file does not exist, safe to move
pass
```
##### 條件 3: 足夠的儲存空間 / Condition 3: Sufficient Storage Space
移動操作需要臨時儲存空間（在複製階段）：
Move operation requires temporary storage space (during copy phase):
- 來源檔案和目標檔案會同時存在（在刪除來源檔案之前）
- Source and target files will exist simultaneously (before deleting source file)
- 確保有足夠的儲存空間容納兩個檔案
- Ensure sufficient storage space to accommodate both files
##### 條件 4: 正確的認證和權限 / Condition 4: Correct Authentication and Permissions
- 必須有讀取來源檔案的權限
- Must have permission to read source file
- 必須有寫入目標路徑的權限
- Must have permission to write to target path
- 必須有刪除來源檔案的權限
- Must have permission to delete source file
#### 3. 實作檔案移動的完整範例 / Complete Example of File Movement
```python
import opendal
def move_file_in_ozone(op, source_key, target_key, overwrite=False):
"""
在 Ozone 中移動文件 / Move file in Ozone
參數 / Parameters:
- op: OpenDAL 操作符 / OpenDAL operator
- source_key: 來源檔案路徑 / Source file path
- target_key: 目標檔案路徑 / Target file path
- overwrite: 是否覆蓋已存在的目標檔案 / Whether to overwrite existing target file
返回 / Returns:
- True: 移動成功 / Move successful
- False: 移動失敗 / Move failed
"""
# 條件檢查 1: 檢查來源檔案是否存在 / Condition 1: Check if source file exists
try:
source_data = op.read(source_key)
# 確保是 bytes 類型 / Ensure it's bytes type
if isinstance(source_data, memoryview):
source_data = bytes(source_data)
elif not isinstance(source_data, bytes):
source_data = bytes(source_data)
except Exception as e:
print(f"❌ 錯誤：無法讀取來源檔案 / Error: Cannot read source file: {source_key}")
print(f" 原因 / Reason: {e}")
return False
# 條件檢查 2: 檢查目標檔案是否存在 / Condition 2: Check if target file exists
if not overwrite:
try:
op.read(target_key)
print(f"⚠️ 警告：目標檔案已存在 / Warning: Target file already exists: {target_key}")
print(f" 使用 overwrite=True 來覆蓋 / Use overwrite=True to overwrite")
return False
except:
# 目標檔案不存在，可以繼續 / Target file does not exist, can proceed
pass
# 步驟 1: 複製檔案到目標位置 / Step 1: Copy file to target location
try:
op.write(target_key, source_data)
print(f"✅ 檔案已複製到目標位置 / File copied to target: {target_key}")
except Exception as e:
print(f"❌ 錯誤：無法寫入目標檔案 / Error: Cannot write target file: {target_key}")
print(f" 原因 / Reason: {e}")
return False
# 步驟 2: 刪除來源檔案 / Step 2: Delete source file
try:
op.delete(source_key)
print(f"✅ 來源檔案已刪除 / Source file deleted: {source_key}")
except Exception as e:
print(f"⚠️ 警告：無法刪除來源檔案 / Warning: Cannot delete source file: {source_key}")
print(f" 原因 / Reason: {e}")
print(f" 目標檔案已建立，但來源檔案仍存在 / Target file created, but source file still exists")
# 可以選擇回滾（刪除目標檔案）/ Can choose to rollback (delete target file)
return False
print(f"✅ 檔案移動成功！/ File move successful!")
print(f" 從 / From: {source_key}")
print(f" 到 / To: {target_key}")
return True
# 使用範例 / Usage Example
op = opendal.Operator(
"s3",
endpoint="http://localhost:9878",
bucket="test-bucket",
access_key_id="testuser/scm@EXAMPLE.COM",
secret_access_key="c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999",
region="us-east-1",
)
# 移動檔案 / Move file
move_file_in_ozone(op, "old-file.txt", "new-file.txt", overwrite=False)
```
#### 4. 傳輸過程中的技術細節 / Technical Details During Transfer
##### 記憶體使用 / Memory Usage
- **小檔案 (< 幾 MB)**: 整個檔案會載入到記憶體中
- **Small files (< few MB)**: Entire file is loaded into memory
- **大檔案**: OpenDAL 可能會使用串流傳輸或分塊上傳
- **Large files**: OpenDAL may use streaming or chunked upload
##### 錯誤處理 / Error Handling
在傳輸過程中可能遇到的錯誤：
Errors that may occur during transfer:
1. **網路錯誤 / Network Errors**
- 連線逾時 / Connection timeout
- 網路中斷 / Network interruption
- 解決方案：實作重試機制 / Solution: Implement retry mechanism
2. **認證錯誤 / Authentication Errors**
- 無效的 access key 或 secret key
- Invalid access key or secret key
- 解決方案：檢查環境變數和設定 / Solution: Check environment variables and configuration
3. **儲存錯誤 / Storage Errors**
- 儲存空間不足 / Insufficient storage space
- 權限不足 / Insufficient permissions
- 解決方案：檢查 Ozone 叢集狀態和權限 / Solution: Check Ozone cluster status and permissions
##### 效能考量 / Performance Considerations
1. **並行上傳 / Concurrent Uploads**
- OpenDAL 支援並行操作
- OpenDAL supports concurrent operations
- 可以同時上傳多個檔案以提高效率
- Can upload multiple files simultaneously for better efficiency
2. **批次操作 / Batch Operations**
- 對於多個檔案，考慮使用批次操作
- For multiple files, consider using batch operations
- 減少網路往返次數
- Reduce network round trips
3. **緩衝區大小 / Buffer Size**
- 可以根據檔案大小調整緩衝區
- Can adjust buffer size based on file size
- 大檔案可能需要更大的緩衝區
- Large files may require larger buffers
#### 5. 最佳實務 / Best Practices
1. **始終驗證傳輸結果 / Always Verify Transfer Results**
```python
# 上傳後驗證 / Verify after upload
uploaded_data = op.read("file.txt")
if uploaded_data == original_data:
print("✅ 檔案驗證成功 / File verification successful")
```
2. **實作錯誤重試機制 / Implement Error Retry Mechanism**
```python
import time
def upload_with_retry(op, key, data, max_retries=3):
for attempt in range(max_retries):
try:
op.write(key, data)
return True
except Exception as e:
if attempt < max_retries - 1:
time.sleep(2 ** attempt) # 指數退避 / Exponential backoff
continue
else:
raise e
```
3. **使用交易性操作（如果可能）/ Use Transactional Operations (If Possible)**
- 對於關鍵操作，考慮先複製，驗證成功後再刪除來源檔案
- For critical operations, consider copying first, then delete source file after verification
4. **監控傳輸進度 / Monitor Transfer Progress**
- 對於大檔案，可以實作進度回呼
- For large files, can implement progress callbacks
- 記錄傳輸日誌以便除錯
- Log transfer logs for debugging
#### 6. 測試檔案移動 / Test File Movement
我們提供了一個測試腳本來示範檔案移動操作：
We provide a test script to demonstrate file movement operations:
```bash
# 設定環境變數 / Set environment variables
export AWS_ACCESS_KEY_ID=testuser/scm@EXAMPLE.COM
export AWS_SECRET_ACCESS_KEY=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
# 執行檔案移動測試 / Run file movement test
python3 test-move-file.py
```
**測試腳本功能 / Test Script Features:**
- ✅ 建立測試檔案 / Create test file
- ✅ 檢查來源檔案是否存在 / Check if source file exists
- ✅ 檢查目標檔案是否存在 / Check if target file exists
- ✅ 複製檔案到目標位置 / Copy file to target location
- ✅ 驗證目標檔案內容 / Verify target file content
- ✅ 刪除來源檔案 / Delete source file
- ✅ 列出桶中的所有檔案 / List all files in bucket
**範例輸出 / Example Output:**
```
=== 移動檔案 / Moving File ===
來源檔案 / Source: file-to-move.txt
目標檔案 / Target: moved-file.txt
1. 檢查來源檔案是否存在...
✅ 來源檔案存在，大小: 65 bytes
2. 檢查目標檔案是否存在...
✅ 目標檔案不存在，可以繼續
3. 複製檔案到目標位置...
✅ 檔案已複製到目標位置
4. 驗證目標檔案...
✅ 目標檔案驗證成功，內容匹配
5. 刪除來源檔案...
✅ 來源檔案已刪除
=== ✅ 檔案移動成功！===
```
---
## 上傳本地檔案到 Ozone / Upload Local Files to Ozone
### 使用測試腳本上傳 / Upload Using Test Script
我們提供了一個專門的測試腳本 `test-upload-local-file.py` 來將本地檔案上傳到 Ozone：
We provide a dedicated test script `test-upload-local-file.py` to upload local files to Ozone:
```bash
# 設定環境變數 / Set environment variables
export AWS_ACCESS_KEY_ID=testuser/scm@EXAMPLE.COM
export AWS_SECRET_ACCESS_KEY=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
# 上傳本地檔案 / Upload local file
# 語法 / Syntax: python3 test-upload-local-file.py <本地檔案路徑> [S3鍵名]
# Syntax: python3 test-upload-local-file.py <local-file-path> [s3-key-name]
python3 test-upload-local-file.py /path/to/your/file.txt my-file-in-ozone.txt
# 如果不指定 S3 key，會使用原檔案名稱 / If S3 key is not specified, original filename will be used
python3 test-upload-local-file.py /path/to/your/file.txt
# 如果不提供任何參數，會建立一個測試檔案並上傳 / If no arguments provided, creates a test file and uploads it
python3 test-upload-local-file.py
```
**測試腳本功能 / Test Script Features:**
- ✅ 讀取本地檔案 / Read local file
- ✅ 上傳到 Ozone S3 Gateway / Upload to Ozone S3 Gateway
- ✅ 驗證上傳的檔案內容 / Verify uploaded file content
- ✅ 列出桶中的所有檔案 / List all files in bucket
- ✅ 顯示詳細的上傳資訊 / Display detailed upload information
**範例輸出 / Example Output:**
```
=== 測試將本地檔案上傳到 Ozone ===
本地檔案 / Local file: /tmp/my-file.txt
檔案大小 / File size: 154 bytes
S3 Key (目標路徑) / S3 Key (target path): my-file.txt
1. 建立 OpenDAL S3 操作符...
✅ 操作符建立成功
2. 讀取本地檔案...
✅ 檔案讀取成功，大小: 154 bytes
3. 上傳檔案到 Ozone...
✅ 檔案上傳成功！
4. 驗證上傳的檔案...
✅ 檔案驗證成功！內容完全匹配
5. 列出桶中的檔案...
✅ 找到 1 個檔案
- my-file.txt
=== ✅ 測試完成！===
檔案已成功上傳到 Ozone S3 Gateway:
s3://test-bucket/my-file.txt
```
### 使用 Python 程式碼上傳 / Upload Using Python Code
```python
import opendal
# 建立操作符 / Create operator
op = opendal.Operator(
"s3",
endpoint="http://localhost:9878",
bucket="test-bucket",
access_key_id="testuser/scm@EXAMPLE.COM",
secret_access_key="c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999",
region="us-east-1",
)
# 讀取本地檔案 / Read local file
local_file_path = "/path/to/your/local-file.txt"
with open(local_file_path, "rb") as f:
file_content = f.read()
# 上傳到 Ozone / Upload to Ozone
s3_key = "my-file-in-ozone.txt"
op.write(s3_key, file_content)
print(f"檔案已上傳到 s3://test-bucket/{s3_key}")
print(f"File uploaded to s3://test-bucket/{s3_key}")
```
---
### Rust 範例 / Rust Example
```rust
use opendal::services::S3;
use opendal::Operator;
use opendal::Result;
#[tokio::main]
async fn main() -> Result<()> {
// 建立 S3 服務建置器 / Create S3 service builder
let mut builder = S3::default();
// 設定 Ozone S3 Gateway 端點 / Configure Ozone S3 Gateway endpoint
builder.endpoint("http://localhost:9878");
builder.bucket("test-bucket");
builder.access_key_id("testuser/scm@EXAMPLE.COM");
builder.secret_access_key("c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999");
builder.region("us-east-1"); // Ozone 可以使用任意 region
// 建置操作符 / Build operator
let op: Operator = Operator::new(builder)?.finish();
// 寫入資料 / Write data
op.write("test.txt", "Hello from OpenDAL!").await?;
// 讀取資料 / Read data
let data = op.read("test.txt").await?;
println!("Content: {}", String::from_utf8_lossy(&data));
// 列出檔案 / List files
let entries = op.list("/").await?;
for entry in entries {
println!("File: {}", entry.path());
}
Ok(())
}
```
### Python 範例 / Python Example
```python
import opendal
# 建立 S3 操作符 / Create S3 operator
op = opendal.Operator(
"s3",
endpoint="http://localhost:9878",
bucket="test-bucket",
access_key_id="testuser/scm@EXAMPLE.COM",
secret_access_key="c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999",
region="us-east-1", # Ozone 可以使用任意 region
)
# 寫入資料 / Write data
op.write("test.txt", b"Hello from OpenDAL!")
# 讀取資料 / Read data
data = op.read("test.txt")
print(f"Content: {data.decode('utf-8')}")
# 列出檔案 / List files
entries = op.list("/")
for entry in entries:
print(f"File: {entry.path()}")
```
### Node.js 範例 / Node.js Example
```javascript
const { Operator } = require("opendal");
async function main() {
// 建立 S3 操作符 / Create S3 operator
const op = new Operator("s3", {
endpoint: "http://localhost:9878",
bucket: "test-bucket",
access_key_id: "testuser/scm@EXAMPLE.COM",
secret_access_key: "c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999",
region: "us-east-1", // Ozone 可以使用任意 region
});
// 寫入資料 / Write data
await op.write("test.txt", Buffer.from("Hello from OpenDAL!"));
// 讀取資料 / Read data
const data = await op.read("test.txt");
console.log(`Content: ${data.toString()}`);
// 列出檔案 / List files
const entries = await op.list("/");
for (const entry of entries) {
console.log(`File: ${entry.path()}`);
}
}
main().catch(console.error);
```
### Java 範例 / Java Example
```java
import io.opendal.BlockingOperator;
import io.opendal.Options;
public class OzoneExample {
public static void main(String[] args) {
// 建立選項 / Create options
Options options = new Options();
options.set("endpoint", "http://localhost:9878");
options.set("bucket", "test-bucket");
options.set("access_key_id", "testuser/scm@EXAMPLE.COM");
options.set("secret_access_key", "c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999");
options.set("region", "us-east-1"); // Ozone 可以使用任意 region
// 建立操作符 / Create operator
BlockingOperator op = new BlockingOperator("s3", options);
// 寫入資料 / Write data
op.write("test.txt", "Hello from OpenDAL!".getBytes());
// 讀取資料 / Read data
byte[] data = op.read("test.txt");
System.out.println("Content: " + new String(data));
// 列出檔案 / List files
op.list("/").forEach(entry -> {
System.out.println("File: " + entry.path());
});
}
}
```
---
## 常見操作範例 / Common Operations Examples
### 建立桶 / Create Bucket
**使用 AWS CLI / Using AWS CLI:**
```bash
aws s3api --endpoint http://localhost:9878 create-bucket --bucket my-bucket
```
**使用 OpenDAL (Rust) / Using OpenDAL (Rust):**
```rust
// OpenDAL 會自動處理桶的建立（如果不存在）/ OpenDAL automatically handles bucket creation (if not exists)
// 只需確保在設定中指定了正確的桶名 / Just ensure the correct bucket name is specified in config
```
### 上傳檔案 / Upload File
**使用測試腳本（推薦）/ Using Test Script (Recommended):**
我們提供了一個專門的測試腳本來上傳本地檔案：
We provide a dedicated test script to upload local files:
```bash
# 設定環境變數 / Set environment variables
export AWS_ACCESS_KEY_ID=testuser/scm@EXAMPLE.COM
export AWS_SECRET_ACCESS_KEY=c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999
# 上傳本地檔案 / Upload local file
python3 test-upload-local-file.py /path/to/your/local-file.txt s3-key-name.txt
# 如果不指定 S3 key，會使用檔案名稱 / If S3 key is not specified, filename will be used
python3 test-upload-local-file.py /path/to/your/local-file.txt
```
**使用 AWS CLI / Using AWS CLI:**
```bash
aws s3 --endpoint http://localhost:9878 cp /path/to/local/file.txt s3://my-bucket/file.txt
```
**使用 OpenDAL (Python) / Using OpenDAL (Python):**
```python
import opendal
# 建立操作符 / Create operator
op = opendal.Operator(
"s3",
endpoint="http://localhost:9878",
bucket="test-bucket",
access_key_id="testuser/scm@EXAMPLE.COM",
secret_access_key="c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999",
region="us-east-1",
)
# 讀取本地檔案並上傳 / Read local file and upload
with open("/path/to/local/file.txt", "rb") as f:
file_content = f.read()
op.write("file.txt", file_content)
print("檔案上傳成功！/ File uploaded successfully!")
```
**使用 OpenDAL (Rust) / Using OpenDAL (Rust):**
```rust
let content = std::fs::read("/path/to/local/file.txt")?;
op.write("file.txt", content).await?;
```
### 下載檔案 / Download File
**使用 AWS CLI / Using AWS CLI:**
```bash
aws s3 --endpoint http://localhost:9878 cp s3://my-bucket/file.txt /path/to/local/file.txt
```
**使用 OpenDAL (Python) / Using OpenDAL (Python):**
```python
# 從 Ozone 下載檔案到本地 / Download file from Ozone to local
data = op.read("file.txt")
# 確保內容是 bytes 類型 / Ensure content is bytes type
if isinstance(data, memoryview):
data = bytes(data)
elif not isinstance(data, bytes):
data = bytes(data)
# 寫入本地檔案 / Write to local file
with open("/path/to/local/file.txt", "wb") as f:
f.write(data)
print("檔案下載成功！/ File downloaded successfully!")
```
**使用 OpenDAL (Rust) / Using OpenDAL (Rust):**
```rust
let data = op.read("file.txt").await?;
std::fs::write("/path/to/local/file.txt", data)?;
```
### 列出檔案 / List Files
**使用 AWS CLI / Using AWS CLI:**
```bash
aws s3 --endpoint http://localhost:9878 ls s3://my-bucket/
```
**使用 OpenDAL (Rust) / Using OpenDAL (Rust):**
```rust
let entries = op.list("/").await?;
for entry in entries {
println!("{}", entry.path());
}
```
### 刪除檔案 / Delete File
**使用 AWS CLI / Using AWS CLI:**
```bash
aws s3 --endpoint http://localhost:9878 rm s3://my-bucket/file.txt
```
**使用 OpenDAL (Rust) / Using OpenDAL (Rust):**
```rust
op.delete("file.txt").await?;
```
---
## 故障排除 / Troubleshooting
### 問題 1: 無法連線到 S3 Gateway / Issue 1: Cannot Connect to S3 Gateway
**症狀 / Symptoms:**
```
Connection refused
```
**解決方案 / Solution:**
1. 檢查 S3 Gateway 是否正在執行 / Check if S3 Gateway is running:
```bash
docker compose ps s3g
```
2. 檢查日誌 / Check logs:
```bash
docker compose logs s3g
```
3. 確認端口映射 / Verify port mapping:
```bash
docker compose port s3g 9878
```
### 問題 2: 認證失敗 / Issue 2: Authentication Failed
**症狀 / Symptoms:**
```
Access Denied
```
**解決方案 / Solution:**
1. 確認存取憑證正確設定 / Verify access credentials are correctly set:
```bash
echo $AWS_ACCESS_KEY_ID
echo $AWS_SECRET_ACCESS_KEY
```
2. 檢查 Ozone 日誌 / Check Ozone logs:
```bash
docker compose logs om | grep -i auth
```
### 問題 3: 桶不存在 / Issue 3: Bucket Not Found
**症狀 / Symptoms:**
```
NoSuchBucket
```
**解決方案 / Solution:**
1. 先建立桶 / Create bucket first:
```bash
aws s3api --endpoint http://localhost:9878 create-bucket --bucket your-bucket-name
```
2. 確認桶名正確 / Verify bucket name is correct:
```bash
aws s3api --endpoint http://localhost:9878 list-buckets
```
### 問題 4: 服務啟動失敗 / Issue 4: Service Startup Failed
**症狀 / Symptoms:**
```
Container exits immediately
```
**解決方案 / Solution:**
1. 檢查完整日誌 / Check full logs:
```bash
docker compose logs --tail=100
```
2. 檢查磁盤空間 / Check disk space:
```bash
df -h
```
3. 重新啟動服務 / Restart services:
```bash
docker compose down -v
docker compose up -d
```
### 問題 5: OpenDAL 連線問題 / Issue 5: OpenDAL Connection Issues
**症狀 / Symptoms:**
```
Connection timeout or endpoint not found
```
**解決方案 / Solution:**
1. 確認端點 URL 正確 / Verify endpoint URL is correct:
- 應該使用 `http://localhost:9878`（本地）或 `http://s3g:9878`（容器內）
- Should use `http://localhost:9878` (local) or `http://s3g:9878` (inside container)
2. 檢查網路連線 / Check network connectivity:
```bash
curl http://localhost:9878
```
3. 確認 OpenDAL 版本支援 S3 服務 / Verify OpenDAL version supports S3 service:
```bash
# Rust
cargo tree | grep opendal
# Python
pip show opendal
python3 -c "import opendal; print(opendal.__version__)"
```
4. 確認 OpenDAL 已正確安裝 / Verify OpenDAL is correctly installed:
```bash
# Python
python3 -c "import opendal; print('OpenDAL installed successfully')"
# 如果出現 ImportError，重新安裝 / If ImportError occurs, reinstall:
pip3 install --upgrade opendal
```
5. 檢查桶是否存在 / Check if bucket exists:
- OpenDAL 可能會自動建立桶，但如果遇到權限問題，請先手動建立桶
- OpenDAL may auto-create buckets, but if you encounter permission issues, create the bucket manually first
---
## 停止和清理 / Stopping and Cleanup
### 停止叢集 / Stop Cluster
```bash
# 停止所有服務（保留資料）/ Stop all services (preserve data)
docker compose stop
# 停止並刪除容器（保留資料卷）/ Stop and remove containers (preserve volumes)
docker compose down
# 停止並刪除所有內容（包括資料卷）/ Stop and remove everything (including volumes)
docker compose down -v
```
### 清理資料 / Clean Data
```bash
# 刪除所有資料卷 / Remove all volumes
docker compose down -v
# 刪除特定服務的資料 / Remove specific service data
docker volume ls | grep ozone
docker volume rm <volume-name>
```
---
## 進階配置 / Advanced Configuration
### 自定義配置 / Custom Configuration
您可以透過修改 `docker-config` 檔案來自訂 Ozone 設定：
You can customize Ozone configuration by modifying the `docker-config` file:
```bash
cd hadoop-ozone/dist/src/main/compose/ozone/
vim docker-config
```
### 啟用監控 / Enable Monitoring
```bash
# 設定環境變數 / Set environment variable
export COMPOSE_FILE=docker-compose.yaml:monitoring.yaml
# 啟動叢集 / Start cluster
docker compose up -d
```
### 調整複製因子 / Adjust Replication Factor
```bash
# 設定複製因子 / Set replication factor
export OZONE_REPLICATION_FACTOR=3
# 啟動叢集 / Start cluster
docker compose up -d --scale datanode=3
```
---
## 參考資源 / Reference Resources
- **Apache Ozone 官方文檔 / Apache Ozone Official Documentation**: https://ozone.apache.org/
- **OpenDAL 官方文檔 / OpenDAL Official Documentation**: https://opendal.apache.org/
- **Docker Compose 文檔 / Docker Compose Documentation**: https://docs.docker.com/compose/
- **Ozone S3 介面文檔 / Ozone S3 Interface Documentation**: https://ozone.apache.org/docs/edge/interface/s3.html
---
## 總結 / Summary
本指南涵蓋了：
This guide covers:
1. ✅ 安裝 OpenDAL 庫 / Installing OpenDAL library
2. ✅ 使用 Docker Compose 啟動 Ozone 叢集 / Starting Ozone cluster using Docker Compose
3. ✅ 設定 S3 存取憑證 / Configuring S3 access credentials
4. ✅ 使用 OpenDAL 連線 Ozone S3 Gateway / Connecting to Ozone S3 Gateway using OpenDAL
5. ✅ 測試連線腳本 / Test connection scripts
6. ✅ 常見操作範例 / Common operation examples
7. ✅ 故障排除指南 / Troubleshooting guide
現在您已經可以開始使用 Ozone 和 OpenDAL 進行開發了！
Now you're ready to start developing with Ozone and OpenDAL!
---
## 完整測試腳本程式碼 / Complete Test Scripts Code
本節包含所有測試腳本的完整程式碼，您可以直接複製使用。
This section contains the complete code for all test scripts that you can copy and use directly.
### 1. test-s3-connection.sh - S3 Gateway 連線測試腳本
這個腳本用於測試 Ozone S3 Gateway 的基本連線。
This script is used to test basic connection to Ozone S3 Gateway.
```bash
#!/bin/bash
# 測試 Ozone S3 Gateway 連接的簡單腳本
# Simple script to test Ozone S3 Gateway connection
set -e
echo "=== 測試 Ozone S3 Gateway 連接 / Testing Ozone S3 Gateway Connection ==="
echo ""
# 設置憑證（非安全模式可以使用任意值）
export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-testuser/scm@EXAMPLE.COM}
export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999}
ENDPOINT="http://localhost:9878"
echo "使用端點 / Using endpoint: $ENDPOINT"
echo "使用 Access Key / Using Access Key: $AWS_ACCESS_KEY_ID"
echo ""
# 檢查 S3 Gateway 是否運行
echo "1. 檢查 S3 Gateway 是否運行 / Checking if S3 Gateway is running..."
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "$ENDPOINT" || echo "000")
if [ "$HTTP_CODE" = "403" ] || [ "$HTTP_CODE" = "400" ]; then
echo " ✅ S3 Gateway 正在運行 (HTTP $HTTP_CODE) / S3 Gateway is running (HTTP $HTTP_CODE)"
else
echo " ⚠️ S3 Gateway 響應異常 (HTTP $HTTP_CODE) / S3 Gateway response abnormal (HTTP $HTTP_CODE)"
echo " 請確認服務是否正常啟動 / Please verify if the service started correctly"
fi
echo ""
# 檢查是否安裝了 AWS CLI
if command -v aws &> /dev/null; then
echo "2. 使用 AWS CLI 測試連接 / Testing connection with AWS CLI..."
echo ""
# 列出桶
echo " 列出所有桶 / Listing all buckets..."
aws s3api --endpoint "$ENDPOINT" list-buckets 2>&1 || echo " 無法列出桶 / Cannot list buckets"
echo ""
# 建立測試桶
TEST_BUCKET="test-bucket-$(date +%s)"
echo " 建立測試桶: $TEST_BUCKET / Creating test bucket: $TEST_BUCKET"
if aws s3api --endpoint "$ENDPOINT" create-bucket --bucket "$TEST_BUCKET" 2>&1; then
echo " ✅ 桶建立成功 / Bucket created successfully"
# 上傳測試檔案
TEST_FILE="/tmp/test-ozone-$(date +%s).txt"
echo "Hello from Ozone S3 Gateway!" > "$TEST_FILE"
echo " 上傳測試檔案 / Uploading test file..."
if aws s3 --endpoint "$ENDPOINT" cp "$TEST_FILE" "s3://$TEST_BUCKET/test.txt" 2>&1; then
echo " ✅ 檔案上傳成功 / File uploaded successfully"
# 列出桶中的檔案
echo " 列出桶中的檔案 / Listing files in bucket..."
aws s3 --endpoint "$ENDPOINT" ls "s3://$TEST_BUCKET/" 2>&1 || true
# 清理
echo " 清理測試檔案 / Cleaning up test file..."
rm -f "$TEST_FILE"
fi
else
echo " ⚠️ 桶建立失敗 / Bucket creation failed"
fi
echo ""
else
echo "2. AWS CLI 未安裝 / AWS CLI not installed"
echo " 請安裝 AWS CLI 或使用 OpenDAL 進行測試 / Please install AWS CLI or use OpenDAL for testing"
echo ""
fi
echo "=== 測試完成 / Test Complete ==="
echo ""
echo "下一步 / Next Steps:"
echo "1. 設置環境變數 / Set environment variables:"
echo " export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID"
echo " export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY"
echo ""
echo "2. 使用 OpenDAL 連接 Ozone S3 Gateway / Use OpenDAL to connect to Ozone S3 Gateway"
echo " 請參考文件中的 OpenDAL 範例 / Please refer to OpenDAL examples in the documentation"
```
**使用方法 / Usage:**
```bash
chmod +x test-s3-connection.sh
./test-s3-connection.sh
```
---
### 2. test-opendal-ozone.py - OpenDAL 連線測試腳本
這個腳本用於測試 OpenDAL 與 Ozone S3 Gateway 的連線和基本操作。
This script is used to test OpenDAL connection to Ozone S3 Gateway and basic operations.
```python
#!/usr/bin/env python3
"""
測試 OpenDAL 連接 Ozone S3 Gateway
Test OpenDAL connection to Ozone S3 Gateway
"""
import os
import sys
# 設置環境變數（如果尚未設置）
os.environ.setdefault('AWS_ACCESS_KEY_ID', 'testuser/scm@EXAMPLE.COM')
os.environ.setdefault('AWS_SECRET_ACCESS_KEY', 'c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999')
try:
import opendal
print("✅ OpenDAL 已安裝 / OpenDAL is installed")
print(f" 版本 / Version: {opendal.__version__ if hasattr(opendal, '__version__') else 'unknown'}\n")
except ImportError:
print("❌ OpenDAL 未安裝 / OpenDAL is not installed")
print("\n請先安裝 OpenDAL / Please install OpenDAL first:")
print(" pip3 install opendal")
sys.exit(1)
def test_opendal_connection():
"""測試 OpenDAL 連接 Ozone S3 Gateway"""
print("=== 測試 OpenDAL 連接 Ozone S3 Gateway ===")
print("=== Testing OpenDAL Connection to Ozone S3 Gateway ===\n")
# Ozone S3 Gateway 設定
endpoint = "http://localhost:9878"
bucket = "test-bucket"
access_key_id = os.environ.get('AWS_ACCESS_KEY_ID')
secret_access_key = os.environ.get('AWS_SECRET_ACCESS_KEY')
print(f"端點 / Endpoint: {endpoint}")
print(f"桶名 / Bucket: {bucket}")
print(f"Access Key ID: {access_key_id}\n")
try:
# 建立 S3 操作符
print("1. 建立 OpenDAL S3 操作符 / Creating OpenDAL S3 operator...")
op = opendal.Operator(
"s3",
endpoint=endpoint,
bucket=bucket,
access_key_id=access_key_id,
secret_access_key=secret_access_key,
region="us-east-1", # Ozone 可以使用任意 region
)
print(" ✅ 操作符建立成功 / Operator created successfully\n")
# 檢查桶是否存在，如果不存在則建立
print("2. 檢查桶是否存在 / Checking if bucket exists...")
try:
# 嘗試列出桶中的檔案來檢查桶是否存在
list(op.list("/"))
print(" ✅ 桶已存在 / Bucket exists\n")
except Exception as e:
print(f" ⚠️ 桶不存在，需要先建立桶 / Bucket does not exist, need to create bucket first")
print(f" 錯誤訊息 / Error message: {str(e)[:100]}")
print(f"\n 請使用以下方法之一建立桶 / Please use one of the following methods to create bucket:")
print(f" 1. 使用 Docker 容器內的命令 / Use command inside Docker container:")
print(f" docker compose exec om bash -c 'ozone sh bucket create /s3v/{bucket}'")
print(f" 2. 使用 Ozone Manager REST API / Use Ozone Manager REST API:")
print(f" curl -X PUT http://localhost:9874/api/v1/bucket/s3v/{bucket}")
print(f" 3. 安裝 AWS CLI 後使用 / After installing AWS CLI:")
print(f" aws s3api --endpoint {endpoint} create-bucket --bucket {bucket}\n")
print(f" 建立桶後，請重新執行此測試腳本 / After creating bucket, please run this test script again.\n")
return False
# 測試寫入
print("3. 測試寫入資料 / Testing write operation...")
test_content = b"Hello from OpenDAL! This is a test file."
test_key = "test-file.txt"
op.write(test_key, test_content)
print(f" ✅ 檔案寫入成功: {test_key} / File written successfully: {test_key}\n")
# 測試讀取
print("4. 測試讀取資料 / Testing read operation...")
read_content = op.read(test_key)
# 確保內容是 bytes 類型
if isinstance(read_content, memoryview):
read_content = bytes(read_content)
elif not isinstance(read_content, bytes):
read_content = bytes(read_content)
print(f" ✅ 檔案讀取成功 / File read successfully")
print(f" 內容 / Content: {read_content.decode('utf-8')}\n")
# 測試列出文件
print("5. 測試列出檔案 / Testing list operation...")
entries = op.list("/")
file_list = list(entries)
print(f" ✅ 找到 {len(file_list)} 個檔案 / Found {len(file_list)} file(s)")
for entry in file_list:
# 取得路徑，可能是 path 屬性或 path() 方法
try:
path = entry.path() if callable(entry.path) else entry.path
except:
path = str(entry)
print(f" - {path}")
print()
# 測試刪除
print("6. 測試刪除檔案 / Testing delete operation...")
op.delete(test_key)
print(f" ✅ 檔案刪除成功: {test_key} / File deleted successfully: {test_key}\n")
print("=== ✅ 所有測試通過！/ All tests passed! ===")
return True
except Exception as e:
print(f"\n❌ 錯誤 / Error: {type(e).__name__}")
print(f" 訊息 / Message: {str(e)}\n")
# 提供故障排除建議
print("故障排除建議 / Troubleshooting suggestions:")
print("1. 確認 Ozone S3 Gateway 正在執行 / Verify Ozone S3 Gateway is running:")
print(" curl http://localhost:9878")
print("2. 確認環境變數已設定 / Verify environment variables are set:")
print(f" AWS_ACCESS_KEY_ID={access_key_id}")
print(f" AWS_SECRET_ACCESS_KEY={'*' * 20}")
print("3. 確認桶是否存在（如果不存在，OpenDAL 可能會自動建立）/")
print(" Verify bucket exists (OpenDAL may auto-create if it doesn't exist)")
return False
if __name__ == "__main__":
success = test_opendal_connection()
sys.exit(0 if success else 1)
```
**使用方法 / Usage:**
```bash
chmod +x test-opendal-ozone.py
python3 test-opendal-ozone.py
```
---
### 3. test-upload-local-file.py - 本地檔案上傳測試腳本
這個腳本用於將本地檔案透過 OpenDAL 上傳到 Ozone。
This script is used to upload local files to Ozone via OpenDAL.
```python
#!/usr/bin/env python3
"""
測試將本地檔案透過 OpenDAL 上傳到 Ozone S3 Gateway
Test uploading local files to Ozone S3 Gateway via OpenDAL
"""
import os
import sys
import tempfile
from pathlib import Path
# 設置環境變數（如果尚未設置）
os.environ.setdefault('AWS_ACCESS_KEY_ID', 'testuser/scm@EXAMPLE.COM')
os.environ.setdefault('AWS_SECRET_ACCESS_KEY', 'c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999')
try:
import opendal
print("✅ OpenDAL 已安裝 / OpenDAL is installed\n")
except ImportError:
print("❌ OpenDAL 未安裝 / OpenDAL is not installed")
print("\n請先安裝 OpenDAL / Please install OpenDAL first:")
print(" pip3 install opendal")
sys.exit(1)
def create_test_file(content, filename=None):
"""建立一個測試檔案 / Create a test file"""
if filename is None:
# 建立臨時檔案
fd, filename = tempfile.mkstemp(suffix='.txt', prefix='ozone-test-', text=True)
os.close(fd)
with open(filename, 'w', encoding='utf-8') as f:
f.write(content)
return filename
def upload_local_file_to_ozone(local_file_path, s3_key=None):
"""將本地檔案上傳到 Ozone / Upload local file to Ozone"""
print("=== 測試將本地檔案上傳到 Ozone ===")
print("=== Testing Upload Local File to Ozone ===\n")
# 檢查本地檔案是否存在
if not os.path.exists(local_file_path):
print(f"❌ 錯誤：本地檔案不存在 / Error: Local file does not exist: {local_file_path}")
return False
# 取得檔案資訊
file_size = os.path.getsize(local_file_path)
file_name = os.path.basename(local_file_path)
print(f"本地檔案 / Local file: {local_file_path}")
print(f"檔案名稱 / File name: {file_name}")
print(f"檔案大小 / File size: {file_size} bytes\n")
# Ozone S3 Gateway 設定
endpoint = "http://localhost:9878"
bucket = "test-bucket"
access_key_id = os.environ.get('AWS_ACCESS_KEY_ID')
secret_access_key = os.environ.get('AWS_SECRET_ACCESS_KEY')
# 如果沒有指定 S3 key，使用文件名
if s3_key is None:
s3_key = file_name
print(f"端點 / Endpoint: {endpoint}")
print(f"桶名 / Bucket: {bucket}")
print(f"S3 Key (目標路徑) / S3 Key (target path): {s3_key}\n")
try:
# 建立 OpenDAL S3 操作符
print("1. 建立 OpenDAL S3 操作符 / Creating OpenDAL S3 operator...")
op = opendal.Operator(
"s3",
endpoint=endpoint,
bucket=bucket,
access_key_id=access_key_id,
secret_access_key=secret_access_key,
region="us-east-1",
)
print(" ✅ 操作符建立成功 / Operator created successfully\n")
# 讀取本地檔案
print("2. 讀取本地檔案 / Reading local file...")
with open(local_file_path, 'rb') as f:
file_content = f.read()
print(f" ✅ 檔案讀取成功，大小 / File read successfully, size: {len(file_content)} bytes\n")
# 上傳檔案到 Ozone
print(f"3. 上傳檔案到 Ozone / Uploading file to Ozone...")
print(f" 目標路徑 / Target path: s3://{bucket}/{s3_key}")
op.write(s3_key, file_content)
print(f" ✅ 檔案上傳成功！/ File uploaded successfully!\n")
# 驗證上傳
print("4. 驗證上傳的檔案 / Verifying uploaded file...")
uploaded_content = op.read(s3_key)
# 確保內容是 bytes 類型
if isinstance(uploaded_content, memoryview):
uploaded_content = bytes(uploaded_content)
elif not isinstance(uploaded_content, bytes):
uploaded_content = bytes(uploaded_content)
if uploaded_content == file_content:
print(f" ✅ 檔案驗證成功！內容完全匹配 / File verification successful! Content matches perfectly")
print(f" 上傳大小 / Uploaded size: {len(uploaded_content)} bytes\n")
else:
print(f" ⚠️ 檔案內容不匹配 / File content mismatch")
print(f" 原始大小 / Original size: {len(file_content)} bytes")
print(f" 上傳大小 / Uploaded size: {len(uploaded_content)} bytes\n")
# 列出桶中的文件
print("5. 列出桶中的檔案 / Listing files in bucket...")
entries = op.list("/")
file_list = list(entries)
print(f" ✅ 找到 {len(file_list)} 個檔案 / Found {len(file_list)} file(s)")
for entry in file_list:
try:
path = entry.path() if callable(entry.path) else entry.path
except:
path = str(entry)
print(f" - {path}")
print()
print("=== ✅ 測試完成！/ Test Complete! ===")
print(f"\n檔案已成功上傳到 Ozone S3 Gateway:")
print(f" s3://{bucket}/{s3_key}")
print(f"\nFile successfully uploaded to Ozone S3 Gateway:")
print(f" s3://{bucket}/{s3_key}\n")
return True
except Exception as e:
print(f"\n❌ 錯誤 / Error: {type(e).__name__}")
print(f" 訊息 / Message: {str(e)}\n")
# 提供故障排除建議
print("故障排除建議 / Troubleshooting suggestions:")
print("1. 確認 Ozone S3 Gateway 正在執行 / Verify Ozone S3 Gateway is running:")
print(" curl http://localhost:9878")
print("2. 確認桶是否存在 / Verify bucket exists:")
print(" docker compose exec om bash -c 'ozone sh bucket create /s3v/test-bucket'")
print("3. 確認環境變數已設置 / Verify environment variables are set:")
print(f" AWS_ACCESS_KEY_ID={access_key_id}")
return False
def main():
"""主函數 / Main function"""
# 如果提供了命令行參數，使用該文件
if len(sys.argv) > 1:
local_file = sys.argv[1]
s3_key = sys.argv[2] if len(sys.argv) > 2 else None
else:
# 否則建立一個測試檔案
print("未提供檔案路徑，建立測試檔案...")
print("No file path provided, creating test file...\n")
test_content = """這是一個測試檔案 / This is a test file
測試將本地檔案上傳到 Ozone / Test uploading local file to Ozone
檔案內容 / File content:
- 中文測試 / Chinese test: 你好，世界！
- English test: Hello, World!
- 數字測試 / Number test: 1234567890
- 特殊字符 / Special characters: !@#$%^&*()
時間戳 / Timestamp: 2025-11-15
"""
local_file = create_test_file(test_content, "test-upload-file.txt")
s3_key = "uploaded-test-file.txt"
print(f"已建立測試檔案 / Test file created: {local_file}\n")
# 上傳檔案
success = upload_local_file_to_ozone(local_file, s3_key)
# 如果是臨時檔案，可選清理
if local_file.startswith(tempfile.gettempdir()):
try:
os.remove(local_file)
print(f"已清理臨時檔案 / Cleaned up temporary file: {local_file}")
except:
pass
sys.exit(0 if success else 1)
if __name__ == "__main__":
main()
```
**使用方法 / Usage:**
```bash
chmod +x test-upload-local-file.py
# 上傳指定檔案 / Upload specified file
python3 test-upload-local-file.py /path/to/your/file.txt [s3-key-name]
# 建立測試檔案並上傳 / Create test file and upload
python3 test-upload-local-file.py
```
---
### 4. test-move-file.py - 檔案移動測試腳本
這個腳本用於測試在 Ozone 中使用 OpenDAL 移動檔案。
This script is used to test moving files in Ozone using OpenDAL.
```python
#!/usr/bin/env python3
"""
測試在 Ozone 中使用 OpenDAL 移動文件
Test moving files in Ozone using OpenDAL
"""
import os
import sys
# 設置環境變數（如果尚未設置）
os.environ.setdefault('AWS_ACCESS_KEY_ID', 'testuser/scm@EXAMPLE.COM')
os.environ.setdefault('AWS_SECRET_ACCESS_KEY', 'c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999')
try:
import opendal
print("✅ OpenDAL 已安裝 / OpenDAL is installed\n")
except ImportError:
print("❌ OpenDAL 未安裝 / OpenDAL is not installed")
print("\n請先安裝 OpenDAL / Please install OpenDAL first:")
print(" pip3 install opendal")
sys.exit(1)
def move_file_in_ozone(op, source_key, target_key, overwrite=False):
"""
在 Ozone 中移動文件 / Move file in Ozone
參數 / Parameters:
- op: OpenDAL 操作符 / OpenDAL operator
- source_key: 來源檔案路徑 / Source file path
- target_key: 目標檔案路徑 / Target file path
- overwrite: 是否覆蓋已存在的目標檔案 / Whether to overwrite existing target file
返回 / Returns:
- True: 移動成功 / Move successful
- False: 移動失敗 / Move failed
"""
print(f"=== 移動文件 / Moving File ===")
print(f"源文件 / Source: {source_key}")
print(f"目標文件 / Target: {target_key}\n")
# 條件檢查 1: 檢查來源檔案是否存在 / Condition 1: Check if source file exists
print("1. 檢查源文件是否存在 / Checking if source file exists...")
try:
source_data = op.read(source_key)
# 確保是 bytes 類型 / Ensure it's bytes type
if isinstance(source_data, memoryview):
source_data = bytes(source_data)
elif not isinstance(source_data, bytes):
source_data = bytes(source_data)
print(f" ✅ 源文件存在，大小 / Source file exists, size: {len(source_data)} bytes\n")
except Exception as e:
print(f" ❌ 錯誤：無法讀取源文件 / Error: Cannot read source file: {source_key}")
print(f" 原因 / Reason: {e}\n")
return False
# 條件檢查 2: 檢查目標檔案是否存在 / Condition 2: Check if target file exists
print("2. 檢查目標文件是否存在 / Checking if target file exists...")
if not overwrite:
try:
op.read(target_key)
print(f" ⚠️ 警告：目標文件已存在 / Warning: Target file already exists: {target_key}")
print(f" 使用 overwrite=True 來覆蓋 / Use overwrite=True to overwrite\n")
return False
except:
print(f" ✅ 目標文件不存在，可以繼續 / Target file does not exist, can proceed\n")
else:
print(f" ℹ️ 覆蓋模式已啟用 / Overwrite mode enabled\n")
# 步驟 1: 複製文件到目標位置 / Step 1: Copy file to target location
print("3. 複製文件到目標位置 / Copying file to target location...")
try:
op.write(target_key, source_data)
print(f" ✅ 文件已複製到目標位置 / File copied to target: {target_key}\n")
except Exception as e:
print(f" ❌ 錯誤：無法寫入目標文件 / Error: Cannot write target file: {target_key}")
print(f" 原因 / Reason: {e}\n")
return False
# 驗證目標文件 / Verify target file
print("4. 驗證目標文件 / Verifying target file...")
try:
target_data = op.read(target_key)
if isinstance(target_data, memoryview):
target_data = bytes(target_data)
elif not isinstance(target_data, bytes):
target_data = bytes(target_data)
if target_data == source_data:
print(f" ✅ 目標文件驗證成功，內容匹配 / Target file verified, content matches\n")
else:
print(f" ⚠️ 警告：目標文件內容不匹配 / Warning: Target file content mismatch\n")
except Exception as e:
print(f" ⚠️ 警告：無法驗證目標文件 / Warning: Cannot verify target file: {e}\n")
# 步驟 2: 刪除源文件 / Step 2: Delete source file
print("5. 刪除源文件 / Deleting source file...")
try:
op.delete(source_key)
print(f" ✅ 源文件已刪除 / Source file deleted: {source_key}\n")
except Exception as e:
print(f" ⚠️ 警告：無法刪除源文件 / Warning: Cannot delete source file: {source_key}")
print(f" 原因 / Reason: {e}")
print(f" 目標文件已創建，但源文件仍存在 / Target file created, but source file still exists\n")
return False
print("=== ✅ 文件移動成功！/ File move successful! ===")
print(f" 從 / From: {source_key}")
print(f" 到 / To: {target_key}\n")
return True
def main():
"""主函數 / Main function"""
# Ozone S3 Gateway 設定
endpoint = "http://localhost:9878"
bucket = "test-bucket"
access_key_id = os.environ.get('AWS_ACCESS_KEY_ID')
secret_access_key = os.environ.get('AWS_SECRET_ACCESS_KEY')
print("=== 測試在 Ozone 中使用 OpenDAL 移動檔案 ===")
print("=== Testing File Movement in Ozone using OpenDAL ===\n")
# 建立 OpenDAL S3 操作符
print("建立 OpenDAL S3 操作符 / Creating OpenDAL S3 operator...")
op = opendal.Operator(
"s3",
endpoint=endpoint,
bucket=bucket,
access_key_id=access_key_id,
secret_access_key=secret_access_key,
region="us-east-1",
)
print("✅ 操作符建立成功 / Operator created successfully\n")
# 建立一個測試檔案
print("建立測試檔案 / Creating test file...")
test_content = b"This is a test file for moving operation.\nCreated at: 2025-11-15\n"
test_key = "file-to-move.txt"
try:
op.write(test_key, test_content)
print(f"✅ 測試檔案已建立: {test_key}\n")
except Exception as e:
print(f"❌ 無法建立測試檔案 / Cannot create test file: {e}\n")
return False
# 移動檔案
target_key = "moved-file.txt"
success = move_file_in_ozone(op, test_key, target_key, overwrite=False)
# 列出桶中的檔案
print("\n列出桶中的檔案 / Listing files in bucket...")
try:
entries = op.list("/")
file_list = list(entries)
print(f"找到 {len(file_list)} 個檔案 / Found {len(file_list)} file(s):")
for entry in file_list:
try:
path = entry.path() if callable(entry.path) else entry.path
except:
path = str(entry)
print(f" - {path}")
except Exception as e:
print(f"無法列出檔案 / Cannot list files: {e}")
sys.exit(0 if success else 1)
if __name__ == "__main__":
main()
```
**使用方法 / Usage:**
```bash
chmod +x test-move-file.py
python3 test-move-file.py
```
---
### 5. 檔案移動函數（可重用程式碼）/ File Movement Function (Reusable Code)
這是一個可以在您的應用程式中重用的檔案移動函數。
This is a reusable file movement function that can be used in your applications.
```python
import opendal
def move_file_in_ozone(op, source_key, target_key, overwrite=False):
"""
在 Ozone 中移動檔案 / Move file in Ozone
參數 / Parameters:
- op: OpenDAL 操作符 / OpenDAL operator
- source_key: 來源檔案路徑 / Source file path
- target_key: 目標檔案路徑 / Target file path
- overwrite: 是否覆蓋已存在的目標檔案 / Whether to overwrite existing target file
返回 / Returns:
- True: 移動成功 / Move successful
- False: 移動失敗 / Move failed
"""
# 條件檢查 1: 檢查來源檔案是否存在 / Condition 1: Check if source file exists
try:
source_data = op.read(source_key)
# 確保是 bytes 類型 / Ensure it's bytes type
if isinstance(source_data, memoryview):
source_data = bytes(source_data)
elif not isinstance(source_data, bytes):
source_data = bytes(source_data)
except Exception as e:
print(f"❌ 錯誤：無法讀取來源檔案 / Error: Cannot read source file: {source_key}")
print(f" 原因 / Reason: {e}")
return False
# 條件檢查 2: 檢查目標檔案是否存在 / Condition 2: Check if target file exists
if not overwrite:
try:
op.read(target_key)
print(f"⚠️ 警告：目標檔案已存在 / Warning: Target file already exists: {target_key}")
print(f" 使用 overwrite=True 來覆蓋 / Use overwrite=True to overwrite")
return False
except:
# 目標檔案不存在，可以繼續 / Target file does not exist, can proceed
pass
# 步驟 1: 複製檔案到目標位置 / Step 1: Copy file to target location
try:
op.write(target_key, source_data)
print(f"✅ 檔案已複製到目標位置 / File copied to target: {target_key}")
except Exception as e:
print(f"❌ 錯誤：無法寫入目標檔案 / Error: Cannot write target file: {target_key}")
print(f" 原因 / Reason: {e}")
return False
# 步驟 2: 刪除來源檔案 / Step 2: Delete source file
try:
op.delete(source_key)
print(f"✅ 來源檔案已刪除 / Source file deleted: {source_key}")
except Exception as e:
print(f"⚠️ 警告：無法刪除來源檔案 / Warning: Cannot delete source file: {source_key}")
print(f" 原因 / Reason: {e}")
print(f" 目標檔案已建立，但來源檔案仍存在 / Target file created, but source file still exists")
# 可以選擇回滾（刪除目標檔案）/ Can choose to rollback (delete target file)
return False
print(f"✅ 檔案移動成功！/ File move successful!")
print(f" 從 / From: {source_key}")
print(f" 到 / To: {target_key}")
return True
# 使用範例 / Usage Example
op = opendal.Operator(
"s3",
endpoint="http://localhost:9878",
bucket="test-bucket",
access_key_id="testuser/scm@EXAMPLE.COM",
secret_access_key="c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999",
region="us-east-1",
)
# 移動文件 / Move file
move_file_in_ozone(op, "old-file.txt", "new-file.txt", overwrite=False)
```
---
### 6. 帶重試機制的上傳函數 / Upload Function with Retry Mechanism
這是一個帶有錯誤重試機制的上傳函數，適用於生產環境。
This is an upload function with error retry mechanism, suitable for production environments.
```python
import opendal
import time
def upload_with_retry(op, key, data, max_retries=3, initial_delay=1):
"""
帶重試機制的上傳函數 / Upload function with retry mechanism
參數 / Parameters:
- op: OpenDAL 操作符 / OpenDAL operator
- key: S3 鍵名 / S3 key name
- data: 要上傳的資料 / Data to upload
- max_retries: 最大重試次數 / Maximum number of retries
- initial_delay: 初始延遲時間（秒）/ Initial delay time (seconds)
返回 / Returns:
- True: 上傳成功 / Upload successful
- False: 上傳失敗 / Upload failed
"""
for attempt in range(max_retries):
try:
op.write(key, data)
if attempt > 0:
print(f"✅ 上傳成功（第 {attempt + 1} 次嘗試）/ Upload successful (attempt {attempt + 1})")
return True
except Exception as e:
if attempt < max_retries - 1:
delay = initial_delay * (2 ** attempt) # 指數退避 / Exponential backoff
print(f"⚠️ 上傳失敗（嘗試 {attempt + 1}/{max_retries}），{delay} 秒後重試...")
print(f" Warning: Upload failed (attempt {attempt + 1}/{max_retries}), retrying in {delay} seconds...")
print(f" 錯誤 / Error: {str(e)[:100]}")
time.sleep(delay)
continue
else:
print(f"❌ 上傳失敗，已達到最大重試次數 / Upload failed, maximum retries reached")
print(f" 錯誤 / Error: {e}")
return False
return False
# 使用範例 / Usage Example
op = opendal.Operator(
"s3",
endpoint="http://localhost:9878",
bucket="test-bucket",
access_key_id="testuser/scm@EXAMPLE.COM",
secret_access_key="c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999",
region="us-east-1",
)
# 讀取本地文件並上傳 / Read local file and upload
with open("/path/to/local/file.txt", "rb") as f:
file_content = f.read()
# 使用重試機制上傳 / Upload with retry mechanism
upload_with_retry(op, "file.txt", file_content, max_retries=3)
```
---
### 7. 批次上傳函數 / Batch Upload Function
這是一個批次上傳多個檔案的函數。
This is a function for batch uploading multiple files.
```python
import opendal
import os
from pathlib import Path
def batch_upload_files(op, local_dir, s3_prefix="", max_workers=5):
"""
批次上傳本地目錄中的檔案到 Ozone / Batch upload files from local directory to Ozone
參數 / Parameters:
- op: OpenDAL 操作符 / OpenDAL operator
- local_dir: 本地目錄路徑 / Local directory path
- s3_prefix: S3 前綴（可選）/ S3 prefix (optional)
- max_workers: 最大並行數 / Maximum concurrency
返回 / Returns:
- dict: 上傳結果統計 / Upload result statistics
"""
results = {
"success": 0,
"failed": 0,
"total": 0,
"errors": []
}
local_path = Path(local_dir)
if not local_path.exists() or not local_path.is_dir():
print(f"❌ 錯誤：目錄不存在 / Error: Directory does not exist: {local_dir}")
return results
# 取得所有檔案 / Get all files
files = list(local_path.rglob("*"))
files = [f for f in files if f.is_file()]
results["total"] = len(files)
print(f"找到 {results['total']} 個檔案 / Found {results['total']} files")
print(f"開始批次上傳... / Starting batch upload...\n")
for file_path in files:
try:
# 計算相對路徑 / Calculate relative path
relative_path = file_path.relative_to(local_path)
s3_key = f"{s3_prefix}/{relative_path}".lstrip("/") if s3_prefix else str(relative_path)
# 讀取檔案 / Read file
with open(file_path, "rb") as f:
file_content = f.read()
# 上傳檔案 / Upload file
op.write(s3_key, file_content)
results["success"] += 1
print(f"✅ [{results['success']}/{results['total']}] {file_path.name} -> {s3_key}")
except Exception as e:
results["failed"] += 1
error_msg = f"❌ [{results['failed']}/{results['total']}] {file_path.name}: {str(e)}"
results["errors"].append(error_msg)
print(error_msg)
print(f"\n=== 批次上傳完成 / Batch upload complete ===")
print(f"成功 / Success: {results['success']}")
print(f"失敗 / Failed: {results['failed']}")
print(f"總計 / Total: {results['total']}")
return results
# 使用範例 / Usage Example
op = opendal.Operator(
"s3",
endpoint="http://localhost:9878",
bucket="test-bucket",
access_key_id="testuser/scm@EXAMPLE.COM",
secret_access_key="c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999",
region="us-east-1",
)
# 批次上傳 / Batch upload
results = batch_upload_files(op, "/path/to/local/directory", s3_prefix="uploads")
```
---
### 腳本檔案位置 / Script File Locations
所有測試腳本都位於專案根目錄：
All test scripts are located in the project root directory:
- `test-s3-connection.sh` - S3 Gateway 連線測試
- `test-opendal-ozone.py` - OpenDAL 連線和基本操作測試
- `test-upload-local-file.py` - 本地檔案上傳測試
- `test-move-file.py` - 檔案移動測試
Sign in with Wallet
Connect another wallet