---
# System prepended metadata

title: '[ML] 8.1: Elyra  Airflow & Minio'
tags: [管線自動化, 流程控制, MLOps, ML]

---

[ML] 8.1: Elyra  Airflow & Minio
===
###### tags: `ML`
###### tags: `ML`, `流程控制`, `MLOps`, `管線自動化`

<br>

[TOC]

<br>

:::info
:bulb: **測試網站**
- ### JupyterLab + Elyra 套件
    http://10.78.26.241:8889/lab?token=95a65ee10228a6611e69a17b0c6e428dc4b8e891b489e856
- ### Airflow-web
    http://10.78.26.241:9090/
    admin/admin
- ### Minio
    http://10.78.26.241:9000/
    minioadmin/minioadmin
:::


# Elyra
## 簡介
- JupyterLab 筆記本的擴充套件
- 功能
    - 編排工作流程(workflow)的工具，也就是安排 node 的執行流程
    - 可在本地端執行，亦可在雲端 airflow, kubeflow 平台上執行

## [elyra-ai](https://github.com/elyra-ai/elyra)
- [hello_world](https://github.com/elyra-ai/examples/tree/master/pipelines/hello_world)
- [hello_world_apache_airflow](https://github.com/elyra-ai/examples/tree/master/pipelines/hello_world_apache_airflow)
    ![](https://i.imgur.com/TODOLaL.png)
    - [Analyzing COVID-19 time series data](https://github.com/CODAIT/covid-notebooks)

<br>

## 安裝 elyra
- ### [Elyra Documentation](https://elyra.readthedocs.io/en/latest/getting_started/installation.html)
    - ### 前製安裝作業
        - [Node.js 12+](https://nodejs.org/en/)
            - [How can I update my nodeJS to the latest version?](https://askubuntu.com/questions/426750/how-can-i-update-my-nodejs-to-the-latest-version)
                變更前：
                ```
                $ node --version
                v10.15.3
                ```
                變更後：
                ```
                $ node --version
                v14.16.1
                ```
        - Python 3.x
    - ### 三種安裝方式
        - pip
        - conda
        - Build from source
    - ### 使用 pip 安裝
        建立虛擬環境
        ```bash
        $ virtualenv -p python3 elyra_env
        $ source elyra_env/bin/activate
        ```
        安裝 Elyra
        ```bash
        # for Ubuntu
        $ pip3 install elyra && jupyter lab build
        ```
        
        啟動 jupyter
        ```bash
        $ jupyter lab --debug
        ```
        
        啟動後，Launcher 頁籤就有 4 個選項
        ![](https://i.imgur.com/Ec3bNeR.png)
    - ### 可以直接透過 docker 來使用
        ```bash
        # 不能 work 的 command
        $ docker run -it -p 8888:8888 elyra/elyra:dev jupyter lab --debug
        ```
        - 不要下 `-d` (daemon)
            - 會看不到 log
            - log 中有要登入的 token
        - 選擇性參數
            - `-v source_path:dest_path` (掛載檔案進來)
            - `-w target_workspace` (設定工作區)
        - 遇到 error: Permission denied
            docker 的使用者是 `jovyan`
            ![](https://i.imgur.com/XHQHOYB.png)
            - `-u root:root`
            - 又遇到 error
                ```
                Running as root is not recommended. Use --allow-root to bypass.
                ```
                - `--allow-root`
        - 完整 command
            ```
            $ docker run --rm -it \
              -v ~/tj_tsai/workspace/elyra_workflow_20210428:/workspace \
              -w /workspace \
              -p 18888:8888 \
              -u root:root \
              elyra/elyra:latest jupyter lab --debug --allow-root
            ```
    - ### 佈署到 K8s
        ```yaml=
        apiVersion: v1
        kind: Pod
        metadata:
          name: tj-elyra-pod
          labels:
            app: tj-elyra-server
        spec:
          containers:
          - name: tj-elyra-container
            image: elyra/elyra:latest
            command: ["jupyter", "lab", "--debug"]
            ports:
            - containerPort: 8888

        ---

        apiVersion: v1
        kind: Service
        metadata:
          name: tj-elyra-service
        spec:
          type: NodePort
          ports:
          - targetPort: 8888
            port: 80
            nodePort: 30002
          selector:
            app: tj-elyra-server
        ```
    - ### 客製化 runtime image
        - [Creating a custom runtime container image](https://elyra.readthedocs.io/en/latest/recipes/creating-a-custom-runtime-image.html)

<br>

## 測試範例
- [下載範例](https://github.com/elyra-ai/examples/tree/master/pipelines/hello_world#setup)
    ```
    git clone https://github.com/elyra-ai/examples.git
    ```

<br>

## 相依性：檔案必須存在
- ### 設定
    ![](https://i.imgur.com/1wcOJA5.png)
- ### 找不到檔案：penguin.csv
    ![](https://i.imgur.com/JHIBqu4.png)
    ```
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/elyra/pipeline/processor.py", line 275, in _upload_dependencies_to_object_store
        dependency_archive_path = self._generate_dependency_archive(operation)
      File "/opt/conda/lib/python3.8/site-packages/elyra/pipeline/processor.py", line 261, in _generate_dependency_archive
        archive_artifact = create_temp_archive(archive_name=archive_artifact_name,
      File "/opt/conda/lib/python3.8/site-packages/elyra/util/archive.py", line 124, in create_temp_archive
        raise FileNotFoundError(filenames_set - matched_set)  # Only include the missing filenames
    FileNotFoundError: {'penguin.csv'}

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "/opt/conda/lib/python3.8/site-packages/tornado/web.py", line 1704, in _execute
        result = await result
      File "/opt/conda/lib/python3.8/site-packages/elyra/pipeline/handlers.py", line 89, in post
        response = await PipelineProcessorManager.instance().process(pipeline)
      File "/opt/conda/lib/python3.8/site-packages/elyra/pipeline/processor.py", line 78, in process
        res = await asyncio.get_event_loop().run_in_executor(None, processor.process, pipeline)
      File "/opt/conda/lib/python3.8/concurrent/futures/thread.py", line 57, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/opt/conda/lib/python3.8/site-packages/elyra/pipeline/processor_airflow.py", line 70, in process
        pipeline_filepath = self.create_pipeline_file(pipeline=pipeline,
      File "/opt/conda/lib/python3.8/site-packages/elyra/pipeline/processor_airflow.py", line 249, in create_pipeline_file
        notebook_ops = self._cc_pipeline(pipeline, pipeline_name)
      File "/opt/conda/lib/python3.8/site-packages/elyra/pipeline/processor_airflow.py", line 210, in _cc_pipeline
        self._upload_dependencies_to_object_store(runtime_configuration,
      File "/opt/conda/lib/python3.8/site-packages/elyra/pipeline/processor.py", line 295, in _upload_dependencies_to_object_store
        raise FileNotFoundError("Node '{}' referenced dependencies that were not found: {}".
    FileNotFoundError: Node 'step2_select_coumns' referenced dependencies that were not found: {'penguin.csv'}
    ```

<br>

## 在 airflow 上執行
- ### 安裝 Airflow
    - 參考 [**安裝-Airflow**](#安裝-Airflow1) 章節
- ### 設定 Apache Airflow runtimes
    [![](https://i.imgur.com/mQhc5nY.png)](https://i.imgur.com/mQhc5nY.png)
    - [GitHub personal access token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token)
        - [Runtime Configuration](https://elyra.readthedocs.io/en/latest/user_guide/runtime-conf.html)
            - [GitHub Personal Access Token (github_repo_token)](https://elyra.readthedocs.io/en/latest/user_guide/runtime-conf.html#github-personal-access-token-github-repo-token)
    - S3 使用 AWS 時 ，不清楚要如何取得 bucket 的端點
    - 這邊是使用 Minio，參考 [**安裝 Minio**](#安裝-Minio) 章節
- ### 啟動執行
    ![](https://i.imgur.com/xISwynf.png)
    
    ![](https://i.imgur.com/LQSCls1.png)
- ### 執行成功
    ==Elyra:==
    ![](https://i.imgur.com/XS4Pcgh.png)
    <br>
    
    ==Gitlab 狀態：==
    ![](https://i.imgur.com/ljRuDER.png)
    ![](https://i.imgur.com/pfaFwAo.png)
    <br>
    
    ==Minio 狀態：==
    ![](https://i.imgur.com/rveacBn.png)
    <br>
    
    ==Airflow 狀態：==
    [![](https://i.imgur.com/jwL4x1Z.png)](https://i.imgur.com/jwL4x1Z.png)
    不淒楚為何 Airflow 都沒有動靜
    
- ### 完整 workflow 的 DAG code
    ![](https://i.imgur.com/ENN2uwk.png)
    ```python=
    from airflow import DAG

    from airflow_notebook.pipeline import NotebookOp
    from airflow.utils.dates import days_ago

    # Setup default args with older date to automatically trigger when uploaded
    args = {
        "project_id": "tj-airflow-0504161135",
    }

    dag = DAG(
        "tj-airflow-0504161135",
        default_args=args,
        schedule_interval="@once",
        start_date=days_ago(1),
        description="Created with Elyra 2.2.4 pipeline editor using tj-airflow.pipeline.",
        is_paused_upon_creation=False,
    )


    notebook_op_2e2461b2_6916_410d_99e3_bc229e779441 = NotebookOp(
        name="load_data",
        namespace="default",
        task_id="load_data",
        notebook="elyra-ai-examples/pipelines/hello_world_apache_airflow/load_data.ipynb",
        cos_endpoint="http://10.78.26.241:9000",
        cos_bucket="minio-workspace",
        cos_directory="tj-airflow-0504161135",
        cos_dependencies_archive="load_data-2e2461b2-6916-410d-99e3-bc229e779441.tar.gz",
        pipeline_outputs=[],
        pipeline_inputs=[],
        image="amancevice/pandas:1.1.1",
        in_cluster=True,
        env_vars={
            "AWS_ACCESS_KEY_ID": "minioadmin",
            "AWS_SECRET_ACCESS_KEY": "minioadmin",
            "ELYRA_ENABLE_PIPELINE_INFO": "True",
        },
        config_file="None",
        dag=dag,
    )


    notebook_op_e7290f33_0616_4879_a00b_643fc5f0e407 = NotebookOp(
        name="Part_1___Data_Cleaning",
        namespace="default",
        task_id="Part_1___Data_Cleaning",
        notebook="elyra-ai-examples/pipelines/hello_world_apache_airflow/Part 1 - Data Cleaning.ipynb",
        cos_endpoint="http://10.78.26.241:9000",
        cos_bucket="minio-workspace",
        cos_directory="tj-airflow-0504161135",
        cos_dependencies_archive="Part 1 - Data Cleaning-e7290f33-0616-4879-a00b-643fc5f0e407.tar.gz",
        pipeline_outputs=[],
        pipeline_inputs=[],
        image="amancevice/pandas:1.1.1",
        in_cluster=True,
        env_vars={
            "AWS_ACCESS_KEY_ID": "minioadmin",
            "AWS_SECRET_ACCESS_KEY": "minioadmin",
            "ELYRA_ENABLE_PIPELINE_INFO": "True",
        },
        config_file="None",
        dag=dag,
    )

    (
        notebook_op_e7290f33_0616_4879_a00b_643fc5f0e407
        << notebook_op_2e2461b2_6916_410d_99e3_bc229e779441
    )


    notebook_op_167f5cee_4145_450f_9c0d_d43fd8607ed3 = NotebookOp(
        name="Part_2___Data_Analysis",
        namespace="default",
        task_id="Part_2___Data_Analysis",
        notebook="elyra-ai-examples/pipelines/hello_world_apache_airflow/Part 2 - Data Analysis.ipynb",
        cos_endpoint="http://10.78.26.241:9000",
        cos_bucket="minio-workspace",
        cos_directory="tj-airflow-0504161135",
        cos_dependencies_archive="Part 2 - Data Analysis-167f5cee-4145-450f-9c0d-d43fd8607ed3.tar.gz",
        pipeline_outputs=[],
        pipeline_inputs=[],
        image="amancevice/pandas:1.1.1",
        in_cluster=True,
        env_vars={
            "AWS_ACCESS_KEY_ID": "minioadmin",
            "AWS_SECRET_ACCESS_KEY": "minioadmin",
            "ELYRA_ENABLE_PIPELINE_INFO": "True",
        },
        config_file="None",
        dag=dag,
    )

    (
        notebook_op_167f5cee_4145_450f_9c0d_d43fd8607ed3
        << notebook_op_e7290f33_0616_4879_a00b_643fc5f0e407
    )


    notebook_op_d8c73222_f332_4134_b8a2_83cb1cd2a066 = NotebookOp(
        name="Part_3___Time_Series_Forecasting",
        namespace="default",
        task_id="Part_3___Time_Series_Forecasting",
        notebook="elyra-ai-examples/pipelines/hello_world_apache_airflow/Part 3 - Time Series Forecasting.ipynb",
        cos_endpoint="http://10.78.26.241:9000",
        cos_bucket="minio-workspace",
        cos_directory="tj-airflow-0504161135",
        cos_dependencies_archive="Part 3 - Time Series Forecasting-d8c73222-f332-4134-b8a2-83cb1cd2a066.tar.gz",
        pipeline_outputs=[],
        pipeline_inputs=[],
        image="amancevice/pandas:1.1.1",
        in_cluster=True,
        env_vars={
            "AWS_ACCESS_KEY_ID": "minioadmin",
            "AWS_SECRET_ACCESS_KEY": "minioadmin",
            "ELYRA_ENABLE_PIPELINE_INFO": "True",
        },
        config_file="None",
        dag=dag,
    )

    (
        notebook_op_d8c73222_f332_4134_b8a2_83cb1cd2a066
        << notebook_op_e7290f33_0616_4879_a00b_643fc5f0e407
    )
    ```
    [![](https://i.imgur.com/Wpnwod3.png)](https://i.imgur.com/Wpnwod3.png)


<br>

## 參考資料
- ### [Running notebook pipelines in JupyterLab](https://medium.com/codait/running-notebook-pipelines-locally-in-jupyterlab-1fae14b8e081)
    ![](https://i.imgur.com/zVLUqos.png)

- ### [Creating notebook pipelines using Elyra and Kubeflow Pipelines](https://medium.com/codait/creating-notebook-pipelines-using-elyra-and-kubeflow-pipelines-f9606449cc53)
    - S3 物件用途：儲存要執行的 code + 輸出入檔案
    - 執行過程：
        - 當一個 node 要執行時
            input: 下載要用的相關檔案(壓縮檔)
            output: 上傳回去
        繼續下一個 node (可並行處理)
    - 輸出入檔案範例
        ![](https://i.imgur.com/fmS1vFM.png)
        ![](https://i.imgur.com/8momSeZ.png)
        ![](https://i.imgur.com/osfEU3v.png)


<br>

<hr>

<br>

# Airflow
## Cynthia 的 Airflow 介紹
- [[slide] Airflow](https://docs.google.com/presentation/d/1Zh5H2YzMv6h3P3s6-UPw1ykw2Rb8biYTszdJljSLYZE/edit?usp=sharing)

<br>

## 安裝 Airflow
- ### 在 K8s 上安裝
    - [Airflow Helm Chart](https://github.com/airflow-helm/charts/tree/main/charts/airflow)
        - charts/airflow/values.yaml b/charts/airflow/values.yaml
            ```git=
            diff --git a/charts/airflow/values.yaml b/charts/airflow/values.yaml
            index ff401f3..7e47839 100644
            --- a/charts/airflow/values.yaml
            +++ b/charts/airflow/values.yaml
            @@ -80,9 +80,9 @@ airflow:
                 - username: admin
                   password: admin
                   role: Admin
            -      email: admin@example.com
            -      firstName: admin
            -      lastName: admin
            +      email: tj_tsai@asus.com
            +      firstName: tj
            +      lastName: tsai

               ## if we update users or just create them the first time (lookup by `username`)
               ##
            @@ -981,7 +981,7 @@ dags:
               gitSync:
                 ## if the git-sync sidecar container is enabled
                 ##
            -    enabled: false
            +    enabled: true

                 ## the git-sync container image
                 ##
            @@ -1011,7 +1011,7 @@ dags:
                 ## EXAMPLE - SSH:
                 ##    repo: "git@github.com:USERNAME/REPOSITORY.git"
                 ##
            -    repo: ""
            +    repo: "https://github.com/tsungjung411/any-private-test.git"

                 ## the sub-path (within your repo) where dags are located
                 ##
            @@ -1023,7 +1023,7 @@ dags:

                 ## the git branch to check out
                 ##
            -    branch: master
            +    branch: main

                 ## the git revision (tag or hash) to check out
                 ##
            @@ -1043,7 +1043,7 @@ dags:

                 ## the name of a pre-created Secret with git http credentials
                 ##
            -    httpSecret: ""
            +    httpSecret: "airflow-http-git-secret"
            ```
        - 按照上面步驟即可安裝完成
        - [[文件補充說明] Airflow Helm Chart](https://artifacthub.io/packages/helm/airflow-helm/airflow)
    - 安裝成功
        [![](https://i.imgur.com/570YiKa.png)](https://i.imgur.com/570YiKa.png)
        <br>
        bad case: 
        pod/tj-airflow-cluster-web 可能需要重啟數次（約數 10 分鐘到 20 分鐘），才能 work
        ![](https://i.imgur.com/LbZq26I.png)
        ![](https://i.imgur.com/uCG7xsq.png)

    - 連線測試
        ```bash
        $ kubectl get all -n tj-airflow
        $ kubectl port-forward -n $NAMESPACE --address 10.78.26.241 service/tj-airflow-cluster-web 9090:18080
        ```
- ### 在 K8s 上解除安裝
    ```bash
    $ kubectl get all -n $NAMESPACE
    $ helm uninstall $RELEASE_NAME -n $NAMESPACE
    $ kubectl get all -n $NAMESPACE
    ```
- ### 套件問題
    - `ModuleNotFoundError: No module named 'airflow_notebook'`
    - [airflow-notebook 0.0.7](https://pypi.org/project/airflow-notebook/)
        ~~`pip install airflow-notebook`~~
        - error
            ```
            ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

            We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

            keyring 23.0.1 requires importlib-metadata>=3.6, but you'll have importlib-metadata 1.7.0 which is incompatible.
            twine 3.4.1 requires importlib-metadata>=3.6, but you'll have importlib-metadata 1.7.0 which is incompatible.
            ```
        - **正確安裝指令 (2021/05/12 已驗證)**
            ```
            pip install airflow-notebook --use-feature=2020-resolver
            ```
            測試 import
            ```python=
            import airflow_notebook
            ```
        - 連線到 pod 裡安裝
            ```bash
            $ kubectl exec -it service/tj-airflow-cluster-web -n tj-airflow -- bash

            $ pip install airflow-notebook --use-feature=2020-resolver
            ```
            - 但還是遇到 `ModuleNotFoundError: No module named 'airflow_notebook'`
            - airflow webserver 沒有重啟?
            - [重啟需要 root 密碼](https://docs.qubole.com/en/latest/faqs/airflow/airflow-service-questions.html)
        - 最終可行方法，在程式碼前面加上
            ```python=
            def pip_install(python_lib):
                import importlib.util
                import sys
                import os

                module = importlib.util.find_spec(python_lib)
                if module == None:
                    python_path = sys.executable
                    cmd = python_path + ' -m pip install ' + python_lib + '  --use-feature=2020-resolver'
                    os.system(cmd)
                else:
                    print('module info:\n  ', module)
                    print('module location:\n  ', module.submodule_search_locations)
                    print('The package "%s" already exists.' % python_lib)


            pip_install('airflow_notebook')
            ```
        - 又遇到 id 衝突
            ![](https://i.imgur.com/eoaMxTj.png)
            相同檔名，不同副檔名，會產生相同 task ID
        - DAG 執行失敗
            [![](https://i.imgur.com/b1Zvnwj.png)](https://i.imgur.com/b1Zvnwj.png)
            ```
            *** Log file does not exist: /opt/airflow/logs/penguin-0512164146/step1_1_load_data/2021-05-11T00:00:00+00:00/1.log
            *** Fetching from: http://:8793/log/penguin-0512164146/step1_1_load_data/2021-05-11T00:00:00+00:00/1.log
            *** Failed to fetch log file from worker. Invalid URL 'http://:8793/log/penguin-0512164146/step1_1_load_data/2021-05-11T00:00:00+00:00/1.log': No host supplied
            ```
    - [apache-airflow-providers-cncf-kubernetes 1.2.0](https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/)
        `pip install apache-airflow-providers-cncf-kubernetes`
        - 尚未驗證
    - [Installing Airflow with extras and providers](https://airflow.apache.org/docs/apache-airflow/stable/installation.html#installation-script)
        ```bash
        AIRFLOW_VERSION=2.0.1
        PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
        CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
        pip install "apache-airflow[async,postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
        ```
        - [(2.0.1)-(3.8.7) 不存在](https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.8.7.txt)
        - [(2.0.1)-(3.8) 存在](https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.8.txt)
            ```
            CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-2.0.1/constraints-3.8.txt"
            pip install "apache-airflow[async,postgres,google]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
            ```
            - 仍然無法解決 (2021/05/12 已驗證)
                `ModuleNotFoundError: No module named 'airflow_notebook'`


<br>

## 使用說明
- ### [Apache Airflow — A New Way To Write DAGs](https://towardsdatascience.com/apache-airflow-a-new-way-to-write-dags-240a93c52e1a)
    ![](https://i.imgur.com/4qEZBt8.png)


<br>

<hr>

<br>

# Minio (S3-compatible)
## 安裝 Minio
- ### [在 Docker 上安裝](https://github.com/minio/minio#docker-installation)
    ```
    $ docker run -p 9000:9000 -v `pwd`:/data minio/minio server /data
    ...
    Detected default credentials 'minioadmin:minioadmin', please change the credentials immediately using 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD'
    IAM initialization complete
    ```
    - IAM：Identity and Access Management 身分與存取管理
    - 預設根憑證 `minioadmin:minioadmin`
    - Apache Airflow runtimes 的設定
        [![](https://i.imgur.com/VCqaHP9.png)](https://i.imgur.com/VCqaHP9.png)

- ### [在 K8s 上安裝](https://docs.min.io/docs/deploy-minio-on-kubernetes.html)
    > MinIO is a high performance distributed object storage server, designed for large-scale private cloud infrastructure. 

<br>

## [Pricing](https://min.io/pricing)
![](https://i.imgur.com/eaLd3XR.png)

<br>

<hr>

<br>

# AWS S3
## IAM
> 建立使用者身份與對應權限

- ### 使用者帳號
    [![](https://i.imgur.com/JnYbDcm.png)](https://i.imgur.com/JnYbDcm.png)

- ### 使用者群組
    [![](https://i.imgur.com/tPPEqDq.png)](https://i.imgur.com/tPPEqDq.png)
    [![](https://i.imgur.com/PHrDYIQ.png)](https://i.imgur.com/PHrDYIQ.png)
    
- ### 其他設定
    略過
    
- ### 建立完成
    [![](https://i.imgur.com/nZqTr4e.png)](https://i.imgur.com/nZqTr4e.png)
    | 中文 | Key | Value |
    | -------- | -------- | -------- |
    | 使用者 | User name | `elyra_s3_user` |
    | 密碼 | Password |  |
    | 存取金鑰 ID | Access key ID | `AKIAYMLWO37GH6RI3PXE` |
    | 私密存取金鑰 | Secret access key | `FKb1xcVcUF7xypwcQDPO+m06ERbqZRbbtRNGPCIR` |

- ### 設定密碼
    ![](https://i.imgur.com/6wMDF8Y.png)

<br>

## 參考資料
- ### [How To Get Amazon S3 Access Keys](https://objectivefs.com/howto/how-to-get-amazon-s3-keys)
- ### [How to Allow Public Access to an Amazon S3 Bucket & Find S3 URLs](https://havecamerawilltravel.com/photographer/how-allow-public-access-amazon-bucket/)

<br>

<hr>

<br>

# Airflow-on-K8s
## build image (包含 airflow-notebook 套件): OK (2021/05/20)
```dockerfile=
# docker build -t apache/airflow:2.0.1-python3.8-by-tj .

FROM apache/airflow:2.0.1-python3.8

USER root
RUN python -m pip install airflow_notebook --use-feature=2020-resolver

USER root airflow
```

## debug
### `no persistent volumes available for this claim and no storage class is set`
- 暫時解法
    將 pstgre 的 persisstent 設成 false
- [Error “no persistent volumes available for this claim and no storage class is set”](https://stackoverflow.com/questions/55780083)

<br>

### `kubernetes.client.rest.ApiException: (403)`
[![](https://i.imgur.com/FkC6o2o.png)](https://i.imgur.com/FkC6o2o.png)
```
*** Log file does not exist: /opt/airflow/logs/penguin-0520073052/step1_load_data/2021-05-19T00:00:00+00:00/1.log
*** Fetching from: http://tj-airflow-cluster-worker-0.tj-airflow-cluster-worker.tj-airflow.svc.cluster.local:8793/log/penguin-0520073052/step1_load_data/2021-05-19T00:00:00+00:00/1.log

[2021-05-20 07:32:00,046] {taskinstance.py:851} INFO - Dependencies all met for <TaskInstance: penguin-0520073052.step1_load_data 2021-05-19T00:00:00+00:00 [queued]>
[2021-05-20 07:32:00,161] {taskinstance.py:851} INFO - Dependencies all met for <TaskInstance: penguin-0520073052.step1_load_data 2021-05-19T00:00:00+00:00 [queued]>
[2021-05-20 07:32:00,161] {taskinstance.py:1042} INFO - 
--------------------------------------------------------------------------------
[2021-05-20 07:32:00,161] {taskinstance.py:1043} INFO - Starting attempt 1 of 1
[2021-05-20 07:32:00,162] {taskinstance.py:1044} INFO - 
--------------------------------------------------------------------------------
[2021-05-20 07:32:00,174] {taskinstance.py:1063} INFO - Executing <Task(NotebookOp): step1_load_data> on 2021-05-19T00:00:00+00:00
[2021-05-20 07:32:00,246] {standard_task_runner.py:52} INFO - Started process 258 to run task
[2021-05-20 07:32:00,252] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'penguin-0520073052', 'step1_load_data', '2021-05-19T00:00:00+00:00', '--job-id', '31', '--pool', 'default_pool', '--raw', '--subdir', '/opt/airflow/dags/repo/penguin-0520073052.py', '--cfg-path', '/tmp/tmpnhagtq7b', '--error-file', '/tmp/tmps1bwwtkq']
[2021-05-20 07:32:00,252] {standard_task_runner.py:77} INFO - Job 31: Subtask step1_load_data
[2021-05-20 07:32:00,645] {logging_mixin.py:104} INFO - Running <TaskInstance: penguin-0520073052.step1_load_data 2021-05-19T00:00:00+00:00 [running]> on host tj-airflow-cluster-worker-0.tj-airflow-cluster-worker.tj-airflow.svc.cluster.local
[2021-05-20 07:32:00,755] {taskinstance.py:1255} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=penguin-0520073052
AIRFLOW_CTX_TASK_ID=step1_load_data
AIRFLOW_CTX_EXECUTION_DATE=2021-05-19T00:00:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2021-05-19T00:00:00+00:00
[2021-05-20 07:32:00,848] {taskinstance.py:1455} ERROR - (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '79d187ea-8c60-456e-a34a-f0fc94abe222', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e4a0b3a5-23f5-4643-a914-1897672eeb07', 'Date': 'Thu, 20 May 2021 07:32:00 GMT', 'Content-Length': '296'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:tj-airflow:tj-airflow-cluster\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1112, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1315, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 323, in execute
    pod_list = client.list_namespaced_pod(self.namespace, label_selector=label_selector)
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 12803, in list_namespaced_pod
    (data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs)  # noqa: E501
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 12891, in list_namespaced_pod_with_http_info
    return self.api_client.call_api(
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 340, in call_api
    return self.__call_api(resource_path, method,
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 172, in __call_api
    response_data = self.request(
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 362, in request
    return self.rest_client.GET(url,
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 237, in GET
    return self.request("GET", url,
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '79d187ea-8c60-456e-a34a-f0fc94abe222', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e4a0b3a5-23f5-4643-a914-1897672eeb07', 'Date': 'Thu, 20 May 2021 07:32:00 GMT', 'Content-Length': '296'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:tj-airflow:tj-airflow-cluster\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}


[2021-05-20 07:32:00,862] {taskinstance.py:1496} INFO - Marking task as FAILED. dag_id=penguin-0520073052, task_id=step1_load_data, execution_date=20210519T000000, start_date=20210520T073200, end_date=20210520T073200
[2021-05-20 07:32:00,984] {local_task_job.py:146} INFO - Task exited with return code 1

```
- 檢查 values.yaml
    ![](https://i.imgur.com/uTsl6FK.png)
    ```yaml=1204

    ###################################
    # Kubernetes - RBAC
    ###################################
    rbac:
      ## if Kubernetes RBAC resources are created
      ##
      ## NOTE:
      ## - these allow the service account to create/delete Pods in the airflow namespace,
      ##   which is required for the KubernetesPodOperator() to function
      ##
      create: true

      ## if the created RBAC Role has GET/LIST on Event resources
      ##
      ## NOTE:
      ## - this is needed for KubernetesPodOperator() to use `log_events_on_failure=True`
      ##
      events: true

    ###################################
    # Kubernetes - Service Account
    ###################################
    serviceAccount:
      ## if a Kubernetes ServiceAccount is created
      ##
      ## NOTE:
      ## - if false, you must create the service account outside of this chart,
      ##   with the name: `serviceAccount.name`
      ##
      create: true

      ## the name of the ServiceAccount
      ##
      ## NOTE:
      ## - by default the name is generated using the `airflow.serviceAccountName` template in `_helpers/common.tpl`
      ##
      name: ""

      ## annotations for the ServiceAccount
      ##
      ## EXAMPLE: (to use WorkloadIdentity in Google Cloud)
      ##   annotations:
      ##     iam.gke.io/gcp-service-account: <<GCP_SERVICE>>@<<GCP_PROJECT>>.iam.gserviceaccount.com
      ##
      annotations: {}
    ```
- 可能解法
    - [Scheduler throw API exception (403) #473](https://github.com/puckel/docker-airflow/issues/473)
    - [403 forbidden when i ran in_cluster_config.py in the pod #519](https://github.com/kubernetes-client/python/issues/519)
        ![](https://i.imgur.com/hLyA8du.png)
        - here is my blog for solving this problem: https://blog.csdn.net/u013431916/article/details/80057369
            - [基於RBAC方式從Pod內訪問API server](https://blog.csdn.net/u013431916/article/details/80057369)
    - [[kubernetes-client / python]in_cluster_config.py](https://github.com/kubernetes-client/python/blob/master/examples/in_cluster_config.py)