# 🦅 DALiuGE & EAGLE Setup Tutorial

This documentation gives a brief overview of how to use DALiuGE as SDP pipeline orchestrator.

## Biblio
:page_facing_up: [DALiuGE](https://arxiv.org/pdf/1702.07617): C. WU, R. Tobar, K. Vinsen, A. Wicenec, D. Pallot, B. Lao, R. Wang, T. An, M. Boulton, I. Cooper, R. Dodson, M. Dolensky, Y. Mei, F. Wang. « *DALiuGE: A Graph Execution Framework for Harnessing the Astronomical Data Deluge* », Astronomy and Computing, 2017.

:speech_balloon: [ADASS 2024](https://pretalx.com/adass2024/talk/LTCUXF/): A Reproducible Science Workflow System: DALiuGE in Action, pretalx.

:blue_book: [DALiuGE Docs](https://daliuge.readthedocs.io/en/latest/index.html)
:blue_book: [EAGLE GUI Docs](https://eagle-dlg.readthedocs.io/en/latest/index.html)

⚠️ The official documentation is a bit messy, so here's a custom step-by-step guide to get things working locally.

---
## I - Introduction to DALiuGE

**Workflow summary:**

- Design your workflow using EAGLE (similar to building an SDF graph).
- Export the graph as a .graph file from EAGLE.
- Translate the graph using the DALiuGE Translator (Docker container).
- Execute the translated graph using the DALiuGE Engine (Docker container).

graph LR
  EAGLE -->|logical graph| TRANSLATOR
  TRANSLATOR -->|physical graph| NODEMANAGER

#### Step 1 – EAGLE: graph editor interface

EAGLE is the graphical editor (web GUI) for building DALiuGE graphs.

Theoretically, you can use EAGLE directly via their hosted service:

👉 [https://eagle.icrar.org/?service=GitHub\&repository=ICRAR/EAGLE-graph-repo\&branch=master\&path=examples\&filename=HelloWorld-simple.graph](https://eagle.icrar.org/?service=GitHub&repository=ICRAR/EAGLE-graph-repo&branch=master&path=examples&filename=HelloWorld-simple.graph)

…but it’s currently broken. So let’s install it locally instead.

###### Step 1.1 – Install EAGLE GUI Locally

git clone https://github.com/ICRAR/EAGLE
sudo apt install npm
sudo npm install -g typescript
cd EAGLE
npm install
tsc

###### Step 1.2 – Install Python Environment (Python 3.6 required)

sudo apt update
sudo apt install -y build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev \
libffi-dev liblzma-dev git

curl https://pyenv.run | bash

export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init --path)"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

source ~/.bashrc

pyenv virtualenv -p python3.6 eagle
pyenv activate eagle


pip install .

###### Step 1.3 – Launch the GUI

Start a local server: `eagleServer -t /tmp` and open the interface in your browser: http://127.0.0.1:8888/

###### Step 1.4 – Draw a graph

[🎥 Watch Hello World Video](https://eagle-dlg.readthedocs.io/en/latest/_static/videos/helloWorld_creation.mp4)

1. drag and drop the hellowordapp and the file component from building component
2. connect them
3. save the graph
- locally: `graph` > `local storage`
- on line: [todo]

At this stage we are able to generate a dataflow.graph wow...

#### Step 2 - DALiuGE translator

(je sais pas comment ça fonctione en ligne)
install DALiuGE:

git clone https://github.com/ICRAR/daliuge.git
cd daliuge
make docker-install

# or manually:
cd daliuge-common && ./build_common.sh dev && cd ..
cd daliuge-engine&& ./build_engine.sh dev && cd ..
cd daliuge-translator && ./build_translator.sh dev && cd ..

#run image
make docker-run

# in case docker fail
cd daliuge-translator
./run_translator

# remove existing docker
docker rm -f daliuge-engine daliuge-translator

1. from EAGLE: translation > set url: http://localhost:8084 (or http://localhost:8084/gen_pgt)
2. run the translation: `translate`
It will open the translater interface in your browser.


#### Step 3 DALiuGE Engine

i don't know, i failed :poop:

---
## II Building and atempting to run an SDP imaging pipeline

1. Dowload [ska-sdp-daliuge-graphs](https://gitlab.com/ska-telescope/ska-sdp-daliuge-graphs)
2. Run eagle: eagleServer -t /tmp
3. browse: http://127.0.0.1:8888/
4. Load the available graph: `graph` > `local storage` > `load` : `SDP Pipelines/cont_img_YAN-960.graph`
_{According to ADASS 2024, this represents 48 billion tasks. However, the three inner loops are usually grouped into one coarse task, reducing the count to 78 million. :warning: Limited to 12 million tasks.}
5. fix all error [F]
6. run DALiuGE: `cd DaLiuGE` > `make docker-run`
7. translate as physical graph from **EAGLE**: `translate`
_{Sankey representation of the SDP Pipeline flow. The translation make parallelism explicit. Thickness is proportional to the volume of data or tasks in circulation.}
8. Add deployement option from **Translator**(http://localhost:8084/gen_pgt):
- url: `http://dlg-engine.local:8000/`
- server: `OOD` (or else I don't see the difference)
9. Deploy:
- via interface web:
- from **Translator**`Deploy`: `OOD`
- `Export` > `Physical graph template` this create `cont_img_YAN-96014_pgt.graph` locally saved.
- browse: http://dlg-engine.local:8000/ this open DALiuGE NodeMananger.
- from **NodeMananger**: `add session` > load `cont_img_YAN-96014_pgt.graph` (this fail every time).
- via API REST:

    curl -X POST http://localhost:8000/api/sessions \
      -H "Content-Type: application/json" \
      -d @cont_img_YAN-96014_pgt.graph

- via command line:

    # install
    cd daliuge-engin/ 
    pip install pyzmq==25.1.1
    pip install . 
    # command
    dlg graph submit --host localhost --port 9000 cont_img_YAN-96014_pgt.graph

:warning: problème de suivi doc vs. daliuge, on peut pas deployer leur truc à ce stade

---
## III PREESM vs DALiuGE

<small>

| Feature / Criterion | **PREESM** | **DALiuGE** |
|-----------------------------------------|--------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
| **Full Name** | Parallel and Real-time Embedded Executives Scheduling Method | Data Activated 流 (flow) Graph Engine |
| **Origin / Domain** | Embedded systems, real-time DSP | Radio astronomy, large-scale distributed computing |
| **Programming Model** | Synchronous Dataflow (SDF), extended with Parametric & Hierarchical SDF | Logical graphs with data-driven execution (DAG-based) |
| **Graph Types Supported** | SDF graphs (`.pi` format), hierarchical and parameterized | Logical graphs (`.graph`), translated to physical execution graphs (`.json`) |
| **Execution Model** | Static scheduling on multicore, **now extended to distributed & GPU architectures** | Dynamic, data-driven execution on distributed infrastructures |
| **Mapping & Scheduling** | Static list mapping + scheduling, with optimization heuristics (also for clusters via SimSDP) | METIS scheduling; data availability triggers execution |
| **Tooling** | Eclipse-based GUI, Java backend, code generation in C or OpenCL | Python-based GUI (EAGLE), Docker-based translation and execution |
| **Target Architectures** | Multicore CPUs, FPGAs, **distributed systems (via SimSDP)**, and **GPUs (CUDA codegen)** | HPC & distributed systems (e.g., SKA pipelines, cloud, clusters) |
| **Simulation Support** | Yes (detailed simulation of time, memory, communication with SimSDP) | Yes (via execution engine, but less detailed) |
| **Granularity Control** | Fine-grain (actor-level), adjustable with transformations or SCAPE methods | Generally coarse-grain (pipeline/task-level), though nested graphs can increase control |
| **Ease of Use** | Medium to advanced (requires SDF concepts and Eclipse familiarity) | Medium (simple GUI but requires Docker and JSON graph knowledge for full execution) |
| **Extensibility** | High (custom actors, code generation backends, schedulers) | Medium (custom tasks in Python or Dockerized components) |
| **Open Source** | Yes ([GitHub](https://github.com/preesm/preesm)) | Yes ([GitHub](https://github.com/ICRAR/daliuge)) |
| **Documentation** | Moderate to good; some older parts but actively maintained and extended | Available but fragmented; EAGLE lacks detailed examples |
| **Main Use Case** | Embedded/HPC hybrid prototyping; performance-aware design of dataflow applications | Workflow orchestration for distributed, data-intensive pipelines (e.g., SKA, astronomy) |
| **Supported Platforms** | Linux, partial Windows support; Eclipse plugin; Singularity Aptainer for SimSDP prototypes | Linux/macOS (Docker required for execution), browser-based GUI |
| **Developer Community** | Academic (INSA Rennes, IETR, collaborations with HPC research teams) | Academic + SKA-related institutions (ICRAR, CSIRO, etc.) |

</small>

---
## IV Conclusion

The **[DARK-ERA](https://dark-era.pages.centralesupelec.fr/)** goal (and mine) is to show the reliability of SimSDP in simulating the behavior of applications deployed on HPC systems, then once validated on a complete pipeline (imaging pipeline with major loops such as DFT, FFT or G2G) we can extrapolate our simulations to the SKA scale and quantify the gap in pipeline acceleration to achieve real time (processing data as fast as it arrives).
In **PREESM/SimSDP**, task times are set in the `scenario` and can be either *integers* or *functions*, facilitating exploration of the simulation according to key parameters such as *visibility*, *grid size* or *deconvolution number* , and so on. `Moldable parameters` are also implemented in the tool, enabling automatic browsing of the configuration set by assigning a set of values to a parameter.
**DALiuGE**, on the other hand, is more of a dynamic, data-driven execution engine: it orchestrates the pipeline as it goes along, without any prior detailed simulation or fine-tuned estimation of execution times. However, as with PREESM/SimSDP, task times can be set from EAGLE (select a `task` > `Node Field table` [T] > `Execution time` (only *intergers* are accepted)), then in translator the execution time is estimated based on the METIS Scheduler. A comparison of simulations is therefore possible.

---
## Contact

For questions or feedback, please contact:

- This tutorial was written by [Ophélie Renaud](mailto:[e-mail]) from PREESM/SimSDP community (previously @INSA, @IETR, now @SATIE paris-saclay, @IRISA, @ENS Rennes :fr:).
- The official DALiuGE repository can be found on [GitHub - ICRAR/daliuge](https://github.com/ICRAR/daliuge), if a pull request is required.