[Back to LOC CLI Guidebook](https://hackmd.io/4LIsNIuWQnuR70JMeuBAEA?view)
[Setup LOC CLI Environment](https://hackmd.io/igLh4azUT2aI8Fv-q-0e-g)
[Getting Started - JavaScript](https://hackmd.io/IiHvmAtjTTGFfakaH0dWuw?view)
[Getting Started - TypeScript](https://hackmd.io/kz93Th7vTCCbO3GFxp3r-A)
[LOC CLI Commands](https://hackmd.io/R4mrz2t1QSyTCHH73_2itg)
# Local Simple Runtime Execution
If you want to quickly test or debug your data process, there is a way to deploy and run it in a local LOC environment. The benefits of doing so are:
- No user login required
- No API routes nor HTTP clients needed
- Does not use LOC deployment environment resources
- See ```ctx.agents.logging``` logs in your console
Most of the [LOC agents](https://hackmd.io/Uj-tC7l9Q82VyGr8R5PL2Q?both) are supported *except* using SMB in file storage agent.
## Setup
### Install Docker and Docker Compose
The local runtime requires **Docker** and **Docker Compose**. You can simply install [**Docker Desktop**](https://docs.docker.com/desktop/) on your machine, which contains both and supports Windows/Mac/Linux.
- [Install on Windows](https://docs.docker.com/desktop/windows/install/)
- [Install on Mac](https://docs.docker.com/desktop/mac/install/)
- Install on Linux
- [Download Docker Desktop for Linux packages](https://docs.docker.com/desktop/linux/install/)
- [Set up the Docker repository](https://docs.docker.com/engine/install/ubuntu/#set-up-the-repository)
- [Install on Debian/Fedora/Ubuntu/Arch](https://docs.docker.com/desktop/linux/install/#generic-installation-steps)
:::info
### What is Docker Compose?
Docker Compose is a way to start up multiple Docker containers *at once*. The concept and effect are very similar to Kubernetes, which is the deployment environment that LOC runs on.
:::
### Create Local Profile
Create a new profile ```local.yaml``` under your LOC CLI workspace directory (for example, ```/FST```):
```yaml
simpleRuntime:
image: public.ecr.aws/m0s8j1u6/saffron/simple-runtime:release-0.5.1
dockerNetwork: host
eventstoreEndpoint: http://127.0.0.1:8087
etcdEndpoints: http://127.0.0.1:2379
```
Set it as your default profile:
```bash
loc profile set -f local.yaml -p local
loc profile default local
```
### Create ```saffron-evenstore.yaml```
Create this file also in your LOC CLI workspace directory, which defines the containers we want to start up together:
```yaml
version: "3.7"
networks:
saffron:
driver: bridge
volumes:
es01:
driver: local
services:
saffron-evenstore:
restart: "always"
image: public.ecr.aws/m0s8j1u6/saffron/eventstore:release-0.5.1
container_name: saffron-evenstore
ports:
- "8200:8200"
- "8087:8087"
- "8007:8007"
environment:
- SAFFRON_EVENTSTORE_API_ASSRESS=0.0.0.0
- SAFFRON_EVENTSTORE_API_PORT=8007
- SAFFRON_EVENTSTORE_API_AUTHORIZATION_SECRET=-----BEGIN PUBLIC KEY-----MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmuzhqHVUKj0CVtnqCS1PbVOyq1TcyykKgXoBwQvPaF8N8zIHXUjCgwjmKbOK+SjLoLC7cgLWlwbV2u12J/9w2sTawbROOno7Td2OZte2qNLugsEf5RxGVKd4IZx5zshxTI/bwnYGvpyVWdbRGvtWMKpbkzkR7O8UumtUMdUl6h1iUKRXkAxoB2ge/CZUDCss643iS0KPO7gIowvcYFCKBX86G/+GBlIHcXWTIa2eEWuTx5T64lmSVJrlN1L6diZRlUIBq6JNuQv26ZJgNAO91DDcCLCxcQgLH5KqwzdXwz2jBOqL7ac1HAW/mov4oJrk7ytWVumoFJ+NlJlSZ+vd5wIDAQAB-----END PUBLIC KEY-----
- SAFFRON_EVENTSTORE_GRPC_ADDRESS=0.0.0.0
- SAFFRON_EVENTSTORE_GRPC_PORT=8087
- SAFFRON_EVENTSTORE_METRICS_ADDRESS=0.0.0.0
- SAFFRON_EVENTSTORE_METRICS_PORT=8200
- SAFFRON_EVENTSTORE_ELASTICSEARCH_ENDPOINT=http://es01:9200
- SAFFRON_EVENTSTORE_ELASTICSEARCH_USERNAME=
- SAFFRON_EVENTSTORE_ELASTICSEARCH_PASSWORD=
command:
- run
networks:
- saffron
depends_on:
- etcd01
- es01
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.16.1
restart: "always"
container_name: es01
environment:
- xpack.security.enabled=false
- bootstrap.memory_lock=true
- discovery.type=single-node
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- es01:/usr/share/elasticsearch/data
ports:
- "9200:9200"
networks:
- saffron
kibana:
image: kibana:7.16.1
restart: "always"
container_name: kibana
volumes:
- ./kibana/kibana.yaml:/usr/share/kibana/config/kibana.yml
ports:
- "5601:5601"
networks:
- saffron
etcd01:
image: docker.io/bitnami/etcd:3.5.1-debian-10-r31
restart: "always"
container_name: etcd01
environment:
- ALLOW_NONE_AUTHENTICATION=yes
- ETCD_NAME=etcd01
- ETCD_INITIAL_ADVERTISE_PEER_URLS=http://etcd01:2380
- ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
- ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
- ETCD_ADVERTISE_CLIENT_URLS=http://etcd01:2379
- ETCD_INITIAL_CLUSTER_TOKEN=saffron-etcd-cluster
- ETCD_INITIAL_CLUSTER_STATE=new
ports:
- "2379:2379"
networks:
- saffron
```
### Create ```/kibana/kibana.yaml```
Add a new dir ```kibana``` under CLI dir and create ```kibana.yaml``` in it:
```yaml
server.name: "kibana"
server.host: "0.0.0.0"
elasticsearch.hosts: [http://es01:9200]
xpack.security.enabled: true
elasticsearch.username: "elastic"
elasticsearch.password: "aa123456"
monitoring.enabled: false
```
:::info
The new files would be structured like this (we will add ```payload.yaml``` later):
```
/FST
/[your data process project]
/kibana
kibana.yaml
loc
local.yaml
payload.yaml
saffron-evenstore.yaml
...
```
:::
## Run Docker Containers
### Simple Runtime Image
Now go back to LOC CLI workspace dir and run the following command to download the "simple runtime" image:
```bash
docker pull public.ecr.aws/m0s8j1u6/saffron/simple-runtime:release-0.5.1
```
Whenever you (re)deploy new data process, a new container would be created from this image.
### Startup Local LOC Services
We also need to create a local runtime so that everything works, including a functional local event store.
Open a new terminal or console, switch to LOC CLI dir and enter the following command:
```bash
docker-compose -f saffron-eventstore.yaml up
```
This command will download several images and start them up in your Docker environment:
- saffron-evenstore
- etcd01
- es01
- kibana
:::info
Please be noted that the first time running this process will take a while, depending on your Internet speed.
When the containers start up, you may see ```saffron-evenstore exited with code 255``` appear a few times. Don't worry - this is simply because the local event store is waiting for other services.
:::
You can check if all of them are up and running in Docker Desktop:
Or you can use the console command ```docker ps -a```:
```
> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
56d44778ce08 public.ecr.aws/m0s8j1u6/saffron/eventstore:release-0.5.1 "/usr/bin/saffron-ev…" 2 hours ago Up 2 hours 0.0.0.0:8007->8007/tcp, 0.0.0.0:8087->8087/tcp, 0.0.0.0:8200->8200/tcp saffron-evenstore
cd621ca37438 kibana:7.16.1 "/bin/tini -- /usr/l…" 2 hours ago Up 2 hours 0.0.0.0:5601->5601/tcp kibana
afa1658206b1 bitnami/etcd:3.5.1-debian-10-r31 "/opt/bitnami/script…" 2 hours ago Up 2 hours 0.0.0.0:2379->2379/tcp, 2380/tcp etcd01
6568da78bdc9 docker.elastic.co/elasticsearch/elasticsearch:7.16.1 "/bin/tini -- /usr/l…" 2 hours ago Up 2 hours 0.0.0.0:9200->9200/tcp, 9300/tcp es01
```
You can press ```Ctrl+C``` to gracefully shut them down and run ```docker-compose -f saffron-evenstore.yaml up``` again to start them up.
:::info
You can start up all containers in the background with ```-d``` flag:
```bash
docker-compose -f saffron-evenstore.yaml up -d
```
Remember you'll have to wait a while for saffron-evenstore to be ready.
Then you can stop/delete all local services like this:
```bash
docker-compose -f saffron-evenstore.yaml down
```
:::
## Run Data Process in Simple Runtime
### Create Payload File
Since we are going to run a data process directly without an API route, we need a .yaml file to pass "payloads" to it.
Create a ```payload.yaml``` in the LOC CLI workspace dir:
```yaml=
tasks:
- dataProcessIdentityContext:
permanentIdentity: 00000000-0000-0000-0000-000000000000
revision: 1
name: ""
taskId:
executionId: AAAAAAAAAAAAAAAAAAAAAA
id: AAAAAAAAAAAAAAAAAAAAAA
payload:
http:
apiGatewayIdentityContext:
id: 00000000-0000-0000-0000-000000000000
name: ""
apiIdentityContext:
id: 00000000-0000-0000-0000-000000000000
name: ""
requestId: "4242"
scheme: https
method: POST
path: /test/test
query: "?name=Arthur&age=42"
version: HTTP/1.1
headers:
Content-Type: "application/json"
host: fst.network
body:
name: "Arthur"
age: 42
```
:::info
DO NOT change the ```executionId``` or ```id``` under ```taskId```.
:::
You can see there are quite a few fields that you can customize. Most of the HTTP-related fields will directly pass to [```ctx.payload.http```](https://hackmd.io/Uj-tC7l9Q82VyGr8R5PL2Q?view#ctxpayloadhttp) as mock-up values.
```body``` represents the POST payload body. For example, if you write it like this
```yaml
body:
name: "Arthur"
age: 42
job:
- company: "FST Network"
- position: "Frontend Developer"
```
It would be transformed to a JSON like this in the data process:
```json
{
"name": "Arthur",
"age": 42,
"job": {
"company": "FST Network",
"position": "Frontend Developer"
}
}
```
### Execute Data Process in Single Runtime
When the local LOC services are up and running, we can run any data processes in a simple runtime.
Here we use the example from [Getting Started - Hello World (JavaScript)](https://hackmd.io/IiHvmAtjTTGFfakaH0dWuw?view) (open a new terminal to enter this command):
```bash
loc dp run-local hello-world -f payload.yaml --docker
```
:::info
For executing **TypeScript** version data processes, add a ```-ts``` flag:
```bash
loc dp run-local hello-world-ts -f payload.yaml -ts --docker
```
:::
If the data process compiled successfully (if not, it will try to tell you where it went wrong), you should see something like this:
Look at the outputs carefully, you can find the logs that you've sent in the data process:
```
2022-06-07T06:43:08.487874Z INFO saffron_process::agent::logger: [execution_id=AAAAAAAAAAAAAAAAAAAAAA,id=AAAAAAAAAAAAAAAAAAAAAA]@7c8c081e-0805-49cb-9ddd-6049b1a00562-0 {"payload":{"name":"Arthur","age":42}}
...
2022-06-07T06:43:08.544840Z INFO saffron_process::agent::logger: [execution_id=AAAAAAAAAAAAAAAAAAAAAA,id=AAAAAAAAAAAAAAAAAAAAAA]@7c8c081e-0805-49cb-9ddd-6049b1a00562-0 emit event: [{"sourceDID":"Hello_World","targetDID":"Arthur","labelName":"Hello, Arthur!","meta":"","type":"default"}]
...
2022-06-07T06:43:08.550495Z INFO saffron_process::agent::logger: [execution_id=AAAAAAAAAAAAAAAAAAAAAA,id=AAAAAAAAAAAAAAAAAAAAAA]@7c8c081e-0805-49cb-9ddd-6049b1a00562-0 {"response":{"message":"Hello, Arthur!"}}
```
### Cleanup Containers
Like we've mentioned above, every time you run a data process in simple runtime, a new container would be created. You can easily clean them up in Docker Desktop (clicking the trashcan icon on the right):
Or you can remove all inactive simple runtime containers with the following console command:
```bash
docker rm $(docker ps -a -q --filter ancestor=public.ecr.aws/m0s8j1u6/saffron/simple-runtime:release-0.5.1)
```
### Access ```localhost```
Any ```localhost``` services you have on your machine can be accessed with the host name **```host.docker.internal```**. For example, you can use the HTTP agent like this:
```javascript=
const response = await ctx.agents.http.post(
"http://host.docker.internal:8080/post", // = localhost:8080/post
{}, // headers
"Json",
new TextEncoder().encode(JSON.stringify(body)), // JSON body
);
```
---
:::info
### Simple Runtime Command Cheatsheet
```bash
# startup local LOC services
docker-compose -f saffron-evenstore.yaml up
# startup data process in local runtime
loc dp run-local <project name> -f payload.yaml --docker
# cleanup local runtime containers
docker rm $(docker ps -a -q --filter ancestor=public.ecr.aws/m0s8j1u6/saffron/simple-runtime:release-0.5.1)
```
### Additional Local Testing
- [View Local Event Logs in Kibana](https://hackmd.io/oE8gHd3hSLyl2XiAn66PGg?view)
- [Local Database Testing with Simple Runtime](https://hackmd.io/l2p0PF5yTC2UUnRtXnp5Qw?view)
- [Local FTP Testing with Simple Runtime](https://hackmd.io/QymXtDwOTcGNaym1YjRbkQ?view)
### Related Reading
- [Agent List](https://hackmd.io/Uj-tC7l9Q82VyGr8R5PL2Q?view#Agent-List)
- [Event Fields Comparison Table](https://hackmd.io/N2Km6-8kRI2tMR5X4tRHeQ?view)
- [Data Process Error Handling Tips](https://hackmd.io/Dfwa4fZkSdSAVzuq1Def2w?both)
- [HTTP Agent Request Examples](https://hackmd.io/xWdMYLD5RXiV8wt5cGVwJA)
:::
###### tags: `LOC CLI`
Sign in with Wallet
Connect another wallet