# Specific Objectives

dora-rs objectives is to enable real-time self-programming robots. LLM gives us the possibility to modify code given a prompt. By using this feature, instead of having code implemented ahead-of-time, we can generate code just-in-time and execute it. This makes it possible to create a new infinite of ways of interacting with robots.

Current self-programming features are limited to simple task, in the likes of changing robot states. We can do the following request and get valid response:

- Can you change the robot LED color?

- Can you change the robot destination goal objectives? ( Given that it will move to the goal objectives )

- Can you set rotate the robot according to visual bounding box? ( Given an image with bounding boxes of objects )

But We hope that by combining multiple LLM in LLM-agents fashion, we'll be able to handle more difficult task:

- sequence of tasks

- breakdown of a complex task into simpler one

- Memorise things in a way that it is reusable for task.

The end goal is to be able to cooperate with the robot intelligent and not limited to the existing code.

# Application Domain

The main application domain is embodied AI, robotics, LLM and VLM.

# Main functionalities

Self-programming robots given a task in natural languages.

This required a new AI framework that makes it easy to play with AI in realtime. dora-rs implements:
- Real-time code generation
- State of the art inter-process-communication performance
- zero-cost Apache Arrow abstraction
- Battery-included tools from logs, tracing and metrics using Opentelemetry
- API in Rust/Python/C/C++
- Bridges for ROS2
- Modular ecosystem of nodes from Speech-to-Text, Computer Vision, planning, ...

# Used AI Techniques

- Speech to text transformation using Whsiper.
- Image object detection using Yolov8
- Text to code using an LLM ( either Mistral or ChatGPT ).
- Hot-reloading feature using dora-rs to execute the code in real-time by the robot.

# Demonstration Scenario

Voice-controlled robot that is going to program itself on the task at hand using the technologies above.

# Visual of the Prototype

- Link to the video:
- Short Demo Version: https://www.youtube.com/watch?v=NvvTEP8Jak8
- Long Talk Version: https://video.fosdem.org/2024/ud2120/fosdem-2024-3225-dora-rs-simplifying-robotics-stack-for-next-gen-robots.mp4

----

# User Review

Super easy to use

Super clear

Understanding the difference between the nodes and operators. We have explain and which situation to use which. Documentation.

Projects:
- Python
- ROS2

Mainly use
- Python
- Rust
- And C
- Typescript

///