dora-drives - HackMD

<style> .reveal pre code { max-height: 1000px; font-size: 0.5em; } #left { left:-8.33%; text-align: left; float: left; width:50%; z-index:-10; } #right { top: 75px; float: right; z-index:-10; width:50%; } </style>  #### Dora-drives: Autonomous-driving made simple! <div class="avatar margin-bottom--sm"><div class="avatar__intro" itemprop="author" itemscope="" itemtype="https://schema.org/Person"><div class="avatar__name"><a href="https://github.com/haixuantao" target="_blank" rel="noopener noreferrer" itemprop="url"><span itemprop="name">Haixuan Xavier Tao</span></a></div><small class="avatar__subtitle" itemprop="description">Maintainer of dora-rs</small></div></div> ---  Let's build an autonomous vehicle together... ---  Block we need: - Sensors - Perception - Localisation - Planning - Control ---  ```mermaid flowchart TB Control Planning Perception Localisation Sensors --> Perception Sensors --> Localisation Perception --> Planning Planning --> Control Localisation --> Planning ``` ---  ```mermaid flowchart TB subgraph Perception perception/yolov5[/yolov5\] end subgraph Localisation Localisation/gps[/gps\] end subgraph Control Control/drive_by_wire[/drive_by_wire\] end subgraph Planning Planning/optimizer[/optimizer\] end subgraph Sensors Sensors/Drivers[/Drivers\] end Sensors --> Perception Sensors --> Localisation Perception --> Planning Planning --> Control Localisation --> Planning ``` ---  ```mermaid flowchart TB subgraph Sensors Sensors/Drivers[/ros2/sdk\] end subgraph Perception perception/yolov5[/ Python \] end subgraph Localisation Localisation/gps[/ ros2/C++/Python \] end subgraph Control Control/drive_by_wire[/ros2/sdk\] end subgraph Planning Planning/optimizer[/c++/Python\] end Sensors --> Perception Sensors --> Localisation Perception --> Planning Planning --> Control Localisation --> Planning ``` #### Long implementation that take months ---  ## ROS2 > Honestly, I don't even get what ROS is about? **ros2 seems an "amalgamation" of tools and scripts to create a general API with some ideas of robotics application in mind.** > [name=HN First Comment ] > From my perspective as a generalist software engineer who isn't a low-level robotics expert – as an ecosystem, **[ros2] seems to have adopted every bad practice available and invented some more of its own**. [...]. **Many of the architectural design decisions are frankly baffling**, although I appreciate that this is in part down to age, legacy, and the open nature of the platform. > [name=HN Second Comment ] ---  ### How can `dora-rs` help? - Simple to use - Declarative dataflow - Excellent Python support - AI friendly - Can be built in days ---  ## Carla simulator <img src=https://hackmd.io/_uploads/HJgQIzPJ6.jpg width=650/> ---  From `carla`: ```python from carla import Client, VehicleControl ... vec_control = VehicleControl(steer, throttle, brake, hand_brake=False) client.apply(vec_control) ``` to `dora-rs`: ```python from dora import Node from carla import Client, VehicleControl, get_car_info node = Node() client = Client() for event in node: if event["type"] == "INPUT": if event["id"] == "control": [throttle, steer, brake] = np.array(event["value"]) vec_control = VehicleControl( steer, throttle, brake, hand_brake=False, ) client.apply(vec_control) elif event["id"] == "tick": camera_frame, speed = get_car_info() node.send_output("image", camera_frame, metadata) node.send_output("speed", speed, metadata) ``` ---  `yolov5` example: ```python import cv2 import numpy as np import torch from dora import Node node = Node() model = torch.hub.load("ultralytics/yolov5", "yolov5n") for event in node: match event["type"]: case "INPUT": match event["id"]: case "image": frame = event["value"] results = model(frame) # includes NMS arrays = np.array(results.xyxy[0].cpu()) node.send_output("obstacles", arrays, event["metadata"]) ``` ---  Planning example: ```python import fot_wrapper from dora import DoraStatus class Operator: def __init__(): # ... def on_event( self, dora_event, send_output: Callable[[str, bytes], None], ): if dora_event["id"] == "position": self.position = dora_event["value"] return DoraStatus.CONTINUE elif dora_event["id"] == "speed": self.speed = dora_event["value"] return DoraStatus.CONTINUE elif dora_event["id"] == "obstacles": obstacles = dora_event["value"] waypoints = fot_wrapper.run_fot( self.position, self.speed, obstacles, ) send_output( "waypoints", waypoints, dora_event["metadata"], ) return DoraStatus.CONTINUE ``` ---  ## Graph example: ```yaml nodes: - id: oasis_agent custom: inputs: waypoints: fot_op/waypoints outputs: - speed - image source: ../../carla/carla_source_node.py - id: carla_gps_op operator: python: ../../carla/carla_gps_op.py outputs: - position - id: yolov5 operator: outputs: - obstacles inputs: image: oasis_agent/image python: ../../operators/yolov5_op.py - id: fot_op operator: python: ../../operators/fot_op.py outputs: - waypoints inputs: position: carla_gps_op/position speed: oasis_agent/speed obstacles: yolov5/obstacles ``` ---  #### Node hub: ![](https://hackmd.io/_uploads/ryVU6l0Cn.png) ---  #### Hot reloading and opentelemetry <iframe width="560" height="315" src="https://www.youtube.com/embed/lTzPTxCUFSM?si=UWMqgddKmksJBpxs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe> --- ### dora-drives: Autonomous Driving Starter Kit ![yolov5](https://user-images.githubusercontent.com/22787340/187723794-3623bee2-91d6-436a-a5d7-d2e363483c76.gif =300x) ![drivable_area](https://user-images.githubusercontent.com/22787340/187723841-7f3ba560-dbbe-4d43-886a-fb3b0be9247a.gif =300x) ![strong_sort](https://user-images.githubusercontent.com/22787340/187723873-473cda4f-573d-4663-a5b9-a4df2611c482.gif =300x) See: https://github.com/dora-rs/dora-drives  --- ### Implemented Operators - Yolov5 Object detection - YoloP Lane and Drivable area detection - Midas Depth Estimation - Strong Sort Object tracking - Frenet Optimal Planning Planning - PID Controller - Integration to Carla - ...  --- ## dora-rover <div> <div id="left"> <img src=https://hackmd.io/_uploads/rJk6Hzwyp.jpg width=350/> </div> <div id="right"> - Lidar - GNSS, IMU - Camera - NVIDIA Orin </div> </div>  --- ## dora-car <div> <div id="left"> <img src=https://hackmd.io/_uploads/B1aS9zwJp.jpg width=550/> </div> <div id="right"> - Chinese Lidar - GNSS, IMU - 6 Cameras - RK3588 ARM Computers </div> </div>  --- ## Hackathon Inviting students to participate in an International Hackathon on Autonomous Driving with a prize pool of 500k RMB. Getting started guide for the Hackathon: https://docs.carsmos.cn/#/en/ Official Website for Hackathon registration: https://competition.atomgit.com/competitionInfo  --- ## dora-rs Github: https://github.com/dora-rs/dora Website: https://dora.carsmos.ai/ DIscord: https://discord.gg/XqhQaN8P