WP2 Review Meeting

--- tags: wp2 title: WP2 Review Meeting --- # WP2 Review Meeting ## Objective 2.3: Deliver proof of concept of tools which evaluate software video of the implemented workflow explanation of the most important CI results ## Objective 2.4: Design guidelines that help developers for writing IoT software; workflow introduction brief tool description implemented ## Objective 2.5: Define policy-based software security compliance; @5rm-sQqGSA-7haXfGVY3uQ (Andrea cambia tag!!!!) @marco-rasori could you please propose what to share in the review presentation ## Objective 2.6 Analyse GDPR compliance, licensing compliance and ethical aspects; privacy traffic light -> privacy dashboard (web-app) # Implemented - C-workflow Meson CI complete (sifis-generate https://github.com/sifis-home/sifis-generate creates this demo https://github.com/sifis-home/meson-demo) - Sifis-API. The hazard is described as API documentation (https://github.com/sifis-home/sifis-api) - Ontology link and description @marco-rasori - Rust code analysis on CI both for Linux and Windows (extract json and zip it) - wcc on CI for Rust both on Linux and Windows (extract json and zip it) - Add Miri to CI # Videos - Max 5 minutes - Background music when no-one is talking - Use programs that record screen to show terminal steps - Use subtitles to explain key concepts (as Otto video) - No slides, but images that represent something communicatively appealing ## First video (ontology) - Title: Equipping Hazards to Things - Explain what are Wot Smart Devices. Why we have modified them adding Sifis hazards. (1 minute) > Why we put hazards on thing descriptions? W3C Web Of Things approach is to make every device self-describe itself and uses json-ld to add semantic meaning to each means to interact with the device. We want to make the user aware of the risks that each interaction implies and felt natural to leverage the Thing Description to deliver the information. We identified three macrocategories of hazards, namely, safety, privacy, and financial. As an example, the "unlock door" action performed on a smart lock might result in an "unauthorized physical access", which is a safety hazard. - Link between thing and ontology: explain. - Ontology explanation (2 minutes) - Introduction of a Thing Description in json and the Interaction Affordance part - Why (and where) we are introducing the hazard. - we have defined an ontology: ... - The ontology is defined like this: ... - It has been published here: ... - To get a definition of the ontology in a specific programming language (we have chosen Rust), we need to parse the formal description of the ontology. So we parse the ontology described as json-ld file and produce the relative file containing the list of Hazards with the relative methods to get their names and descriptions. (1 minute) - Show the json-ld ontology file used as input - Show the command to parse the ontology and produce the relative Rust file (maybe ouput something on the terminal) - Show Rust output file (ontology.rs) containing the list of hazards and categories and the methods to interact with them. - ontology.rs is part of lib-sifis, a library that consumes an augmented Thing Description. An augmented Thing Description is a description that contains Sifis-Hazards too. The library binds together APIs to interact with a Thing and APIs to get information about the Sifis-Hazards and Sifis-Categories contained in the ontology. In a more practical way, it parses a json Augmented Thing Description, providing methods to interact with a WoT device, and some methods that allow to verify whether the Hazards contained in the description are coherent with the ontology. - `lib-sifis` is used by the `sifis-api` library, a high-level API library, which exposes APIs, such as `turnOnOven`, for third-party developers. Each API is associated with one or more hazards. We developed an autocomplete plug-in for VisualStudio Code that shows in real time the hazards associated with an API. In this way, a third-party developer can be aware of the risks he is introducing by using each API. ## Second video (sifis-generate) - Title: Automatically generate project structures and Continuous Integration scripts for some commonly-used IoT programming languages - Why have we created this tool? We detailed how a good continuous integration setup should be, but t However, the very same process of creating continuous integration scripts can be in itself fairly repetitive and time consuming. To reduce the effort that developers have to usually put when they need to create a new project and continuous integration scripts, we prepared a tool, called sifis-generate, that generates the bulk of it at least for some programming languages, those commonly-used within the IoT context. Practically, this tool generates new projects structures and the respective Continuous integration scripts automatically. As programming languages, we currently support C, C++, Java, Rust, Python, and JavaScript. For languages that already implement a way or a command to generate a new project structure, the tool generates only CI scripts. - Show the commands to generate each project from terminal (look at the command section below) - Show a CI example for GitHub (let's use the Rust CI because it is more complete and because Rust is an innovative language) and the relative series of CI steps while they are running on GitHub. ### Video 2 sentences - SIFIS-Home intends to design guidelines that help developers to write a secure and well-tested software and provide tools to evaluate it. - One possible way to evaluate a software is to use Continuous Integration which is a practice ...(https://en.wikipedia.org/wiki/Continuous_integration) che consiste nel 2. 3. - architecture description - https://github.com/sifis-home/webthing-rust/runs/6224857699?check_suite_focus=true #### CI jobs description - **clippy-rustfmt**: it verifies whether the Rust code is formatted according to style guidelines and that does not contain some common bad-programming mistakes, thus if the code could be further improved. - (x3 architecture) **build**: It builds the software and run unit and integration tests. - **code-coverage**: It computes the code coverage, so the portion of code covered by tests. If the value is less than 60 (it means that the 60% of code is covered by tests), the CI job fails - **memory-and-threads-dynamic-analysis**: It runs the software (with Valgrind) to check whether there are some memory and thread faults that would make the software insecure. - **undefined-behaviour-fuzzy-dynamic-analysis**: It runs the software (with Miri) to check whether there are some undefined behaviours (the result of executing a program whose behavior is prescribed to be unpredictable in the language specification to which the computer code adheres) and run fuzzy tests (with cargo-fuzz). Fuzzy tests consists of providing invalid, unexpected, or random data as inputs to a computer program in order to monitor if there are exceptions such as crashes, failing built-in code assertions, or potential memory leaks in the software. - (x2 ubuntu and windows architectures) **static-code-analysis**: It analyzes and extracts data from source codes in order to provide information on the quality of a code, point out which part of the code is more tough to handle, and discover errors before the compilation and building phase. - (x3 architecture) **weighted-code-coverage**: It combines code coverage value and complexity in order to better highlight the portion of code which is not covered by tests but at the same time presents an high value of complexity. # Commands for Rust software - Install Rust `https://www.rust-lang.org/it/learn/get-started` - Go into a Sifis-Home Rust repository (the folder with the `Cargo.toml` inside) - Produce a binary without debug symbols inside (reduced filesize and much optimizations) with this command ``` cargo build --release ``` - Inside the `target/release` folder you find the binary - Run the binary ## Sifis-API High level https://github.com/sifis-home/sifis-api ## Run generate-sifis-hazards https://github.com/sifis-home/libsifis-rs ``` ./generate-sifis-hazards -v --template rust ontology-file path/you/want/to/produce/the/file ```  ## Run sifis-generate https://github.com/sifis-home/sifis-generate - Meson C ``` ./sifis-generate new -v --template meson-c project-name ``` - Meson C++ ``` ./sifis-generate new -v --template meson-c++ project-name ``` - Python (poetry) ``` ./sifis-generate new -v --template poetry project-name ``` - Java (maven) ``` ./sifis-generate new -v --template maven project-groupid.project-name ``` - Rust (only CI scripts) ``` ./sifis-generate new -v --template cargo-ci project-name ``` - Yarn (only CI scripts) ``` ./sifis-generate new -v --template yarn-ci project-name ``` # Adding output to software - libsifis-rs - sifis-api - sifis-generate: - [x] Change job names - [x] Remove beta and nightly - [x] Output the created project - [x] Create a dummy repository or a dummy project in Rust to show continuous integration (webthing-rust: https://github.com/sifis-home/webthing-rust/actions/runs/2233116769). To make `project-name.yaml` work for our goal: - Replace line 3 with `on: [push, pull_request]` in order to run CI not only on the `master` branch and delete lines from 4 to 9 - Replace lines from 161 to 166 ```yaml - name: Run cargo-valgrind run: | cargo valgrind run --example single-thing & EXAMPLE_PID=$! sleep 5 kill -15 $EXAMPLE_PID ``` - Replace lines from 221 to 229 ```yaml - name: Run AddressSanitizer env: RUSTFLAGS: -Zsanitizer=address RUSTDOCFLAGS: -Zsanitizer=address run: | cargo run -Zbuild-std --target x86_64-unknown-linux-gnu --example single-thing & EXAMPLE_PID=$! sleep 5 kill -15 $EXAMPLE_PID ``` - [x] Write about meson and setuptools (what's inside the directories) ## Possible reviewers questions - Is this ontology thought only for WoT? Can it be integrated with other frameworks (Fiware for example)? # Workflow in Rust ## Fast Static Analysis - [x] Style checks - rustfmt - clippy - [ ] Code Clarity / Complexity - Rust-Code-Analysis (Cognitive & Cyclomatic) - Set a threshold (Match SonarCube), 15 - Get snippet of code with complexity spaces Tools: - [jql](https://github.com/yamafaktory/jql) ## Compile test - [x] cargo build ## Unit Test & Coverage - [x] We can copy the https://docs.codecov.com/docs/codecov-delta logic. (see also the [tutorial](https://docs.codecov.com/docs/codecov-tutorial)) (missing Windows for a bug in grcov related to Path) - [x] Weighted Code Coverage ## Static Analisys - [x] rustc works ## Dynamic Analisys - [x] cargo-fuzz (only on Ubuntu because we are doing fuzzy so architecture-independent) (rav1e) - [x] ASAN (on real architecture) (passing rustflags) [see here](https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/sanitizer.html) - [x] Valgrind (on virtual architecture) (cargo documentation about setting the [RUNNER](https://doc.rust-lang.org/cargo/reference/config.html?highlight=runner#environment-variables) or [the applet](https://github.com/jfrimmel/cargo-valgrind)) - [x] Miri (on virtual architecture) (bevy) # TODO @Luni-4 - [x] Split CI files for each architecture (a possible alternative implementation for sifis-generate), so one can use only the CI needed for the chosen architecture (lots of duplicated code, but we don't care since we generate that automatically) - [x] Replace `setuptools` with `Poetry` (https://python-poetry.org/) - [x] Deploy script - [ ] GitLab is now optional since all our code is on GitHub, but it could be useful for other stuff (VLC, Gstreamer)(low priority) # Thoughts - Run Miri, ASAN and Valgrind on Windows too? They all build and run Rust code on their own virtual machine, but `cgf` code should be considered too in my opinion - Miri works on Windows - Valgrind is Windows specific (replacement??) - ASAN is not supported on Windows