VII WPIETF Hackathon

# VII WPIETF Hackathon ###### tags: `wpietf` `ietf` `bmwg` `benchmarking` `vnf` `gym` > **Table of Contents** [ToC] --- ## Context To establish some context, first we need to read a few words about an actual IETF Hackathon (extracted from https://ietf.org/blog/how-many-words/): > "The first thing that typically comes to mind when thinking of the IETF is Internet standards, published as RFCs. RFCs are the basis on which the Internet as we know it is built. RFCs are evidence of the importance of words, especially carefully crafted and reviewed words that specify what each network protocol does and how it works. The IETF Hackathon promotes running code to complement existing RFCs and accelerate the creation of new network protocols that improve the functionality, manageability, and security of the Internet." As the WPIETF is an annual event whose main objective is to foster and expand the presence of the Internet Engineering Task Force (IETF) in Latin America, it aims as well to exemplify the IETF Hackathon. ## Goals The goal of this WPIETF Hackathon 2020 is to make users get familiar with some IETF concepts, in particular get involved in a project, understand its reference implementation, and contribute to it coding, and aiming prizes. This year, the WPIETF Hackathon refers to one of the oldest Working Groups in IETF, the Benchmarking Methodology Working Group (BMWG), addressing a new draft: [Methodology for VNF Benchmarking Automation. ](https://datatracker.ietf.org/doc/draft-rosa-bmwg-vnfbench/) This draft proposes an architectural framework, YANG modules, and the means to realize its content with a reference implementation called gym. The WPIETF Hackathon is about enhancing gym with new tools to perform custom benchmarking setups for VNFs. You can find details about that in the "Hackathon Instructions" below. :::success Why coding the support for a VNF benchmarking tool is important? Similar to VNFs, some benchmarking tools, e.g., packet generators, have been built in software lately. Therefore, the flexible means to use these tools to bencuhmark VNFs is really important. Besides, different benchmarking tools can enable different benchmarking methodologies for VNFs. ::: ## Coding Skills The required coding skills are mostly in the language Python version 3. ## Source Code The source code referred to the WPIETF Hackathon can be obtained here: https://github.com/raphaelvrosa/gym # Schedule The WPIETF Hackathon will take place: - **From November 25th - 12am BRT** - **Until November 30th - 12am BRT** Meeting to announce the winners and prizes: 30th 2pm. # Hackathon Instructions ## Step 1: Read the draft In IETF and here as well, a good start is always getting acquainted with the subject in question, in this case of WPIETF Hackathon it is the draft: [Methodology for VNF Benchmarking Automation. ](https://datatracker.ietf.org/doc/draft-rosa-bmwg-vnfbench/) Take some time to understand the main concepts of the draft, read a bit about the YANG models, and felling familiar with the methodology proposed. In this part you need to understand what is ann agent and a prober, i.e.: > Agent (Active Prospection) -- executes active stimulus using probers, to benchmark and collect network and system performance metrics. A single Agent can perform localized benchmarks in execution environments (e.g., stress tests on CPU, memory, storage Input/Output) or can generate stimulus traffic and the other end be the VNF itself where, for example, one-way latency is evaluated. The interaction among two or more Agents enable the generation and collection of end-to-end metrics (e.g., frame loss rate, latency) measured from stimulus traffic flowing through a VNF. An Agent can be defined by a physical or virtual network function, and it must provide programmable interfaces for its life cycle management. > Prober -- defines an abstraction layer for a software or hardware tool able to generate stimulus traffic to a VNF or perform stress tests on execution environments. Probers might be specific or generic to an execution environment or a VNF. For an Agent, a Prober must provide programmable interfaces for its life cycle management, e.g., configuration of operational parameters, execution of stilumus, parsing of extracted metrics, and debugging options. Specific Probers might be developed to abstract and to realize the description of particular VNF benchmarking methodologies. ## Step 2: Understand gym If you read the draft, in the reference implementation section Gym is enlisted and shortly described, this section aims to give a better (not longer) picture of Gym. One of the major points of Gym’s architecture sits in the fact is was built by the draft at the same time it also influenced the draft content. Those characteristics make possible back and forth reviews of both the draft as much as the Gym source code simultaneously. For instance, the definition of the draft concepts about probers and listerners derived from the Gym architecture, while the utilization of YANG models by Gym was something new brought by the draft. In essence Gym is faithful to the draft in what concerns the definition and utilization of YANG models, however its internal architecture components differ from the draft. Such freedom of choices is something important that was realized while designing the Generic VNF Benchmarking Architectural Framework. In Gym, the Manager component is divided into two components, a Manager and a Player. In summary, it conceives a decoupling of functions, Player performs the execution of Tests while Manager performs the execution of Trials. This design choice created a modular architecture for Gym, facilitating its implementation and utilization. Besides that, Gym implements Agent (Probers) and Monitor (Listeners) components as defined by the draft. Each component in Gym is a micro-service, communicating with each other via gRPC interfaces and protobuf messages. All the components are written in the Python programming language, version 3.8. The support of YANG methods was provided thanks to the pyangbind python package, which enables the compiling of a YANG model into a python source code, which is used as an API by Gym when utilizing VNF-BR, VNF-BD and VNF-PP models. In Gym the workflow of execution of a VNF-BR takes place in the following manner: 1. Player parses a submitted VNF-BR, defines the set of VNF-BD instances by multiplexing all the input variables of the VNF-BR. It might interface different plugins to perform instantiation of the deployment scenario by an orchestration engine. Performed the scenario instantiation, Player reaches the Manager component with a Task message, containing the set of Trials to be executed in a realization of a VNF-BD instance Test. 2. Manager receives the Task and properly deliver a set of Instruction messages to Agent and Monitor components, containing Actions that trigger each of the probers or listeners to be used for the extraction of VNF metrics. Manager repeats this set of instructions according to the number of Trials. After those are completed, Manager returns a Report message to Player signaling the Test ended. 3. Player iterates throughout all the Tests of each VNF-BD instance, until finishing the whole combination of the variables in the inputs of the VNF-BR. Each VNF-BD instance has a VNF-PP generated as the Test metrics are extracted. Finally, Player creates the outputs part of the VNF-BR and returns it through the interface it was submitted. ## Step 3: Play: clone, install, and run gym Gym source code is provided at: https://github.com/raphaelvrosa/gym. The gym source code is built and tested in Ubuntu 20.04, gym requirements are mostly python 3.8 and some python packages. Let's clone it: ```bash= sudo apt install git git clone https://github.com/raphaelvrosa/gym ``` ### Step 3.1: Installing Different ways exist on how to install gym components. 1. Locally: ```bash= cd gym make install ``` 2. Docker ```bash= cd gym make docker-build ``` 3. Virtual Machine (virtual-box or qemu/kvm) ```bash= cd gym make [ vagrant-run-virtualbox | vagrant-run-libvirt ] ``` ### Step 3.2: Running To execute gym examples, take a look at the README.md file of https://github.com/raphaelvrosa/gym. Gym used gym-cli component to automate the execution of examples, in the README you can find more about it. Every component in gym has a --help option, thus you can run: ```bash= cd gym-[ cli | agent | monitor | manager | player ] --help ``` ### Step 3.3: Look at some Tests Gym is not perfect (for sure), so it does not contain a set of comprehensive tests, however there are useful tests to understand how each gym component works. Take a look and execute the examples (test_agent.py, test_monitor.py, test_manager.py and test_player.py) in https://github.com/raphaelvrosa/gym/tree/master/gym/tests/integration Those tests are going to be useful in the next step. ## Step 4: Code he objective is to code one or more gym agent probers. Currently gym supports some probers, such as ping, iperf3, tcpreplay. Probers are located inside agent folder https://github.com/raphaelvrosa/gym/tree/master/gym/agent/probers Every time gym-agent is started, it loads all the probers in that folder, making them available to be called in benchmarking tests/trials. The candidate tools to become probers are listed below: - Moongen: https://github.com/emmericp/MoonGen - T-REX: https://github.com/cisco-system-traffic-generator/trex-core - dns64perfpp: https://github.com/bakaid/dns64perfpp - siitperf: https://github.com/lencsegabor/siitperf In the following steps there are instructions on how to code and evaluate a prober in gym. :::warning Choose one or more probers from the list above and proceed with the next steps to make it available for gym. Or choose your own tool as well and propose a prober. ::: ### Step 4.1: Code the instructions that install the prober requirements New to gym, now every tool that is utilized by a prober has a file with instructions to install/uninstall its requirements and the tool itself. These files are simple, examples of them are contained in: https://github.com/raphaelvrosa/gym/tree/master/util/tools, as explained in the commented example below. ```yaml= name: iperf3 # Name of the tool mode: prober # How the tool is used by gym (i.e., prober or listener) install: # List of bash commands that gym will use to install the tool - sudo apt install iperf3 uninstall: # List of bash commands that gym will use to uninstall the tool - sudo apt remove iperf3 ``` To evaluate this file, in the cloned folder of gym you can run: ```bash= cd gym make install-tools ARGS="iperf3" # ARGS can be a list of names of tools e.g., ARGS="ping iperf3" ``` In the example above, ARGS is the string containing the names of the tools separated by spaces. To install the tools referenced by ARGS run `make install-tools ARGS=""` and to uninstall them `make uninstall-tools ARGS=""`. In case of ARGS="all", all the tools will be installed/uninstalled. :::info Code a file with the name of the tool you choose, so it enables the install/uninstall of the tool by gym. Try it out and check if your tool was actually installed/uninstalled. ::: ### Step 4.2: Code the prober To understand what a prober does, first try to run one. After installed gym, try the command: ```bash= /usr/bin/python3 gym/agent/probers/prober_ping.py --info ``` And then try to see the prober arguments: ```bash= /usr/bin/python3 gym/agent/probers/prober_ping.py --help ``` Then, run the prober: ```bash= /usr/bin/python3 gym/agent/probers/prober_ping.py --duration 3 --target 127.0.0.1 ``` Now you have a good idea of what a prober does! To code a gym-agent prober, you will need to code a file similar to https://github.com/raphaelvrosa/gym/blob/master/gym/agent/probers/prober_ping.py And after coded you should place the file named "prober_<name_of_the_tool>" in the folder /gym/agent/probers, i.e., in the same place as prober_ping.py. The source code of a prober is simple, let's go through the prober_ping.py file. - In the imports part of the file, see "from gym.common.defs import PROBER_PING". You should place a name as PROBER_<name_of_the_tool> and update the file gym/common/defs.py with that definition, i.e., a simple code similar to the others in that file. - PARAMETERS inside the class ProberPing contain in its keys the name of the parameters to be exposed by the prober to gym-agent, and the values are those to be translated to be used by the ping tool. - METRICS are the metrics that the prober will produce when correctly executed, the keys contain the names of the metrics and the values contain the explanation of them. - The __ init __() method in ProberPing makes all the configuration of the inherited class Prober with the parameters, metrics, name and code of the ping prober. - In the init method the self._command variable is initialized with the name of the tool, as it can be called in the bash terminal (i.e., ping) - The options method realizes any customization of the parameters of the tool. I.e., when the file prober_ping.py is called it is going to receive the parsed parameters and output a dict with opts and timeout fields. The key opts must define in its value a list of options in the correct sequence to execute the tool. The option timeout must exist only when the tool needs to be stopped by gym-agent when it is being executed. For instance ping already has the options packets (-c) or duration (-i) that were set as mandatory to run the prober, so it does not need the timeout option. - The parser method receives the output of the ProberPing self._command with the parsed options and parses the results (out:stdout and err:stderror) and produces the ProberPing metrics (i.e., METRICS). The output/return of parser must be a list of metrics. Each metrics must be in the format of a dict containing the fields name, type, unit, and scalar or series. Scalar refers to a single value of the metrics. Series refer to a key-valued metric (e.g, timeseries). A metric must only contain one of the fields, scalar or series. Instructions: 1. Make a copy of the file prober_ping.py in the same folder where it resides. The edit this file. 2. Create the PROBER_<name_of_the_tool> variable for your file. 3. Change the name of the class ProberPing to something else. 4. Define the class self._command name to be the command of the tool. 5. Edit the PARAMETERS inside your class to set the tool parameters. 6. Similar to PARAMETERS, define the METRICS and their explanation. 7. Code the options method. 8. Code the parser method, producing metrics 9. Test your prober, similar to the ways you learned how to run the prober_ping.py file (i.e., python3 prober_ping.py --help or --info) ### Step 4.3: Code the prober test 1. Make a copy of the file https://github.com/raphaelvrosa/gym/blob/master/gym/tests/integration/test_agent.py and rename it to test_agent_prober.py. 2. Open the file and edit the part of the function `call_instruction` changing the action.name and action.args with the correct name and args of your coded prober. 3. In the function `test_instruction` change the expected_metrics variable to the METRICS defined in the prober you just coded. 4. Then, try to run the test file and see if it passes with success. ```bash= /usr/bin/python3 gym/tests/integration/test_agent_prober.py ``` ## Step 5: Document Clearly document the prober and its test, in order to do that, try to: - Document the prober install/uninstall script placed in gym/util/tools (e.g., explain why the packages are needed and provide the source of the instructions) - Document the main functions of the prober (e.g., options and parser) - Document the prober test file with the intended tests and expected results ## Step 6: Code the VNF-BR example(s) Similar to the gym/examples/vnf-br-001.json example that utilizes the prober_ping, create a VNF-BR example that makes usage of the prober you just coded. Be creative in the set of parameters to be utilized. Tip: Just make a copy of vnf-br-001.json and edit the prober part of the ping (from lines 305 to 323). In the prober options inside the VNF-BR file, the name must be the name you defined for the prober as initialized in the inherited Prober class, and the parameters must be the ones in the PARAMETERS that will trigger the tool execution. --- # Code Delivery The source code and documentation must be sent by e-mail to: raphaelvrosa@gmail.com Set the subject of the e-mail as: "WPIETF Hackathon" Add to the e-mail content the names of the members of the team. As a tiebreaker evaluation criteria, preference will be given for earlier code deliveries. Finally, after sending the source code by e-mail, in order to complete the Hackathon in the 30th of November at 12am, you need to make a pull request as ([Tutorial](https://yangsu.github.io/pull-request-tutorial/)): - Create a fork of the github project https://github.com/raphaelvrosa/gym - Clone the forked project to your machine - Add the coded probers, tests, and examples to the forker-cloned project - Create a commit with the first line: your name - WPIETF Hackathon - Push the source code to your github repository - Create a pull request of your source code repository to the project https://github.com/raphaelvrosa/gym --- # Evaluation Criteria In order to succeed in the WPIETF Hackathon each team must: - [x] Implement one or more agent probers - [x] Implement the tests to exemplify the execution of the coded prober(s) - [x] Document the prober(s) source code, test, and the VNF-BR example(s) The evaluation of the Hackathon will consider: 1. The amount of coded agent probers and their tests 3. The amount and details of the documentation of the prober(s) and test(s) 4. The quality of the provided source code (e.g., correctedness) 5. The amount of VNF-BR examples provided.