# Programming FPGA on USRP 2944R/Ettus X310 with UHD & RFNoc -- II
# Mutilple I/O RFNoC module

Since 1In,1Out Rfnoc development has been done. Right now we can have a gain module in FPAG controlled by GunRadio. If you haven't seen this, the links are down below.
+ Workflow for single I/O module: [Programming FPGA on USRP 2944R/Ettus X310 with UHD & RFNoc -- I](/lTGoLnG3TEaOSiuwgf6RzQ)
+ Problems so far: [Problem Collection for USRP](/GHUV5o10Q9yw0GgNtNsTVQ)

On this page, we are going to implement multiple input and output ports on the RFNoC module with custom FPGA logic. However, having some knowledge about RFNoc frame is necessary before we start coding.

## 0. Brief of RFNoC Architecture
As the following figure, we can roughly divide the whole USRP workflow into several parts. For people that developing functions with USRP, we especially focus on FPGA and General-Purpose Processors(GPP).
Most of time we can do DSP stuff on GPP generally. USRP receives/transmit the signal to/from GPP and the user processes the digital data with software like Matlab or python. Even so, things go different when datarate got higher or the DSP is required to be real-time. Thus, FPGA is introduced into the SDR.
To let a custom FPGA module work on USRP, there are three layers we'll work on - the FPGA module with custom logic, the application side for GPP to apply and control the FPGA module on GnuRadio or Matlab, and the driver between the hardware and software.

Just like what we talked about in the last article, we know each layer can be indicated by the file structure. Generally, the files to corresponding lays can be done by completing the .yaml file. The system will read the description and generate the corresponding files. However, when it comes to custom configuration, it's necessary to program them manually. We'll have more details about how to modify them later on.
In the last part, let's focus on the architecture of RFNoC. RFNoC is just an interface module between the top module and user logic. Actually, there are lots of other FPGA logic in the USRP, like peripheral interface. That's why the RFNoC was introduced. It's developed to simply connect between control signal flow and data streamflow. RFNoC is the one that strings up all the DSP modules. Take a look at the slide from Ettus. It shows the whole architecture of RFNoC. If we remind the last article, the *rfnoc* folder is the one that enables us to handle the whole RFNoC frame.
From far to close:
+ Image Core YAML under *rfnoc/icore* controlls the connections between Stream Endpoints, Static Router and NoC Blocks.
+ Block Description YAML under *rfnoc/blocks* controlls the NoC Blocks.
+ noc_shell_gain.v & rfnoc_block_gain.v under *rfnoc/fpga* are the verilog file to creat NoC Shell and User Logic.
///CHDR stuf & AXIS
Another connection configuration is showed above, indicating that it's possible to have multiple same blocks and connected to each other.

Overall, the development workflow based on my experience would be like this:


## 1. Hardware
### 1.1 rfnoc.yml
If the default configurations are unsatisfying, like the I/O port number, clk domain, or data type, rfnoc.yml is the first step to work on. There are lots of examples of rfnoc.yml in UHD repository. It's suggested to check them out first.
For me, I tried *add* module to demonstrate two I/O module, thus I edited the .yml file under block as:

data:
  fpga_iface: axis_data
  clk_domain: ce
  inputs:
    in_a:
      item_width: 32
      nipc: 1
      context_fifo_depth: 32
      payload_fifo_depth: 32
      format: sc16
      mdata_sig: ~
    in_b:
      item_width: 32
      nipc: 1
      context_fifo_depth: 32
      payload_fifo_depth: 32
      format: sc16
      mdata_sig: ~
  outputs:
    out:
      item_width: 32
      nipc: 1
      context_fifo_depth: 32
      payload_fifo_depth: 32
      format: sc16
      mdata_sig: ~

Then after executing the *rfnoc_create_verilog.py*, I got the following result in .v file, which proved that the noc_shell had been modified correctly.

 // Data Stream to User Logic: in_A
  wire [32*1-1:0]    m_in_A_axis_tdata;
  wire [1-1:0]       m_in_A_axis_tkeep;
  wire               m_in_A_axis_tlast;
  wire               m_in_A_axis_tvalid;
  wire               m_in_A_axis_tready;
  wire [63:0]        m_in_A_axis_ttimestamp;
  wire               m_in_A_axis_thas_time;
  wire [15:0]        m_in_A_axis_tlength;
  wire               m_in_A_axis_teov;
  wire               m_in_A_axis_teob;
  // Data Stream to User Logic: in_B
  wire [32*1-1:0]    m_in_B_axis_tdata;
  wire [1-1:0]       m_in_B_axis_tkeep;
  wire               m_in_B_axis_tlast;
  wire               m_in_B_axis_tvalid;
  wire               m_in_B_axis_tready;
  wire [63:0]        m_in_B_axis_ttimestamp;
  wire               m_in_B_axis_thas_time;
  wire [15:0]        m_in_B_axis_tlength;
  wire               m_in_B_axis_teov;
  wire               m_in_B_axis_teob;   

Or in a more clear way, which's the verilog code in UHD repository as well.

  // Payload Stream to User Logic: in_a
  output wire [32*1-1:0]    m_in_a_payload_tdata,
  output wire [1-1:0]       m_in_a_payload_tkeep,
  output wire               m_in_a_payload_tlast,
  output wire               m_in_a_payload_tvalid,
  input  wire               m_in_a_payload_tready,
  // Context Stream to User Logic: in_a
  output wire [CHDR_W-1:0]  m_in_a_context_tdata,
  output wire [3:0]         m_in_a_context_tuser,
  output wire               m_in_a_context_tlast,
  output wire               m_in_a_context_tvalid,
  input  wire               m_in_a_context_tready,
  // Payload Stream to User Logic: in_b
  output wire [32*1-1:0]    m_in_b_payload_tdata,
  output wire [1-1:0]       m_in_b_payload_tkeep,
  output wire               m_in_b_payload_tlast,
  output wire               m_in_b_payload_tvalid,
  input  wire               m_in_b_payload_tready,
  // Context Stream to User Logic: in_b
  output wire [CHDR_W-1:0]  m_in_b_context_tdata,
  output wire [3:0]         m_in_b_context_tuser,
  output wire               m_in_b_context_tlast,
  output wire               m_in_b_context_tvalid,
  input  wire               m_in_b_context_tready,

### 1.2 noc_shell.v
As we discussed before, it's the module to create the interface to RFNoC network and user logic, which's known as CHDR and AXIS as well. It's worth noting that the skeleton generated by rfnocmodtool will be like the following code, which's not easy to read and a little different from the build-in gain example. Though I just quote parts of the code, it's obvious that the context stream is named as the original definition.

// Data Stream to User Logic: in_a
  output wire [32*1-1:0]    m_in_a_axis_tdata,
  output wire [1-1:0]       m_in_a_axis_tkeep,
  output wire               m_in_a_axis_tlast,
  output wire               m_in_a_axis_tvalid,
  input  wire               m_in_a_axis_tready,
  output wire [63:0]        m_in_a_axis_ttimestamp,
  output wire               m_in_a_axis_thas_time,
  output wire [15:0]        m_in_a_axis_tlength,
  output wire               m_in_a_axis_teov,
  output wire               m_in_a_axis_teob,
  // Data Stream to User Logic: in_b
  output wire [32*1-1:0]    m_in_b_axis_tdata,
  output wire [1-1:0]       m_in_b_axis_tkeep,
  output wire               m_in_b_axis_tlast,
  output wire               m_in_b_axis_tvalid,
  input  wire               m_in_b_axis_tready,
  output wire [63:0]        m_in_b_axis_ttimestamp,
  output wire               m_in_b_axis_thas_time,
  output wire [15:0]        m_in_b_axis_tlength,
  output wire               m_in_b_axis_teov,
  output wire               m_in_b_axis_teob,

For me, it's hard to read. Thus after reading the rfnoc example in UHD repository, I modified the stream like this.

// Payload Stream to User Logic: in_a
  output wire [32*1-1:0]    m_in_a_payload_tdata,
  output wire [1-1:0]       m_in_a_payload_tkeep,
  output wire               m_in_a_payload_tlast,
  output wire               m_in_a_payload_tvalid,
  input  wire               m_in_a_payload_tready,
  // Context Stream to User Logic: in_a
  output wire [CHDR_W-1:0]  m_in_a_context_tdata,
  output wire [3:0]         m_in_a_context_tuser,
  output wire               m_in_a_context_tlast,
  output wire               m_in_a_context_tvalid,
  input  wire               m_in_a_context_tready,
  // Payload Stream to User Logic: in_b
  output wire [32*1-1:0]    m_in_b_payload_tdata,
  output wire [1-1:0]       m_in_b_payload_tkeep,
  output wire               m_in_b_payload_tlast,
  output wire               m_in_b_payload_tvalid,
  input  wire               m_in_b_payload_tready,
  // Context Stream to User Logic: in_b
  output wire [CHDR_W-1:0]  m_in_b_context_tdata,
  output wire [3:0]         m_in_b_context_tuser,
  output wire               m_in_b_context_tlast,
  output wire               m_in_b_context_tvalid,
  input  wire               m_in_b_context_tready,

### 1.3 rfnoc.v
This file contains our custom logic, which's an add module. However, if you take a look at this file. You found that it's the top module of the whole RFNoC module. It receives the CHDR stream from the host module then passes it to the noc_shell. Noc_shell sends back the split stream, which will be AXIS formate, to the rfnoc.v file.
The structure of rfnoc.v is shown clearly with the official slide, in which all the yellow area is the rfnoc.v covering.
Another thing, the add module example provided in repository is too complex for me. Here I use my logic and verify it in vivado first.

### 1.4 test bench
:::success

Vivado Simulator does not support tracing of System Verilog Dynamic Type object.
========================================================
TESTBENCH STARTED: rfnoc_block_orgate_tb
========================================================
[TEST CASE   1] (t =         0 ns) BEGIN: Flush block then reset it...
[TEST CASE   1] (t =      2455 ns) DONE... Passed
[TEST CASE   2] (t =      2455 ns) BEGIN: Verify Block Info...
[TEST CASE   2] (t =      2455 ns) DONE... Passed
[TEST CASE   3] (t =      2455 ns) BEGIN: Test random packets...
[TEST CASE   3] (t =     18460 ns) DONE... Passed
[TEST CASE   4] (t =     18460 ns) BEGIN: Test without back pressure...
[TEST CASE   4] (t =     32095 ns) DONE... Passed
[TEST CASE   5] (t =     32095 ns) BEGIN: Test back pressure...
[TEST CASE   5] (t =     56725 ns) DONE... Passed
[TEST CASE   6] (t =     56725 ns) BEGIN: Test underflow...
[TEST CASE   6] (t =     81500 ns) DONE... Passed
[TEST CASE   7] (t =     81500 ns) BEGIN: Test min packet size...
[TEST CASE   7] (t =     82185 ns) DONE... Passed
========================================================
TESTBENCH FINISHED: rfnoc_block_orgate_tb
 - Time elapsed:  82185 ns
 - Tests Run:     7
 - Tests Passed:  7
 - Tests Failed:  0
Result: PASSED   
========================================================
INFO: [USF-XSim-96] XSim completed. Design snapshot 'rfnoc_block_orgate_tb_behav' loaded.
INFO: [USF-XSim-97] XSim simulation ran for 1000000000us

:::
### 1.5 Icore.yml

## 2. Driver
### _ctrl_impl.cc

It's the main code for the driver, which provides the control and data steam exchange between the hardware and software, like the block argument in GnuRadio. Bascially we apply the api in UHD for our design, and there is some example in UHD repository as well. Go check them out. Be careful that they only provide the cpp file, other files like header should be completed by yourself. Fortunately, they're easy to code with the skeleton generated by rfnocmodtool.

## 3. Application
### /grc/.yml

For GnuRadio newer, let's have some clarification first. If you run the grc in ./example folder, which's the graphical configuration of GnuRadio. It can be understood as the project file of GnuRadio, which records the block arguments you using and the connection. Or we can take it as the whole configuration of SDR. As long as the grc file is executed in GnuRadio, GnuRadio'll generate a .py file automatically. The python file records the real code behind the configuration of grc or the whole SDR project. It's different from the .yml file we talk about here, which only describes how should the custom RFNoC block be represented in the GnuRadio.
Like the port number, the date type, or the control stream. There's no example about this file. Thus my add block.yml is provided below.

parameters:
# - id: user_reg
#   label: User Register
#   dtype: int
#   default: 0
- id: block_args
  label: Block Args
  dtype: string
  default: ""
- id: device_select
  label: Device Select
  dtype: int
  default: -1
- id: instance_select
  label: Instance Select
  dtype: int
  default: -1

# Make one 'inputs' node per input. Include:
#    label (an identifier for the GUI)
#    dtype (data type of expected data)
#    optional (set to 1 for optional inputs)
inputs:
- domain: rfnoc
  label: in_a
  dtype: 'sc16'
- domain: rfnoc
  label: in_b
  dtype: 'sc16'


# Make out 'outputs' node per output.
#    label (an identifier for the GUI)
#    dtype (data type of expected data)
#    optional (set to 1 for optional outputs)
outputs:
- domain: rfnoc
  label: out
  dtype: 'sc16'

## 4. Debugging in runtime
Insert this before the wire we want to probe.

(* dont_touch="true", mark_debug="true" *)

like

(* dont_touch="true", mark_debug="true" *) wire [32*1-1:0]  m_in_a_payload_tdata;
(* dont_touch="true", mark_debug="true" *) wire             m_in_a_payload_tlast;
(* dont_touch="true", mark_debug="true" *) wire             m_in_a_payload_tvalid;
(* dont_touch="true", mark_debug="true" *) wire             m_in_a_payload_tready;
...

Then we build the files and use rfnoc_image_builder, which's we mentioned in the last article before, to force building image with Vivado GUI. After the synthesis stage, we can save the build as a project. Then as long as we save the project, the tool on the left-hand side will appear.
We click in the synthesis stage, the debug probe will be shown in Netlist window. We can configure them with the Set Up Debug tool, which's circled by the red line below.


There may be some problems after using debug probe. It's due to probe constraints still left in .xdc file. Removing them will solve it. See more in [Problem Collection for USRP](/GHUV5o10Q9yw0GgNtNsTVQ).
## 5. Result


## Reference
+ RFNoC4 Workshop Part 2, Jonathon Pendlum – Ettus Research, Neel Pandeya – Ettus Research, GRCon 2020
+ https://kb.ettus.com/Getting_Started_with_RFNoC_in_UHD_4.0
+ https://kb.ettus.com/Getting_Started_with_RFNoC_Development
+ https://wiki.gnuradio.org/index.php?title=Creating_Python_OOT_with_gr-modtool
+ https://kb.ettus.com/RFNoC_(UHD_3.0)
+ https://kb.ettus.com/Debugging_FPGA_images
###### tags: `USRP`, `FPGA`, `Ettus Research`, `UHD`, `RFNoC`, `RFNoC搭建`, `Multiple I/O RFNoC`, `RFNoC Vivado`

>[e-mail]
Department of Space Science & Engineering
Center for Astronautical Physics & Engineering
National Central University, Taiwan
[name=PieappleJ] [time=Sun, Mar 27, 2022]