---
# System prepended metadata

title: TVM with Fixed-point Type (RFC)

---

# TVM with Fixed-point Type (RFC)
> Bing-Sung Lu
---
While we were trying to implement RISC-V p extension in TVM, we found the arithmetic instructions in RISC-V 
p extension only worked with integer type. RISC-V p extension supports fixed point instructions. Therefore, we tried to add fixed-point type in TVM to quantize the inference programs from floating-point type to fixed-point type.

## Transform Floating-point to Fixed-point

The value of a fixed-point type variable is determined by the value of its floating-point counter part and the point position which can be set by the users. The higher point position means more accuracy in decimal part after quantization. However, there will be less bits to express integer part thus resulting a smaller number range. In another word, we could set a higher point position if the data set consists with smaller number range. Otherwise, the quantiztion will cause too much saturation and result in low accuracy prediction from inference program. 

The actual relationship between fixed-point variable and floating-point vairable can be depicted as follow:

We refer Fxp as fixed-point value, Fp as floating-point value and PP as point position

$Fxp = Fp * pow(2,PP)$

We can see that point position is actually the exponent of 2, we multiply it with floating-point value and retain the integer part as fixed-point value

:::info
e.g.
If we set point position as 10. The floating-point value 0.25 is equal to 256 in fixed-point value. (0.25*pow(2,10) = 256)
:::
## Arithmetic of Fixed-point

The arithmetic operations of fixed-point are similar to the ones of integer, but there are some operations need some tweaks to get the correct result.
:::info
e.g.

One of the most common operation we will encounter is multiplication. Assume point position is 10. The fixed-point value of 0.5 is 512. If we multiply 0.5 with itself, we should get 0.25(256 in fixed-point) as result. However, we won't get same result from multiplication of fixed-point value. We have to 
divide it with pow(2,10) for the correct anwser(or shift right 10 digits).
:::

In order to get the correct arithmetic result in fixed-point, the binary point position is an essential information.


## Implementing Fixed-point type in TVM

If we want to build a NN model with fixed-point type while using nnvm or relay as below, we need to set the fixed-point type first in python code.

```
func, params = relay.frontend.from_mxnet(block, shape_dict, "fxp16_13")
with relay.build_config(opt_level=level):
            graph, lib, params = relay.build(func, target, params=params)
```

In this case, the fixed-point type is presented as 'fxp16_13'. 16 is the number of bits in the fixed-point variable. 13 is the point position.

We use llvm as our target in experiment. The main goal is to pass down the information of point position to backend compiler. The method is to add a new type in TVM.

### TVM IR Type

We base our fixed-point type on HalideIR::Type. It is similar to HalideIR::Int. The only different is that we use 'halideir_handle_cplusplus_type' to carry the point position.

```
inline Type Fxp(int bits, int fxp_pos, int lanes = 1) {
    halideir_handle_cplusplus_type *fxp_info = new halideir_handle_cplusplus_type{
            halideir_cplusplus_type_name(halideir_cplusplus_type_name::Simple, "fxp16_"+std::to_string(fxp_pos)),
            {}, {}, {}};
    return Type(Type::Int,bits, lanes,fxp_info);
}

```

### TVMType

Since TVMType is costructed by three integer variables (code, bits and lanes). There is no variable to carry point position. We have to give each possible point position a unique type code to distinguish them from each other. If we want to add fxp16 type into TVMType, we will need 16 codes(numbers) for each possible position.

### Codegen

As mentioned above, we need to add extra instructions so fixed-point multiplication would return correct result. Our method is to modify the codegen of llvm and override 'VisitExpr_(const Mul* op)' function to verify the fixed-point multiplication. The type class in codegen phase is TVM IR type and we can get the data of point position from it. Then we add shift-right instruction according to the point position and generate the fixed-point version of multiplication in llvm IR form.

In our experiment, we use a llvm intrinsic to replace normal integer multiplications with fixed-point multiplications. 
```
llvm::Value* CodeGenRISCV::VisitExpr_(const Mul* op) {

    int fxp_pos=0;
        
    if(op->type.handle_type!=nullptr) {
      std::string fxp_name = op->type.handle_type->inner_name.name;
      if(fxp_name.find("fxp")!=std::string::npos)
        fxp_pos = std::stoi(fxp_name.substr(fxp_name.find("_")+1));
    }
    
    if(op->type.bits()==16||op->type.bits()==8) {
      int total_bits = op->type.bits()*op->type.lanes();
      if(total_bits==32||total_bits==64) {
        Expr e = CreateMulIntr(op,op->type.bits(),op->type.lanes(),fxp_pos);
        return CodeGenCPU::CreateIntrinsic(e.as<Call>());
      }       
    }    
    return CreateMul(op->type, MakeValue(op->a), MakeValue(op->b));  
    
}
```


We should use llvm backend compiler to compile llvm IR and get the correct multiplication library with fixed-point type. 
