Lab 3: Benchmarking
===
Code base: https://github.com/Edward-Sun/11-767/tree/main/labs/lab2-benchmarking

===
The goal of this lab is for you to benchmark and compare model inference efficiency on your devices. **You should benchmark 2*N* models or model variants, where *N* is the size of your group (so, two models per person.)** For now, if you don't have appropriate evaluation data in place that's fine; you can provide pretend data to the model for now and just evaluate efficiency.

Ideally, the models you benchmark will be related to and useful for your class project, but at the very least the efficiency metrics should be useful.

Include any code you write to perform this benchmarking in your Canvas submission (either as a link to a folder on github, in a shared drive, zip, etc).

Group name:
---
Group members present in lab today:
Donghan Yu, Zhiqing Sun, Ruohong Zhang

1: Models
----
1. Which models and/or model variants will your group be benchmarking? Please be specific.
We will chose several Transformer-based model including: BERT, Distill-BERT, ALBERT, Funnel Transformer, MobileBERT, TinyBERT.
For all the later experiments, we set sequence length as 128.
3. Why did you choose these models?
We will use pre-trained Transformer for our project, so the models we choose are related.
5. For each model, you will measure parameter count, inference latency, and energy use. For latency and energy, you will also be varying a parameter such as input size or batch size. What are your hypotheses for how the models will compare according to these metrics? Do you think latency will track with energy use, and parameter count? Explain.

2: Parameter count
----
1. Compute the number of parameters in each model. Remember, in Torch you should be able to start with something like this:

   num_params = sum([np.prod(p.size()) for p in model.parameters()])

Report your results in a table.

| Model | #param |
|------- |--------|
| BERT | 108M |
| DistillBERT | 64M |
| ALBERT-base | 12M |
| Funnel Transformer (B4-4-4 w/o decoder) | 116M |
| Funnel Transformer (B4-4-4 w/ decoder) | 132M |
| MobileBERT (24L w/o decoder) | 24.6M |
| MobileBERT (24L w decoder) | 36.6M |
| TinyBERT | 14.5M |

2. Does this number account for any parameter sharing that might be part of the model you're benchmarking?

Yes. ALBERT use parameter sharing across all the layers, so its number of parameters is much smaller than other models.

3. Any difficulties you encountered here? Why or why not?

Yes. We were confused why Funnel Transformer without decoder (116M) has a slightly larger #param than BERT (110M). It turns out it uses Relative Positional Attention which includes another attention transformation matrix $W_R$. Besides, it seems that the decoder part in MobileBERT account around 1/3 parameters, which suprises us. For other models we don't encounter any difficulties since we already have some prior knowledge about them.

3: Latency
----
1. Compute the inference latency of each model. You should do this by timing the forward pass only. For example, using `timeit`:

    from timeit import default_timer as timer

    start = timer()
    # ...
    end = timer()
    print(end - start) # Time in seconds, e.g. 5.38091952400282

Best practice is to not include the first pass in timing, since it may include data loading, caching, etc.* and to report the mean and standard deviation of *k* repetitions. For the purposes of this lab, *k*=10 is reasonable. (If standard deviation is high, you may want to run more repetitions. If it is low, you might be able to get away with fewer repetitions.)

For more information on `timeit` and measuring elapsed time in Python, you may want to refer to [this Stack Overflow post](https://stackoverflow.com/questions/7370801/how-to-measure-elapsed-time-in-python).
2. Repeat this, varying one of: batch size, input size, other. Plot the results (sorry this isn't a notebook):

4. Any difficulties you encountered here? Why or why not?
I it hard to tell what level of variance is acceptable. As a solution, we compared running with large k for one model and choose an acceptable value.

4: Energy use
----
1. Compute the energy use of each model. You can use the `powertop` tool on RPi and Jetson (must be run as root):

    sudo apt install powertop

and/or the `jtop` tool on Jetson (see installation instructions [here](https://github.com/rbonghi/jetson_stats/)).

Follow the same procedure as you used to compute latency, but this time compute energy: (avg) watts * time. You will likely need to sample power a number of times throughout each inference, and average.

By default, `powertop` takes measurements every 20 seconds. You can change it with the `--time` parameter, which specifies number of seconds and allows for non-integer intervals (0.5 for half a second) e.g. to poll every second for 10 seconds and write to 10 csv files:

    sudo powertop --time=1 --csv=powertop.log --iteration=10

Here is a link to the [`powertop` users guide](https://01.org/sites/default/files/page/powertop_users_guide_201412.pdf) [PDF].

The total data size is 256.

2. Any difficulties you encountered here? Why or why not?

Yes, we found that sometimes the power usage can be very unstable during inference. Then we found out the reason was that we were running multiple jobs at the same time so that our device will allocate powers to different jobs.

5: Discussion
----
1. Analyze the results. Do they support your hypotheses? Why or why not? Did you notice any strange or unexpected behavior? What might be the underlying reasons for that behavior?

From the latency experiment, the batch size vs inference time is nearly linear, which is expected. Also, smaller models like TinyBERT take less inference time than larger model like BERT.
The throughput figure shows that larger batch are more efficient than smaller batches. The reason could be that for batch size greater than 1, the four core machine allows the prediction to run parallelly. With larger batch size, the vacacy time can be reduced, and thus the speed per instance is faster.

For the energy experiment, we found that the power of different models and different batch sizes are almost the same, around 6W on our RPi device. Since we don't have much prior knowledge about the power usage of models on different devices, this is surprising to us and we previously thought that for a larger model or a larger batch size, the power will be higher. Based on the results, we think these models already take the full power usage of RPi even when batch size is 1.

5: Extra
----
A few options:
1. Compute FLOPs for each of your models. If you're using Transformer-based models, you might be able to use or modify the [`flops_counter.py`]() script in this directory. If you're using a CNN-based model, then you might be able to use or modify code in [this project](https://github.com/1adrianb/pytorch-estimate-flops) or similar.

| Model | #GFLOPs |
|------- |--------|
| BERT | 28.4 |
| DistillBERT | 17.2 |
| ALBERT-base | 23.5 |
| Funnel Transformer | 19.3 |
| MobileBERT | 6.6 |
| TinyBERT | 3.7 |

2. Evaluate on different hardware (for example, you might run the same benchmarking on your laptop.) Compare the results to benchmarking on your device(s).

We further evaluate our six models on a single NVIDIA GeForce RTX 2080 Ti GPU.

On the GPU, latency does not increase or incearse sub-linearly with batch size. So larger batch size leads to higher throughput.

3. Use real evaluation data and compare accuracy vs. efficiency. Describe your experimental setup in detail (e.g. did you control for input size? Batch size? Are you reporting average efficiency across all examples in the dev set?) Which model(s) appear to have the best trade-off? Do these results differ from benchmarking with synthetic data? Why or why not?

----
\* There are exceptions to this rule, where it may be important to include data loading in benchmarking, depending on the specific application and expected use cases. For the purposes of this lab, we want to isolate any data loading from the inference time due to model computation.