---
# System prepended metadata

title: HW4 Report

---

# HW4 Report

### What was done

We performed a NMT by using a pretrained model mad by Facebook AI Research. The model showed great results achieving state-of-the-art BLEU score on Ru-En translation during WMT2019 contest. [1]

### How it was done

Facebook FAIR system is a pure sentence level system, it is an ensemble of 3 Big Transformer models with FFN layers of size 8192. 
Trained on the mix of bitext and back-translated newscrawl data, oversampling was used to keep the effective ratio of bitext and back-translated data the same. Sampling from an ensemble of 3 models trained on bitext only was used to generate back-translations. The models were fine-tuned on in-domain data and a final noisy channel reranking was applied. All the training data (bitext and monolingual) was cleaned using langid filtering. [2]


### Results

On a small dataset we achieved 51.00 BLEU score
On a large dataset we achieved 52.57 BLEU score.

### Additional information

We also tried to implement our own transformer but with little success.
All the materials (submitted solution file is `NMT.ipynb`) may be found at [project github page](https://github.com/BeardedWhale/NeuralMachineTranslation2020)

### References

[1] [Facebook FAIR’s WMT19 News Translation Task Submission](https://arxiv.org/pdf/1907.06616.pdf)

[2] [Findings of the 2019 Conference on Machine Translation (WMT19)](https://www.statmt.org/wmt19/pdf/53/WMT01.pdf)
