# Baselines

###### `DistMult`

> The obtained results of DistMult on WN11, WN18, WN18RR, FB13, FB15K, and FB15k-237
> I used the same hyper-parameters for training as SimplE.

---

## Results [:link:][code]

- ***DistMult*** performance on the test set with **no** negative labels

| Dataset | Raw MRR | Filtered MRR | Best Epoch|
| ----------|:--------|:-------------|:----------|
| WN18 | 0.537 | 0.785 | 1000 |
| WN18RR | 0.235 | 0.380 | 1000 |
| FB15K | 0.239 | 0.691 | 1000 |
| FB15k-237 | 0.075 | 0.160 | 100 |

[code]: https://bitbucket.org/rbcmllab/negsample/src/distMult/

- ***DistMult*** performance on the test set **with** negative labels

| Dataset | Neg Log Likelihood | Brier Score | Best Epoch |
| ----------|:-------------------|:------------|:-----------|
| WN11 | 3.459 | 0.484 | 1000 |
| FB13 | 2.956 | 0.486 | 1000 |
| YAGO39K | ? | ? | ? |

We get much better results (NLL **smaller than one**) if we change the hyper-params of the model i.e. the learning rate, the embedding size, etc.
One other thing to note is that when I train the regularized version with the same learning rate and embedding size it performs worse.

- ***SimplE*** performance on datasets **with** negative labels

| Dataset | Neg Log Likelihood | Brier Score | Best Epoch |
| ----------|:-------------------|:------------|:-----------|
| WN11 | 3.380 | 0.490 | 1000 |
| FB13 | 3.256 | 0.493 | 200 |

- ***SimplE*** performance on the test set with **no** negative labels

| Dataset | Raw MRR | Filtered MRR | Best Epoch|
| ----------|:--------|:-------------|:----------|
| WN18 | 0.582 | 0.941 | 1000 |
| WN18RR | 0.230 | 0.378 | 1000 |
| FB15K | 0.234 | 0.722 | 500 |
| FB15k-237 | 0.068 | 0.159 | 1000 |

## References

- ***SimplE*** (based on the ComplEx paper)

| Dataset | Raw MRR | Filtered MRR |
| ----------|:--------|:-------------|
| WN18 | 0.532 | 0.822 |
| FB15K | 0.242 | 0.654 |

- ***Calibration Paper*** (reported **uncalibrated** results)

| Dataset | Neg Log Likelihood | Brier Score |
| ----------|:-------------------|:------------|
| WN11 | 5.625 | 0.488 |
| FB13 | 2.177 | 0.473 |

- Calibrated results (ground truth: Platt and Iso) from the calibration paper

| Dataset | Neg Log Likelihood | Brier Score |
| ----------|:-------------------|:------------|
| WN11 | 0.618, 0.604 | 0.213, 0.208|
| FB13 | 0.533, 0.518 | 0.178, 0.170|