---
# System prepended metadata

title: Baselines

---

# Baselines

###### `DistMult`

> The obtained results of DistMult on WN11, WN18, WN18RR, FB13, FB15K, and FB15k-237
> I used the same hyper-parameters for training as SimplE.


---

## Results [:link:][code]

- ***DistMult*** performance on the test set with **no** negative labels

| Dataset   | Raw MRR | Filtered MRR | Best Epoch|
| ----------|:--------|:-------------|:----------|
| WN18      |  0.537  |    0.785     |    1000   |
| WN18RR    |  0.235  |    0.380     |    1000   |
| FB15K     |  0.239  |    0.691     |    1000   |
| FB15k-237 |  0.075  |    0.160     |    100    | 

[code]: https://bitbucket.org/rbcmllab/negsample/src/distMult/

- ***DistMult*** performance on the test set **with** negative labels

| Dataset   | Neg Log Likelihood | Brier Score | Best Epoch |
| ----------|:-------------------|:------------|:-----------|
| WN11      |  3.459             |    0.484    |    1000    |
| FB13      |  2.956             |    0.486    |    1000    |
| YAGO39K   |  ?                 |    ?        |    ?       |

We get much better results (NLL **smaller than one**) if we change the hyper-params of the model i.e. the learning rate, the embedding size, etc.
One other thing to note is that when I train the regularized version with the same learning rate and embedding size it performs worse.

- ***SimplE*** performance on datasets **with** negative labels

| Dataset   | Neg Log Likelihood | Brier Score | Best Epoch |
| ----------|:-------------------|:------------|:-----------|
| WN11      |  3.380             |    0.490    |    1000    |
| FB13      |  3.256             |    0.493    |    200     |

- ***SimplE*** performance on the test set with **no** negative labels

| Dataset   | Raw MRR | Filtered MRR | Best Epoch|
| ----------|:--------|:-------------|:----------|
| WN18      |  0.582  |    0.941     |    1000   |
| WN18RR    |  0.230  |    0.378     |    1000   |
| FB15K     |  0.234  |    0.722     |    500    |
| FB15k-237 |  0.068  |    0.159     |    1000   | 

## References

- ***SimplE*** (based on the ComplEx paper)

| Dataset   | Raw MRR | Filtered MRR |
| ----------|:--------|:-------------|
| WN18      |  0.532  |    0.822     |
| FB15K     |  0.242  |    0.654     |

- ***Calibration Paper*** (reported **uncalibrated** results)

| Dataset   | Neg Log Likelihood | Brier Score |
| ----------|:-------------------|:------------|
| WN11      |  5.625             |    0.488    |
| FB13      |  2.177             |    0.473    |

- Calibrated results (ground truth: Platt and Iso) from the calibration paper

| Dataset   | Neg Log Likelihood | Brier Score |
| ----------|:-------------------|:------------|
| WN11      |  0.618, 0.604      | 0.213, 0.208|
| FB13      |  0.533, 0.518      | 0.178, 0.170|

