# Coursework B Group 27
```{r}
library(RCurl)
library(bitops)
library(plyr)
library(stringr)
library(NLP)
library(tm)
library(RColorBrewer)
library(wordcloud)
library(reshape)
library(pander)
library(ez)
library(car)
library(ggplot2)
library(nlme)
library(ggpubr)
library(dplyr)
```
# Results
In this section, we will discuss the results of the analysis of data collected on the performance of the model types, given dataset combinations.
This analysis will take place in context of our two main research questions.
As a first exploration step, we divide the models by type, and compare their scores per test dataset.
```{r, ECHO=FALSE}
data <- import("data.csv")
transferE <- data.frame(model=c("B1","B2","B3","M1","M2","M3","MF","MN","S"), transfer=c("Baseline", "Baseline", "Baseline", "Transfer", "Transfer", "Transfer", "Transfer", "Transfer", "Transfer"))
data_frame_temp <- merge(data, transferE, by="model")
typeED <- data.frame(TeD=c("TeD1","TeD2","TeD3","TeD4","TeD5","TeD6","TeD7"), type=c("Classification","Classification","Classification","Classification","Recommendation","Regression","Regression"))
data_frame <- merge(data_frame_temp, typeED, by="TeD")
ggplot(data_frame, aes(TeD, score)) + geom_boxplot(notch = FALSE) + facet_wrap(~transfer)
```
As we can see in the figure, the means for both types of models differ starkly, with the transfer learning models having higher means for most test datasets.
In order to gain further insights into relations within the data, we consider a model using the insights gained from the figure above: the score differs between model types, for test datasets.
```{r}
model_0 <- lm(score ~ transfer, data_frame)
summary(model_0)
model_1 <- lm(score ~ transfer*TeD, data_frame)
model_1_type <- lm(score ~ transfer*type, data_frame)
summary(model_1)
summary(model_1_type)
```
Initial models allow us to estimate the influence of transfer learning to be approximately $19.9\pm4.1$ percent points with $p=5.9e-14$, with the model itself having $R^2\approx0.92$ as opposed to the null-model's $R^2\approx0.00$. Splitting it up by testing dataset types (making for a less accurate model with $R^2\approx0.66$), we can see that improvements are approximately $9\pm 4$, $45 \pm 13$, and $47 \pm 10$ percent points for classification, recommendation, and regression respectively. It should be noted that the results gathered from the type-comparing model only serve as indications, as occurances of transfer learning models within the task type sets is not equal.
As becomes evident from the low $R^2$ of the type-comparing model, there seem to be large differences in the performance of models for test datasets within types. In order to gather more informative insights into the true difference transfer learning makes for the different types of testing datasets, we need to find the cause of these unexplained variations.
As the unexpected variation seems to occur between models, we investigate differences between models which are not caught by the distinction whether or not they use transfer learning.
Upon inspection of the data, it soon becomes clear that various numbers of training datasets were used for various models. In addition, the fact that a given specific training dataset is included or not in the training of a model might also cause variation -- for example: if the training set is particularly helpful for training for the type of problem examined. The former (number of training datasets) will be analysed immediately, whilst more information on the latter (training- and test dataset interactions) can be found in the analysis in light of our second research question. An additional cause for variation, specifically for the transfer learning models, is the amount of retraining done for the test dataset examined for.
```{r}
sample_sum <- data_frame %>%
group_by(training_dataset_count, model) %>%
summarize(mean = mean(score)) %>%
ungroup()
ggplot(sample_sum, aes(x = training_dataset_count, group=model, color = model, fill = model)) +
geom_line(aes(y= mean)) + geom_point(aes(y=mean))
```
As can be seen in the figure above, the number of training dataset seems to influence the score, hence it will as a factor in the expanded model.
```{r}
sample_sum <- subset(data_frame,transfer=="Transfer" & model != "S") %>%
group_by(refinement, model) %>%
summarize(mean = mean(score)) %>%
ungroup()
ggplot(sample_sum, aes(x = refinement, group=1, fill = model)) +
geom_line(aes(y= mean)) + geom_point(aes(y=mean))
```
For transfer learning models, the amount of refinement (fine tuning of the model) seems to also influence the score, hence it will also be included as a factor.
```{r}
data_frame$training_dataset_count <- factor(data_frame$TrD1+data_frame$TrD2+data_frame$TrD3+data_frame$TrD4+data_frame$TrD5+data_frame$TrD6+data_frame$TrD7+data_frame$TrD8,levels=c(0:8))
refinementE <- data.frame(model=c("B1","B2","B3","M1","M2","M3","MF","MN","S"), refinement=c(0,0,0,1,2,3,4,0,0))
data_frame <- merge(data_frame, refinementE, by="model")
data_frame$refinement <- factor(data_frame$refinement, c(0:4))
model_2 <- lm(score ~ transfer+TeD+training_dataset_count+refinement+model, data_frame)
model_2_type <- lm(score ~ transfer+type+training_dataset_count+refinement+model, data_frame)
summary(model_2)
summary(model_2_type)
confint(model_2, level = 0.99)
```
Taking the number of training datasets used and the amount of retraining into account, whilst ignoring the influence of individual training datasets for now, we are now able to construct a model which explains the relation between these factors, transfer learning, test dataset, to reach given scores. This new model has a slightly higher $R^2\approx0.94$, and estimates the influence of transfer learning (now excluding the influence of both training set count and retraining/refining) on the score to be approximately $10.6\pm1.4$ percent points, with $p=9.62e-14$. The $99\%$ confidence interfal for improvement though transfer learning in this model is given by $[0.06972365,0.143015784]$. Hence, with $\alpha=0.01$, transfer learning can be said to significantly improve upon model scores, therefore achieving better performance (on average). Once again, taking the types of task into account instead of the test dataset, we see imrovements of $7.8\pm4.2$, $44.8\pm12.8$, and $47.0\pm9.9$ percent points for the task types respectively, none of which are significant, and are taken from a model with $R^2\approx0.68$.
Given the fact that the $R^2$ of the expanded model including the most obvious factors except train - and test dataset interactions is still unexpectedly low. This might suggest that test datasets perform quite differently when provided with a model trained on the same training dataset. In other words, there might be interaction effects between the test and training data, which ties in with our second research question.
```{r}
inclusion_dataframe = subset(melt(data_frame, measure.vars = c("TrD1","TrD2","TrD3","TrD4","TrD5","TrD6","TrD7","TrD8")), value==1)
ggplot(inclusion_dataframe, aes(x=variable,y=score)) + geom_boxplot(notch = FALSE)+ facet_wrap(~TeD)
```
An initial exploration of the data reveals that, indeed, test datasets respond differently to models trained on various training datasets. A particularly extreme example can be found for the interaction between test dataset 1 and training dataset 2. An initial theory could be that training dataset 2 is particularly suited to training models for classification problems. This can, however, quickly be disregarded due to the fact the models trained with dataset 2 do not perform as well for the other classification problems. This suggests that the data in both sets, thus the problems, are rather similar for test dataset 1 and training dataset 2.
Additionally, it can be observed that for the other combinations, even though they are more similar, means and variances do seem to differ for training datasets.
In order to model these effects, direct interactions between test and training datasets are included.
```{r}
ml <- lm(score ~ transfer + training_dataset_count + refinement + type * TeD * (TrD1+TrD2+TrD3+TrD4+TrD5+TrD6+TrD7+TrD8), data_frame)
summary(ml)
```
This yields a model with $R^2\approx0.97$, which shows that the influence of transfer learning itself, thus excluding the effects of the number of datasets, the test dataset and its interactions with training datasets, and the amount of refinement in transfer learning datasets, is approximately $18.4\pm3.6$ percent points (with $p=4.44e-07$) (RQ1).
```{r}
ml <- lm(score ~ training_dataset_count + type * TeD * (model+TrD1+TrD2+TrD3+TrD4+TrD5+TrD6+TrD7+TrD8), subset(data_frame), transfer=="Transfer")
summary(ml)
```
Lastly, in order to compare the effect of individual training datasets on the final model performance alone, we construct a model in which the model itself, the training datasets, test datasets, the task and its interactions with the test datasets and the individual training datasets, and the number of training data sets are accounted for (refinement is already part of the model and is therefore omittable).
This yields a model (transfer learning only) with $R^2\approx0.98$. Looking at the influence of the inclusion of a given training dataset ignoring the intersects and the influence of the number of training datasets, we gather:
| Training Dataset | Classification Score Improvement | Recommendation Score Improvement | Regression Score Improvement |
| -------- | -------- | -------- | ------- |
| Training dataset 1 | $-0.052\pm0.006$ with $p<2e-16$ | $0.029\pm 0.006$ with $p=3.37e-07$ | $0.011\pm0.006$ with $p=0.064$ *(!)* |
| Training dataset 2 | $+0.167\pm0.006$ with $p< 2e-16$ | $-0.189\pm0.006$ with $p< 2e-16$ | $-0.142\pm0.006$ with $p<2e-16$ |
| Training dataset 3 | $-0.014\pm0.006$ with $p=0.013$ | $-0.004\pm0.006$ with $p=0.536$ *(!)* | $-0.005\pm0.006$ *(!)* with $p=0.349$ *(!)*|
| Training dataset 4 | $-0.023\pm0.006$ with $p=0.000$ | $0.005\pm 0.006$ *(!)* with $p=0.367$ *(!)* | $-0.020\pm0.006$ with $p=0.001$|
| Training dataset 5 | $+0.001\pm0.006$ *(!)* with $p=0.886$ *(!)* | $-0.015\pm0.006$ with $p=0.0096$ | $0.013\pm0.006$ with $p=0.033$ |
| Training dataset 6 | $-0.012\pm0.006$ with $p=0.029$ | $-0.005\pm0.006$ *(!)* with $p=0.342$ *(!)* | $0.001\pm0.006$ *(!)* with $p=0.916$ *(!)* |
| Training dataset 7 | $-0.007\pm0.006$ with $p=0.208908$ *(!)* | $-0.009\pm0.006$ with $p=0.113$ *(!)* | $0.017\pm0.006$ with $p=0.004$ |
| Training dataset 8 | - | $-0.013\pm0.006$ with $p=0.021$ | $0.014\pm0.006$ with $p=0.017$|
Excluding results for which their $95\%$ confidence interval includes zero, and excluding insignificant results ($\alpha=0.05$), yields (RQ2):
Inclusion of training dataset 1 has a small net negative effect ($\approx-5.2\pm0.6$ percent points) on classification tasks, and a small net positive effect ($\approx2.9\pm0.6$ percent points) on recommendation tasks.
Inclusion of training dataset 2 has a large positive effect ($\approx16.7\pm0.6$ percent points) on classification tasks, and a large net negative effect ($\approx-18.9\pm0.6$ and $\approx-14.2\pm0.6$ percent points respectively) on recommendation and regression scores.
Inclusion of training dataset 3 has a small net negative effect ($\approx-1.4\pm0.6$ percent points) on classification tasks.
Inclusion of dataset 4 has a small net negative effect ($\approx-2.3\pm0.6$ and $\approx-2.0\pm0.6$ percent points respectively) on classification and regression tasks.
Inclusion of dataset 5 has s small net negative effect ($\approx-1.5\pm0.6$ percent points) on recommendation tasks, and a small net positive effect ($\approx1.3\pm0.6$ percent points) on regression tasks.
Inclusion of dataset 6 has a small net negative effect ($\approx-0.7\pm0.6$ percent points) on classification tasks.
Inclusion of training dataset 7 has a small net positive effect ($\approx1.7\pm0.6$ percent points) on regression tasks.
Inclusion of training dataset 8 has a small net negative effect ($\approx-1.3\pm0.6$ percent points) on recommendation tasks and a small net positive effect ($\approx1.4\pm0.6$ percent points) on regression tasks.
It should be noted that we find some rather unexpectedly large effects within these results, particularly for the negative effect of training dataset 2 on recommendation problems. Based on our intuitions gathered from the figure depicting test/training dataset combinations, we would expect effects to be small, as no score seems to reach above $12.5$ percent points. An effect of approximately $-18$ percent points for inclusion (Training dataset 2 & Recommendation) therefore seems strange, yet we have not been able to deduce a cause for this phenomenon.
# OLD
## Question1
```{r}
library(rio)
library(pracma)
data <- import("data.csv")
transferE <- data.frame(model=c("B1","B2","B3","M1","M2","M3","MF","MN","S"), transfer=c("Baseline", "Baseline", "Baseline", "Transfer", "Transfer", "Transfer", "Transfer", "Transfer", "Transfer"))
data_frame <- merge(data, transferE, by="model")
library(ggplot2)
ggplot(data_frame, aes(TeD, score)) + geom_boxplot(notch = FALSE)
ggplot(data_frame, aes(TeD, score)) + geom_boxplot(notch = FALSE) + facet_wrap(~transfer)
```
The difference between model types is clear.
```{r}
t.test(score~transfer, data_frame)
```
From t-test, we can conclude that there is a significant improvement(0.118) in score for transfer learning than not using it($t(20)=-2.26,p=.03$).
```{r}
m <- lm(score~transfer, data_frame)
summary(m)
confint(m, "transferTransfer", level = 0.95)
```
The model yields similar results to the t-test, confirming, with significance ($R^2=0.001,F(2,994)=4.719,p=.03$), that the transfer model results are better. The main problem is that the model only explains 1 promille of the variation.
As we can see in the plots in the beginning, scores are variant in different test datasets. So we fit the interaction of transfer learning and test dataset in the linear model.
```{r}
m <- lm(score~transfer*TeD, data_frame)
summary(m)
#confint(m)
```
```{r}
ggplot(data_frame, aes(TeD, score)) + geom_boxplot(notch = FALSE)
ggplot(data_frame, aes(TeD, score)) + geom_boxplot(notch = FALSE) + facet_wrap(~model)
ggplot(data_frame, aes(TeD, score)) + geom_boxplot(notch = FALSE) + facet_wrap(~TrD1)
ggplot(data_frame, aes(TeD, score)) + geom_boxplot(notch = FALSE) + facet_wrap(~TrD2)
ggplot(data_frame, aes(TeD, score)) + geom_boxplot(notch = FALSE) + facet_wrap(~TrD3)
ggplot(data_frame, aes(TeD, score)) + geom_boxplot(notch = FALSE) + facet_wrap(~TrD4)
ggplot(data_frame, aes(TeD, score)) + geom_boxplot(notch = FALSE) + facet_wrap(~TrD5)
ggplot(data_frame, aes(TeD, score)) + geom_boxplot(notch = FALSE) + facet_wrap(~TrD6)
ggplot(data_frame, aes(TeD, score)) + geom_boxplot(notch = FALSE) + facet_wrap(~TrD7)
ggplot(data_frame, aes(TeD, score)) + geom_boxplot(notch = FALSE) + facet_wrap(~TrD8)
```
The difference between model types is clear. In addition, there is a noticeable influence of all training datasets.
Fitting a model through not only whether or not it is transfer learning, but also the training datasets. And according to the ANOVA below, the interaction between test and train datasets also affects.
```{r}
m0 <- lm(score ~transfer*TeD + TrD1 + TrD2 + TrD3 + TrD4 + TrD5 + TrD6 + TrD7 + TrD8, data_frame)
m1 <- lm(score ~transfer*TeD + (TrD1 + TrD2 + TrD3 + TrD4 + TrD5 + TrD6 + TrD7 + TrD8)*TeD, data_frame)
anova(m0,m1)
summary(m1)
```
#TODO consider about the score differences on each test datasets?
Then we use estimated marginal means to interpret the results. Using transfer learning can get 0.06 score higher than not using it.
```{r}
library(emmeans)
#as.data.frame(ref_grid(m1))
#emmeans(m1, ~transfer)
pairs(emmeans(m1, ~transfer))
```
## Question2
We test the model by using estimated marginal means whether including the training dataset or not. It is shown that there are significant differences on including training dataset 1,2,5,6,7 and 8.
```{r}
pairs(emmeans(m1, ~TrD1))
pairs(emmeans(m1, ~TrD2))
pairs(emmeans(m1, ~TrD3))
pairs(emmeans(m1, ~TrD4))
pairs(emmeans(m1, ~TrD5))
pairs(emmeans(m1, ~TrD6))
pairs(emmeans(m1, ~TrD7))
pairs(emmeans(m1, ~TrD8))
```
Google Drive
Gist
Clipboard
Sign in with Wallet
