knitr::opts_chunk$set(echo = TRUE)

# Assignment 1 - Language development in autistic and neurotypical children

## Quick recap
Autism Spectrum Disorder is often related to language impairment. However, this phenomenon has rarely been empirically traced in detail: i) relying on actual naturalistic language production, ii) over extended periods of time.

We therefore videotaped circa 30 kids with ASD and circa 30 comparison kids (matched by linguistic performance at visit 1) for ca. 30 minutes of naturalistic interactions with a parent. We repeated the data collection 6 times per kid, with 4 months between each visit. We transcribed the data and counted:
i) the amount of words that each kid uses in each video. Same for the parent.
ii) the amount of unique words that each kid uses in each video. Same for the parent.
iii) the amount of morphemes per utterance (Mean Length of Utterance) displayed by each child in each video. Same for the parent.

This data is in the file you prepared in the previous class, but you can also find it here:https://www.dropbox.com/s/d6eerv6cl6eksf3/data_clean.csv?dl=0

pacman::p_load(tidyverse, dplyr, brms)
d <- read_csv("CleanedChild_Data.csv")

## The structure of the assignment

We will be spending a few weeks with this assignment. In particular, we will:

Part 1) simulate data in order to better understand the model we need to build, and to better understand how much data we would have to collect to run a meaningful study (precision analysis)

Part 2) analyze our empirical data and interpret the inferential results

Part 3) use your model to predict the linguistic trajectory of new children and assess the performance of the model based on that.

As you work through these parts, you will have to produce a written document (separated from the code) answering the following questions:

Q1 - Briefly describe your simulation process, its goals, and what you have learned from the simulation. Add at least a plot showcasing the results of the simulation. Make a special note on sample size considerations: how much data do you think you will need? what else could you do to increase the precision of your estimates?

Q2 - Briefly describe the empirical data and how they compare to what you learned from the simulation (what can you learn from them?). Briefly describe your model(s) and model quality. Report the findings: how does development differ between autistic and neurotypical children (N.B. remember to report both population and individual level findings)? which additional factors should be included in the model? Add at least one plot showcasing your findings.

Q3 - Given the model(s) from Q2, how well do they predict the data? Discuss both in terms of absolute error in training vs testing; and in terms of characterizing the new kids' language development as typical or in need of support.

Below you can find more detailed instructions for each part of the assignment.

## Part 1 - Simulating data

Before we even think of analyzing the data, we should make sure we understand the problem, and we plan the analysis. To do so, we need to simulate data and analyze the simulated data (where we know the ground truth).

In particular, let's imagine we have n autistic and n neurotypical children. We are simulating their average utterance length (Mean Length of Utterance or MLU) in terms of words, starting at Visit 1 and all the way to Visit 6.
In other words, we need to define a few parameters:
- average MLU for ASD (population mean) at Visit 1 and average individual deviation from that (population standard deviation)
- average MLU for TD (population mean) at Visit 1 and average individual deviation from that (population standard deviation)
- average change in MLU by visit for ASD (population mean) and average individual deviation from that (population standard deviation)
- average change in MLU by visit for TD (population mean) and average individual deviation from that (population standard deviation)
- an error term. Errors could be due to measurement, sampling, all sorts of noise.

Note that this makes a few assumptions: population means are exact values; change by visit is linear (the same between visit 1 and 2 as between visit 5 and 6). This is fine for the exercise. In real life research, you might want to vary the parameter values much more, relax those assumptions and assess how these things impact your inference.

We go through the literature and we settle for some values for these parameters:
- average MLU for ASD and TD: 1.5 (remember the populations are matched for linguistic ability at first visit)
- average individual variability in initial MLU for ASD 0.5; for TD 0.3 (remember ASD tends to be more heterogeneous)
- average change in MLU for ASD: 0.4; for TD 0.6 (ASD is supposed to develop less)
- average individual variability in change for ASD 0.4; for TD 0.2 (remember ASD tends to be more heterogeneous)
- error is identified as 0.2

This would mean that on average the difference between ASD and TD participants is 0 at visit 1, 0.2 at visit 2, 0.4 at visit 3, 0.6 at visit 4, 0.8 at visit 5 and 1 at visit 6.

With these values in mind, simulate data, plot the data (to check everything is alright); and set up an analysis pipeline.
Remember the usual bayesian workflow:
- define the formula
- define the prior
- prior predictive checks
- fit the model
- model quality checks: traceplots, divergences, rhat, effective samples
- model quality checks: posterior predictive checks, prior-posterior update checks
- model comparison

Once the pipeline is in place, loop through different sample sizes to assess how much data you would need to collect. N.B. for inspiration on how to set this up, check the tutorials by Kurz that are linked in the syllabus.

BONUS questions for Part 1: what if the difference between ASD and TD was 0? how big of a sample size would you need? What about different effect sizes, and different error terms?

#Simulate data

# predictor variable is visit, which is a discrete variable (and diagnosis) and outcome variable is MLU which is a continuous variable

hist(rnorm(10000, 1.5, 0.5)) #ASD at first visit

hist(rnorm(10000, 1.5, 0.3)) #neurotypical children (TD) at first visit

SimADS <- tibble(
  Kid = seq(30),
  Intercept = rnorm(30, 1.5, 0.5),
  Slope = rnorm(30, 0.4, 0.4),
  Diagnosis = "ASD"
)
  
SimTD <- tibble(
  Kid = seq(30),
  Intercept = rnorm(30, 1.5, 0.3),
  Slope = rnorm(30, 0.6, 0.2),
  Diagnosis = "TD"
)

SimADS1 <- tibble(
  expand_grid(Kid = seq(30),
              Visit = seq(6))
)

SimTD1 <- tibble(
  expand_grid(Kid = seq(30),
              Visit = seq(6))
)

SimADS <- merge(SimADS,SimADS1)
SimTD <- merge(SimTD, SimTD1)

for (i in seq(nrow(SimADS))){
  SimADS$MLU[i] <- rnorm(1, SimADS$Intercept[i] + SimADS$Slope[i] * (SimADS$Visit[i] - 1))
}

for (i in seq(nrow(SimTD))){
  SimTD$MLU[i] <- rnorm(1, SimTD$Intercept[i] + SimTD$Slope[i] * (SimTD$Visit[i] - 1))
}

SimD <- full_join(SimADS, SimTD)

#Plot data
SimD %>%
  ggplot(aes(x = Visit, y = MLU, color = Diagnosis, group = Kid)) + geom_point() + geom_line(alpha = 0.3)

WIP by Niels :))

#Bayesian workflow - making a model

kfold(MLU_model, MLU_model_1, compare = TRUE, K = 10, save_fits = TRUE)

sum(is.na(MLU_model_1))

waic(MLU_model)
waic(MLU_model_1)

loo_compare(loo(MLU_model), loo(MLU_model_1))

kfold1 <-kfold(MLU_model, folds = "stratified", group = "Diagnosis", K = 5, save_fits= TRUE)
kfold2 <- kfold(MLU_model_1, folds = "stratified", group = "Diagnosis", K = 5, save_fits= TRUE)

rmse<-function(y, yrep) {
  yrep_mean<-colMeans(yrep)
  sqrt(mean((yrep_mean-y)^2))
}

kfp1<-kfold_predict(kfold1)
kfp_test1<-kfold_predict(kfold1)

kfp2<-kfold_predict(kfold2)
kfp_test2<-kfold_predict(kfold2)



rmse_df <- list("MLU_Model_Out" = rmse(kfp_test1$y, kfp_test1$yrep), "MLU_Model_In"=rmse(kfp1$y, kfp1$yrep), "MLU_Model1_Out" =rmse(kfp_test2$y, kfp_test2$yrep), "MLU_Model1_In" = rmse(kfp2$y, kfp2$yrep))


data_frame()

remse(kpf)

rmse_df %>%
  ggplot(aes())

# Part 2 - Strong in the Bayesian ken, you are now ready to analyse the actual data

- Describe your sample (n, age, gender, clinical and cognitive features of the two groups) and critically assess whether the groups (ASD and TD) are balanced. Briefly discuss whether the data is enough given the simulations in part 1.
- Describe linguistic development (in terms of MLU over time) in TD and ASD children (as a function of group). Discuss the difference (if any) between the two groups.
- Describe individual differences in linguistic development: do all kids follow the same path? Are all kids reflected by the general trend for their group?

- Include additional predictors in your model of language development (N.B. not other indexes of child language: types and tokens, that'd be cheating). Identify the best model, by conceptual reasoning, model comparison or a mix. Report the model you choose (and name its competitors, if any) and discuss why it's the best model.

Part 3 - From explanation to prediction

N.B. There are several datasets for this exercise, so pay attention to which one you are using!

1. The (training) dataset from last time (the awesome one you produced :-) ).
2. The (test) datasets on which you can test the models from last time:
* Demographic and clinical data: https://www.dropbox.com/s/ra99bdvm6fzay3g/demo_test.csv?dl=1
* Utterance Length data: https://www.dropbox.com/s/uxtqqzl18nwxowq/LU_test.csv?dl=1
* Word data: https://www.dropbox.com/s/1ces4hv8kh0stov/token_test.csv?dl=1

Relying on the model(s) you trained in part 2 of the exercise, create predictions for the test set and assess how well they do compared to the actual data.

- Discuss the differences in performance of your model in training and testing data. Is the model any good?
- Let's assume you are a speech therapy clinic. You want to assess whether the kids in your test sample will have a typical (like a TD) development, or they will have a worse one, in which case they should get speech therapy support. What do your predictions tell you about that? Which kids would you provide therapy for? Is the model any good?