# A Neurocomputational Model of the N400 and the P600 in Language Processing
#### Harm Brouwer, Matthew W. Crocker, Noortje J. Venhuizen, John C. J. Hoeks
##### Cognitive Science 41 (2017, Suppl. 6) 1318–1352

---
# Introduction

**N400 and P600**:
- **N400** and **P600** are two key ERP components in language comprehension.

:::spoiler **N400**:
- Negative deflection peaking around 400 ms.
- Sensitive to semantic anomalies.
- Example: “He spread his warm bread with socks,” relative to “butter” (Kutas & Hillyard, 1980).
:::

:::spoiler **P600**:
- Positive deflection peaking around 600 ms.
- Often associated with syntactic processing.
- Example: “The spoilt child throw . . .,” relative to “throws” (Hagoort, Brown, & Groothusen, 1993).
:::

---

## Semantic P600 Effect

:::danger
- Some grammatically correct but semantically odd sentences (e.g., “The javelin has the athletes thrown”) do **not** elicit the expected N400.
- Instead, they trigger a **P600** response.
- This phenomenon is known as the **Semantic Illusion** or **Semantic P600 Effect**.
:::

---

## Present Paper

:::success
- Proposes a **single-stream model** – the **Retrieval–Integration (RI) account** – to explain Semantic P600 effects.
- Implements the model as a neurocomputational (neural network) model.
- Simulates emergent N400 and P600 amplitudes during language processing.
:::

---

## Recap: Challenge to Traditional ERP Views

:::spoiler **Semantic Illusion/P600 Effect:**
- Semantically odd, syntactically correct sentences elicit P600, *not* N400.
:::

:::spoiler **Multi-Stream Models:**
- Assume separate/parallel semantic and syntactic processing streams.
- Semantic stream finds *a* meaning, avoiding N400.
- Conflict with syntactic stream causes P600.
:::

:::spoiler **Critique:**
- These models fail to explain:
- Biphasic N400/P600 effects
- Isolated P600 effects in discourse.

:::

---

# Retrieval–Integration (RI) Account

:::info
- **N400:** Reflects the **retrieval** of a word’s meaning.
- If context pre-activates lexical-semantic features, retrieval is easier -> **smaller N400**.
- **P600:** Reflects the integration of the retrieved meaning into the unfolding sentence interpretation.
- Difficult integration leads to a **larger P600**.
- **Key Idea:** Language processing occurs in N400/P600 (**RI**) cycles.
:::

---

# Neurocomputational Model Overview

- Based on an extended [**Simple Recurrent Network (SRN)**](https://hackmd.io/@Fd6wOjYES3OJ9NKSpvQ8wQ/SyH7lnsTJe) .
- Processes language in **RI cycles**:
- **Retrieval:** Maps (word form, utterance context) → word meaning.
- **Integration:** Maps (word meaning, utterance context) → updated utterance representation.

---
*Figure: a schematic overview of the model*

---

---

## Model Architecture

:::spoiler **Two Core Modules:**
- **Retrieval :** Maps word forms to word meaning (?).
- **Integration :** Combines retrieved meanings into a utterence representation.
:::

:::spoiler **Key Layers:**
- **INPUT:** Represents the current word (localist one-hot encoding).
- **RETRIEVAL:** Hidden layer for semantic retrieval.
- **RETRIEVAL_OUTPUT:** 100-dimensional binary word meaning (via COALS).
- **INTEGRATION:** Hidden layer for integrating word meaning.
- **INTEGRATION_OUTPUT:** 300-dimensional utterance representation (split into agent, action, patient).
- **INTEGRATION_CONTEXT:** Copies previous INTEGRATION state to provide context.
:::

---

---

### More Details About Neuron Network
[How the neurocomputational model actually works](/s-3GobRNT2aB5u7kOnwpxw)

---

## Representations

:::spoiler **Word Form Representations (Input layer):**
- 35-dimensional one-hot vectors (20 nouns + 10 verbs + 2 auxiliary verbs + 2 determiners + 1 preposition = 35 words)
:::

:::spoiler **Word Meaning Representations (Retrieval output):**
- 100-dimensional binary vectors (derived via COALS from Dutch newspaper texts).
:::

:::spoiler **Utterance Representations:**
- 300-dimensional vectors (three slots for agent, action, and patient).
:::

---

---
## (De)constructing the Integration Module:

:::spoiler **Objective:**
- Combine retrieved word meaning with prior context to update the utterance representation.
:::

:::spoiler **Integration Module Structure:**
- SRN with layers: **RETRIEVAL_OUTPUT** (input), **INTEGRATION** (hidden), **INTEGRATION_OUTPUT** (output), and **INTEGRATION_CONTEXT** (context).
:::

:::spoiler **Function:**
- `integrate(word meaning, utterance context) → utterance representation`
:::

 graph LR
     B(RETRIEVAL_OUTPUT);
    D[INTEGRATION_CONTEXT] -- "utterance context" --> E(INTEGRATION);
    B -- "word meaning" --> E;
    E --> F(INTEGRATION_OUTPUT);
    E -."Copy".-> D;
    F --> G[Utterance Representation];
    subgraph "Integration Module"
        F;
        E;
        B;
    
    end
    
    style B fill:#f9f,stroke:#333,stroke-width:2px;
    style E fill:#ccf,stroke:#333,stroke-width:2px;
    style F fill:#f9c,stroke:#333,stroke-width:2px;
    style D fill:#efe,stroke:#333,stroke-width:2px;
    style G fill:#cfc,stroke:#333,stroke-width:2px;

---

## Training the Integration Module

* :::spoiler **Training Objective:**
To enable the module to map word meanings within a context to an overall utterance representation.
:::
* :::spoiler **Training Data:**

- **Sentence Templates:**
- **Active:** `de [AGENT] heeft het/de [PATIENT] [ACTION]`
- **Passive:** `het/de [PATIENT] werd door de [AGENT] [ACTION]`
- **Data Composition:**
- 16,000 training items per simulation (50% active, 50% passive).
- Stereotypical combinations occur more frequently than non-stereotypical ones.
* :::spoiler **Training Procedure:**
* Bounded gradient descent
* The model updates its weights after processing 100 items to minimize mean-squared error (MSE).
* The learning rate is gradually reduced during training.
* A "zero error radius" is used to prevent minor errors from affecting the learning process.

* :::spoiler **Evaluation:**
* The model's comprehension is evaluated using a **cosine similarity** to compare the output vector to target vectors.
* Performance is measured by checking if the output is more similar to the correct target than to any other.
:::success
The **Integration module** achieves 100% correct comprehension in testing.
:::
---
## (De)constructing the Retrieval Module

- **Goal:**
To create a Retrieval Module that can map a word's form to its meaning, taking into account the surrounding context
- **Retrieval Module Structure:**
SRN with layers: **INPUT** (input), **RETRIEVAL** (hidden), **RETRIEVAL_OUTPUT** (output), and **INTEGRATION_CONTEXT** (context_input).
- **Function:**
`retrieve(word form, utterance context) → word meaning`

  graph LR
    A[INPUT] -->|word form| B[RETRIEVAL]
    D[INTEGRATION_CONTEXT]
    B --> C[RETRIEVAL_OUTPUT]
    D -->|utterence context| B
    E[utterence representation]
    
    subgraph G[Overall model]
        subgraph "Retrieval Module"
            A
            B
            C
            D
        end
        
        subgraph "Integration Module (Fixed)"
            F[INTEGRATION]
        end
    end
    
    C -->|Activated word meaning| F
    F --> E
    
    style A fill:#cff,stroke:#333,stroke-width:2px;
    style G color:#f66, padding-left: 5    
    style B fill:#f9f,stroke:#333,stroke-width:2px;
    style C fill:#ff9,stroke:#333,stroke-width:2px;
    style D fill:#cff,stroke:#333,stroke-width:2px;
    style E fill:#fdd,stroke:#333,stroke-width:2px;
    style F fill:#ccf,stroke:#333,stroke-width:2px;

---
## **Training Procedure:**

:::spoiler **Strategy**:
- Train the Retrieval module as part of the overall network by **freezing** Integration weights, forcing context-sensitive retrieval.
:::

:::spoiler **Why Not a Simple Approach?**:
- If the Retrieval Module is trained independently, the context becomes just noise, and the module learns to ignore it.
:::

:::spoiler **Deterministic Integration Module**:
- After the Integration Module is trained, its weights are "frozen." This makes the Integration Module behave deterministically.
:::

::::spoiler **Outcome**
:::success
Achieves 100% comprehension.
Word meanings from the Retrieval Module align with Integration Module representations.
High values (cosmodel1 = .951, cosmodel2 = .961) confirm the model's effectiveness.
:::
::::

---
>[!Tip]Congratulations! we have arrived at the full model :3

>[!Tip]Take a break:coffee: :+1:
---

## Processing in the model

:::spoiler **Model's Sentence Processing**
- **Word-by-word processing** using [Hoeks et al. (2004)](#cite:a8ca46b171467ceb2d7652fbfb67fe701ad86092) examples.
- **Example (Hoeks et al., 2004):**
- **Control (Passive):** “De maaltijd werd door de kok bereid”
(The meal was prepared by the cook)
- **Reversal (Active):** “De maaltijd heeft de kok bereid”
(The meal has the cook prepared)
- **Mismatch (Passive):** “De maaltijd werd door de kok gezongen”
(The meal was by the cook sung)
- **Mismatch (Active):** “De maaltijd heeft de kok gezongen”
(The meal has the cook sung)
- Anticipates sentence meaning after the first noun.
:::

:::spoiler **Active vs. Passive**
- **Passive:** Confirms predictions.(Since the first two words are "the meal")
- **Active:** Revises interpretation.
- Differentially anticipates active/passive meanings by the **auxiliary verb**.
:::

:::spoiler **Layer Activity & Context**
- **Integration** layer's internal representation differs between active/passive.
- Contextual inputs to **Retrieval** & **Integration** layers also differ.
- Cosine similarity highlights activation differences (e.g., "kok" differs with cos = 0.569).
:::

:::spoiler **Context Modulation**
- Activity patterns in **Retrieval** & **Integration** layers are context-dependent.
- Even same final word shows different activations in active vs. passive.
- Context effect NOT reflected in **RETRIEVAL_OUTPUT** layer. (**Interesting !**)
:::

:::spoiler **Different Endings**
- Varying sentence-final words modulate **Retrieval** & **Integration** layer activations.
:::

:::spoiler **Layer Functionality**
- **Retrieval** and **Integration** layers implement *retrieve* and *integrate* functions.
:::

<font color="#289699">**kok(cook)**</font>
<font color="#75C7CB">**maaltijd(meal)**</font>
<font color="#E82C88">**bereid(prepared)**</font>
<font color="#F29DC4">**gezongen(sung)**</font>
<font color="#F49728">**maaltijd(meal)**</font>
<font color="#FECE60">**kok(cook)**</font>


---
## Linking Hypotheses: RI Account & Neurocomputational Model

:::spoiler **N400 - Lexical Retrieval Effort**
- **Core:** N400 amplitude indexes **processing** load to activate word's conceptual knowledge, not mismatch itself
- **Mechanism:** Change in semantic memory state due to new word. High amplitude = unexpected word = large state change.
- **Model Layer:** RETRIEVAL layer captures this retrieval process.
- **Formula:** $$N400 = 1 - cos(retrieval_t, retrieval_{t-1}) $$
:::

:::spoiler **P600 - Utterance Integration Effort**
- **Core:** P600 amplitude reflects effort to construct/update utterance interpretation.
- **Mechanism:** Reflects difficulty of integrating word meaning into current context.
- **Model Layer:** INTEGRATION layer handles meaning integration.
- **Formula:** $$ P600 = 1 - cos(integration_t, integration_{t-1}) $$
:::

:::info **Key Takeaways**
- Both N400 & P600 are emergent properties of network activity changes.
- Cosine dissimilarity provides a quantifiable link to ERP amplitudes.
- RI account explains both semantic and syntactic effects within a single framework.
:::

---

# Simulation Results: N400 & P600 Effects

:::spoiler **Goal**
- The model aimed to simulate human brain responses (N400 and P600) during language comprehension, especially **Semantic P600** and **biphasic N400/P600** effects.
:::

:::spoiler **Method**
- The researchers generated two set of 40 sentences, each set included these four types of sentences:
- **Control (Passive)**、**Reversal (Active)**、**Mismatch (Passive)**、**Mismatch (Acive)**
- The model processed these sentences, and researchers measured its **N400** and **P600** responses. (cosine dissimilarities)
:::

:::spoiler **Results**
- :::spoiler **N400:**
- The model showed N400 effects for **Mismatch** sentences.
- No N400 for **Semantic illusion** sentences, matching Hoeks et al. (2004)

- :::spoiler **P600:**
- The model showed **P600 effects** for all anomalous sentences.
- Ordering of P600 effects slightly differed from human data (possible N400/P600 overlap).

:::

:::success **Conclusion**
- The model successfully captured key aspects of human **sentence processing** related to **semantic anomalies** and **brain responses**.
:::

---

---

# Discussion

## **Neurocomputational Model of N400 and P600**

:::spoiler **Overview**
- This model supports the **Retrieval-Integration (RI) account** of N400 and P600 in language processing.
- Captures **traditional N400/P600 effects** and **Semantic P600** effects.
- Demonstrates that **N400 reflects context-driven retrieval**, while **P600 represents integration difficulty**.
:::

:::spoiler **Key Findings**
- **N400 effects** are **context-dependent** and arise from processing **critical words**, not pre-critical material.
- Confirms that a **single-stream model** can explain both **N400 and P600** without separate semantic pathways.
- The model uniquely accounts for **sentence-level P600 effects**, which previous models did not address.
:::

---

## **Context Sensitivity, Model Comparisons, and Future Directions**

:::spoiler **N400 and Context Sensitivity**
- **Context primes words**, influencing **N400 amplitude**.
- **Role-reversed sentences** show **increased N400** due to unexpected thematic roles.
:::

:::spoiler **Comparison with Other Models**
- Prior models focus on either **N400** (Laszlo & Plaut, Rabovsky & McRae) or **P600** (Crocker et al.).
- This model **integrates both effects**, making it **more comprehensive**.
:::

:::spoiler **Future Directions**
- Expanding the model to **syntactic violations** (e.g., **agreement errors, garden-path sentences**).
- Developing **richer representations** for **pragmatic P600 effects**.
- Combining the P600 estimate at **timestep t** with the N400 estimate at **timestep t + 1**.
:::

---
## **Cortical Implementation of the Retrieval-Integration Model**

:::spoiler **Key Components & Brain Areas**
- **Retrieval Module**: Left posterior middle temporal gyrus (**lpMTG, BA 21**)
- Mediates **lexical retrieval**
- Generates the **N400** component
- **Integration Module**: Left inferior frontal gyrus (**lIFG, BA 44/45/47**)
- Supports **sentence integration**
- Generates the **P600** component
:::

:::spoiler **Neural Evidence**
- **Neuroimaging & lesion studies** confirm **lpMTG** involvement in **word recognition & retrieval**.
- **lIFG** supports **complex language processing**, including **syntax, semantics, & memory**.
- **The author:** a variety of processes involved in the
(re)composition of an utterance representation
:::

---

## **Neural Pathways & Processing Cycle**

:::spoiler **Information Flow**
1. **Word Input**: Enters via **auditory (spoken) or visual (written) cortex**.
2. **Retrieval**: **lpMTG** retrieves word meaning (**triggers N400**).
3. **Integration**: Meaning sent to **lIFG** for **sentence integration** (**triggers P600**).
4. **Updating**: Integrated meaning sent back to **lpMTG** to **anticipate upcoming words**.
:::

:::spoiler **Pathways**
- **Dorsal & Ventral Pathways**: Enable **bidirectional communication** between **lpMTG & lIFG**.
- Supports the **Retrieval-Integration (RI) processing cycle**, aligning with **neurocomputational models**.
:::

---

<style>
/* Styling for better readability */
h1, h2, h3 {
color: #2E86C1;
font-family: Arial, sans-serif;
border-bottom: 2px solid #ddd;
padding-bottom: 5px;
}
h1 {
font-size: 28px;
}
h2 {
font-size: 24px;
}
h3 {
font-size: 20px;
}
p {
font-size: 16px;
line-height: 1.6;
}
code {
background-color: #f4f4f4;
padding: 3px 5px;
border-radius: 4px;
}
blockquote {
font-style: italic;
color: #7B7D7D;
border-left: 4px solid #3498DB;
padding-left: 10px;
}
</style>