自然語言處理 - 李龍豪 (2024 Spring)

Class info.

課程資訊

Temporary Grading Policy:

• 自選論文書面報告: 2020~2023 ACL/EMNLP (20%)
• 指定論文口頭報告 (30%)
• 期末專題實作與報告 (50%)
  中文維度型情感分析
  • Training Set: Chinese EmoBank
  • Test Set: 1000+ Mental Health Texts

Date

2/22

Course Overview

Encoding

• ASCII
• Unicode
• UTF-8
• BIG-5 (繁體中文)
• GB_2312 (簡體中文)

Words

• Morphology
• Word Segmentation
• Part-of-Speech Tagging

Syntax

• Constituency Grammars
• Syntactic Parsing
• Dependency Parsing

Semantics

• Lexical Semantics
• Semantic Role Labeling
• Words Sense Disambiguation

Pragmatics

• Coreference Resolution
• Discourse Analysis
• Simile vs Metaphor

Approach

• Rule-Based Approach
• Corpus-Based Approach
• Statistical Language Models
• Neural Language Models
• Pre-Trained Language Models
• Large Language Models

Applications

• Machine Translations
• Question Answering
• Summarization
• Dialog Systems and Chatbots
• Grammatical/Spelling Error Correction
• Sentiment Analysis

2/29

NLP Fundamental Tasks

Regular Expression

Text Normalization

• Tokenizing (segmenting) words
• Normalizing word formats
• Segmenting sentences

Tokenization: the task of segmenting running text into words

Name Entity Recognition (NER)

• BiLSTM with CRF Model

Evaluation of NER

• Recall
• Precision
• F-measure

$$F=2\cdot \frac{\text{precision}\cdot \text{recall}}{\text{precision}+\text{recall}}$$

3/7

Statistical Language Models

Language Model

Models that assign probability to upcoming words, or sequences of words.

N-gram Language Models

The n-gram model is the simplest kind of language model.

The probability of a word

$w$ give some history

$h$.

$$P(w\mid h)$$

The bi-gram model approximates the probability of a word given all the previous words.

$$P(w_n\mid w_{1:n-1})\approx P(w_n\mid w_{n-1})$$

Chain Rule of Probability

Applying the chain rule of words

$$\begin{array}{rl}P(w_{1:n})& =P(w_1)P(w_2\mid w_1)P(w_3\mid w_{1:2})\dots P(w_n\mid w_{1:n-1})\\ & =\prod _{k=1}^{n}P(w_k\mid w_{1:k-1})\end{array}$$

Hidden Markov Models

Markov models are the class of probabilistic models the assume we can predict the probaility of some future unit wiyhout looking too far into the past.

Three fundamental problems:

• Likelihood
• Decoding
• Learning

Perplexity

$$\text{perplexity}(W)=P(w_1{w}_2\dots w_N{)}^{\frac{1}{N}}=(\prod _{i=1}^N\frac{1}{P(w_i\mid w_{i-1})}{)}^{\frac{1}{N}}$$

3/14

Embedding & Neural Language Models

Lexical Semantics

Connotation

• Valence
• Arousal
• Dominance

Word-Word Co-occurrence Matrix

TF-IDF: Weighting terms in the vector

• TF (Term Frequency)
• IDF (Inverse Document Frequency)

$${\mathrm{tf}}_{t,d}=\{\begin{array}{ll}1+{\log }_{10}\mathrm{count}(t,d)& \text{ if }\mathrm{count}(t,d)>0\\ 0& \text{ otherwise }\end{array}$$

$${\mathrm{idf}}_t={\log }_{10}(\frac{N}{{\mathrm{df}}_t})$$

$$w_{t,d}={\mathrm{tf}}_{t,d}\times {\mathrm{idf}}_t$$

Word2vec

Other Kinds of Static Embeddings

• GloVe
• Fasttext

Visualizing Embeddings

Probably the most common visualization method is to project the 100 dimensions of a word down into 2 dimensions suing a projection method called t-SNE

3/21

Pre-trained Language Model

• MLP is linear transfer
• Feed forward network: non-linear transfer (activation)

Computational Graphs

Computational Graphs 是一種表示神經網路計算過程的結構。它將神經網路的各個層、節點和運算操作以圖形的方式連接起來，形成一個有向圖。這個圖描述了正向傳播和反向傳播的過程，以及每個節點之間的數值流動。

Embedding Layer

• 嵌入層是一種將高維度資料轉換為低維度表示的技術，同時保留原始資料的特性。
• 在處理像是稀疏向量等大量輸入的情況下，使用嵌入層可以使模型訓練更簡單。

RNN & LSTM

RNN: any network that contains a cycle within its network connections

Bi-RNN

LSTM: forget/add/output gates

Bi-LSTM

GRU

CNN for NLP

3/28

Large Language Model

停課

4/4

Tomb Sweeping Day (Skip)

4/11

Large Language Model

Transformer

• Backward looking
• Bidirectional

How to compare words?

• Inner product
• Query / Key / Value
• Attention score dividing the square root of the dimensionality of the query and key vectors

Multihead attention

Transformer block

Positional Embedding

Sampling

• Top-K
• Top-P
• Temperature

A Survey of Large Language Models

4/18

Paper presentation

• bert2BERT: Towards Reusable Pretrained Language Models
• We’re Afraid Language Models Aren’t Modeling Ambiguity
• Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning
• WhitenedCSE: Whitening-based Contrastive Learning of Sentence Embeddings
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models

4/25

Paper presentation

• DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models
• Controllable Text Generation via Probability Density Estimation in the Latent Space
• Translation-Enhanced Multilingual Text-to-Image Generation
• Language Model is Suitable for Correction of Handwritten Mathematical Expressions Recognition
• Self-Supervised Multimodal Opinion Summarization

5/2

Paper presentation

• Multi-Party Empathetic Dialogue Generation: A New Task for Dialog Systems
• E-CORE: Emotion Correlation Enhanced Empathetic Dialogue Generation
• Synchronous Dual Network with Cross-Type Attention for Joint Entity and Relation Extraction
• OD-RTE: A One-Stage Object Detection Framework for Relational Triple Extraction
• Selectively Answering Ambiguous Questions

5/9

Project progress

I use MoE Bert.

5/16

Project related paper presentation

• Non-Linear Text Regression with a Deep Convolutional Neural Network
• Predicting Valence-Arousal Ratings of Words Using a Weighted Graph Method
• Dimensional Sentiment Analysis Using a Regional CNN-LSTM Model
• Sentiment Composition of Words with Opposing Polarities
• Knowledge-enriched Two-layered Attention Network for Sentiment Analysis

5/23

Project related paper presentation

• Community-Based Weighted Graph Model for ValenceArousal Prediction of Affective Words
• EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis
• Tensor Fusion Network for Multimodal Sentiment Analysis
• A Multilayer Perceptron based Ensemble Technique for Fine-grained Financial Sentiment Analysis
• Volatility Prediction Using Financial Disclosures Sentiments with Word Embedding-based IR Model

5/30

Project related paper presentation

• Investigating Dynamic Routing in Tree-Structured LSTM for Sentiment Analysis
• Adversarial Attention Modeling for Multi-dimensional Emotion Regression
• Pipelined Neural Networks for Phrase-Level Sentiment Intensity Prediction
• Tree-Structured Regional CNN-LSTM Model for Dimensional Sentiment Analysis
• From Polarity to Intensity: Mining Morality from Semantic Space
• All-in-One: Emotion, Sentiment and Intensity Prediction Using a Multi-Task Ensemble Network

6/6

Project presentation

Reference

• Introduction to Natural Language Processing
• Speech and Language Processing 3rd edition