AI Mastery

PREVIOUS PAGE INDEX PAGE NEXT PAGE

Mathematical Models in NLP: Embeddings Probabilistic Approaches & Language Understanding

N.B.- All my books are exclusively available on Amazon. The free notes/materials on globalcodemaster.com do NOT match even 1% with any of my PUBLISHED BOoks. Similar topics ≠ same content. Books have full details, exercises, chapters & structure — website notes do not.No book content is shared here. We fully comply with Amazon policies.

TABLE OF CONTENT

0. Orientation & How to Use These Notes

0.1 Target Audience & Learning Pathways 0.2 Prerequisites (Probability, Linear Algebra, Information Theory, Basic Deep Learning) 0.3 Notation & Mathematical Conventions 0.4 Evolution of Mathematical NLP: From 1950s to 2026 0.5 Version History & Update Log

1. Foundations of Language Modeling

1.1 Probability & Information Theory Essentials 1.1.1 Entropy, Cross-Entropy, Perplexity 1.1.2 KL Divergence & Mutual Information 1.1.3 Chain Rule & Conditional Probability in Sequences 1.2 N-gram Language Models 1.2.1 Maximum Likelihood Estimation & Smoothing (Laplace, Kneser-Ney) 1.2.2 Backoff & Interpolation Models 1.2.3 Perplexity Decomposition & Evaluation 1.3 Neural Language Models – Early Architectures 1.3.1 Bengio et al. (2003) Neural Probabilistic Language Model 1.3.2 Recurrent Neural Networks (RNN, LSTM, GRU) for Sequences 1.3.3 Vanishing/Exploding Gradients & Long-Range Dependencies

2. Word & Subword Embeddings

2.1 Distributional Semantics & Word Embeddings 2.1.1 Count-based Methods (PMI, PPMI, SVD on Co-occurrence Matrices) 2.1.2 Skip-gram with Negative Sampling (SGNS) & CBOW 2.1.3 GloVe: Global Vectors for Word Representation 2.2 Subword & Character-level Embeddings 2.2.1 Byte-Pair Encoding (BPE), WordPiece, SentencePiece, Unigram LM 2.2.2 FastText: Subword n-grams + Morphological Information 2.2.3 Contextual vs. Static Embeddings – Transition Point 2.3 Geometric & Algebraic Properties of Embeddings 2.3.1 Vector Space Analogies & Linear Substructures 2.3.2 Intrinsic Evaluation (WordSim, SimLex-999) 2.3.3 Extrinsic Evaluation & Downstream Transfer

3. Contextual Embeddings & Transformer Foundations

3.1 Sequence-to-Sequence & Attention Mechanisms 3.1.1 Bahdanau Attention & Luong Attention 3.1.2 Self-Attention & Multi-Head Attention 3.2 Transformer Architecture (Vaswani et al. 2017) 3.2.1 Scaled Dot-Product Attention & Positional Encoding 3.2.2 Feed-Forward Layers & Layer Normalization 3.2.3 Encoder–Decoder vs. Encoder-only vs. Decoder-only 3.3 BERT, RoBERTa, ELECTRA & Pre-training Objectives 3.3.1 Masked Language Modeling (MLM) & Next Sentence Prediction 3.3.2 Replaced Token Detection (ELECTRA) 3.3.3 DeBERTa v3, DeBERTaV3 – Disentangled Attention & Enhanced Masking

4. Autoregressive & Decoder-only Language Models

4.1 GPT Family & Scaling Laws 4.1.1 GPT-1 → GPT-4o → GPT-5 Era Scaling (Kaplan, Hoffmann/Chinchilla) 4.1.2 Emergent Abilities & Phase Transitions 4.1.3 Instruction Tuning, RLHF, RLAIF 4.2 Mixture-of-Experts & Sparse Activation 4.2.1 Switch Transformers, GLaM, Mixtral, DeepSeek-MoE 4.2.2 Routing Mechanisms & Load Balancing 4.3 Efficient Autoregressive Decoding 4.3.1 Speculative Decoding (Medusa, Lookahead, Eagle) 4.3.2 KV Cache Compression & PagedAttention

5. Probabilistic & Bayesian Approaches in Modern NLP

5.1 Variational Inference & Amortized Inference 5.1.1 VAE for Text & β-VAE 5.1.2 Amortized Variational Inference in LLMs (e.g., VQ-VAE-2 extensions) 5.2 Bayesian Neural Networks & Uncertainty in Language 5.2.1 Deep Ensembles, MC Dropout, SWAG 5.2.2 Laplace Approximation & Last-Layer Laplace 5.2.3 LLM Uncertainty Quantification (Verbalized Confidence, Semantic Entropy) 5.3 Diffusion Models & Continuous-Time Generative Models for Text 5.3.1 Diffusion-LM, SSD-LM, GenAI Diffusion Variants 5.3.2 Score-based Generative Modeling on Discrete Spaces

6. Structured Prediction & Constrained Decoding

6.1 CRF, HMM & Sequence Labeling 6.1.1 Viterbi Decoding & Forward-Backward Algorithm 6.1.2 Neural CRF & Lattice-based Models 6.2 Constrained & Controlled Generation 6.2.1 Beam Search with Length & Lexical Constraints 6.2.2 Grid Beam Search, Diverse Beam Search 6.2.3 Energy-based Models & Gradient-based Decoding 6.3 Semantic Parsing & Logical Reasoning 6.3.1 Tree-based Models & AM-PM (Algebraic Machine Parsing) 6.3.2 LLM-based Program Synthesis & Tool-use

7. Evaluation, Interpretability & Robustness

7.1 Evaluation Metrics for Language Understanding 7.1.1 Perplexity vs. Downstream Tasks vs. Human Judgments 7.1.2 LLM-as-a-Judge, Pairwise Comparison, Elo Rankings 7.1.3 Benchmark Suites (GLUE → SuperGLUE → BIG-bench → MMLU → LMSYS Arena → HELM) 7.2 Mechanistic Interpretability 7.2.1 Logit Lens, Activation Patching, Causal Tracing 7.2.2 Circuit Discovery & Sparse Autoencoders (Anthropic 2024–2026) 7.3 Robustness & Adversarial Attacks in Language Models 7.3.1 Prompt Injection, Adversarial Suffixes, Jailbreaking 7.3.2 Adversarial Training & Robust Alignment

8. Advanced Topics & Research Frontiers (2025–2026)

8.1 Long-Context & Memory-Augmented Models 8.1.1 Ring Attention, Infini-Transformer, RWKV, Mamba-2 8.1.2 State Space Models & Linear RNN Alternatives 8.2 Multimodal & Grounded Language Models 8.2.1 CLIP, Flamingo, LLaVA, Qwen-VL, Kosmos-2 8.2.2 Vision-Language Alignment & Visual Prompting 8.3 Agents, Reasoning & Tool Use 8.3.1 ReAct, Reflexion, Tree-of-Thoughts, Graph-of-Thoughts 8.3.2 Self-Consistency, Self-Refine, Constitutional AI 8.4 Open Problems & Thesis Directions

9. Tools, Libraries & Implementation Resources

9.1 Core Frameworks (PyTorch, Hugging Face Transformers, vLLM, llama.cpp) 9.2 Interpretability Tools (TransformerLens, nnsight, CircuitsVis) 9.3 Evaluation Suites & Benchmarks (EleutherAI LM Harness, OpenCompass) 9.4 Datasets & Preprocessing Pipelines

10. Assessments, Exercises & Projects

10.1 Conceptual & Proof-Based Questions 10.2 Coding Exercises (BPE from scratch, LoRA fine-tuning, speculative decoding) 10.3 Mini-Projects (Long-context RAG, constrained generation, circuit discovery) 10.4 Advanced / Thesis-Level Project Ideas

0. Orientation & How to Use These Notes

Welcome to Mathematical Models in NLP: Embeddings, Probabilistic Approaches & Language Understanding — a rigorous, up-to-date (2026) resource that bridges classical probabilistic foundations with the mathematical machinery powering modern large language models (LLMs), contextual embeddings, reasoning, and controllable generation.

This section orients the reader, clarifies prerequisites, defines notation, provides historical context, and explains how to navigate the material effectively.

0.1 Target Audience & Learning Pathways

Primary audiences

AudienceBackground / GoalRecommended Pathway through the NotesMSc / early PhD studentsBuilding strong theoretical foundation for NLP / ML researchSequential: 0 → 1 → 2 → 3 → 4 → 5 → 7 → 10 (exercises & projects)Advanced undergraduatesDeepening understanding beyond black-box Transformers0 → 1 → 2 → 3.1–3.2 → 7.1 (evaluation & interpretability)NLP / ML researchersKeeping up with 2025–2026 mathematical frontiers (reasoning, long-context, uncertainty, agents)4 (autoregressive scaling), 5 (Bayesian & diffusion), 8 (frontiers), 7.2–7.3 (interpretability & robustness)Industry engineers (LLM fine-tuning, serving, safety)Understanding why certain techniques work / fail at scale3.3–4 (Transformer & autoregressive), 6 (constrained decoding), 9 (tools), 7.3 (robustness)Professors / lecturersStructured lecture material, proofs, exercises, project ideasFull sequential read + 10.1–10.3 for assignments, 10.4 for thesis/capstone supervision

Suggested learning tracks (2026)

• Fast practical track (3–5 months): 0 → 2 → 3 → 4.1–4.3 → 6 → 9 → selected parts of 7

• Research-oriented track (9–18 months): Full sequential + deep dives into 1, 5, 8, papers from Appendix C

• Interpretability & safety focus (mid-career): 3 → 7.2 (mechanistic interpretability) → 7.3 (robustness & attacks) → 8.3 (agents & reasoning)

• Long-context & efficiency focus: 4.3 + 8.1 (long-context architectures) + 6.2 (constrained decoding)

0.2 Prerequisites

To extract maximum value, readers should already be comfortable with:

Mathematics

• Probability & statistics: random variables, expectation, variance, common distributions (Bernoulli, categorical, Gaussian), Bayes’ rule, maximum likelihood estimation

• Information theory: entropy, cross-entropy, KL divergence, mutual information

• Linear algebra: matrix multiplication, eigenvalues/eigenvectors, norms, SVD, basic inner products

• Multivariate calculus: gradients, chain rule, basic optimization (gradient descent intuition)

Machine learning / deep learning

• Feed-forward neural networks, backpropagation

• Basic sequence models (RNN intuition, LSTM/GRU gates)

• Familiarity with attention mechanism (even high-level)

• Comfort reading PyTorch / JAX-style pseudocode

Nice-to-have (reviewed when needed)

• Introductory NLP (tokenization, n-grams, bag-of-words)

• Basic Transformer architecture (self-attention, positional encoding)

• Experience fine-tuning small models (e.g., BERT-base or GPT-2 small)

Recommended refreshers (free & concise)

• Probability & information theory: “Deep Learning” (Goodfellow et al.) Chapters 3 & 5

• Linear algebra for ML: “Mathematics for Machine Learning” (Deisenroth et al.) Chapters 2–4

• Transformers basics: “The Illustrated Transformer” (Jay Alammar) or original “Attention is All You Need” paper

• PyTorch: official tutorials (tensor operations, autograd, nn.Module)

0.3 Notation & Mathematical Conventions

Standard modern NLP/ML notation is used throughout (aligned with 2023–2026 papers).

Symbol / ConventionMeaning / UsageBold lowercaseVectors: x, h, e (embedding)Bold uppercaseMatrices: W, Q, K, VCalligraphicSets: 𝒳 (token space), 𝒱 (vocabulary), ℒ (loss)Blackboard boldNumber fields: ℝ, ℕExpectation, probability𝔼[·], ℙ(·) or E[·], P(·)Indicator𝟙{condition}TransposeAᵀ or A^THadamard product⊙Softmaxsoftmax(z)_i = exp(z_i) / ∑ exp(z_j)Cross-entropy lossH(p, q) = –∑ p(x) log q(x)PerplexityPPL = 2^{H(p,q)}≜Defined as∼Distributed as (x ∼ Categorical(p))≈Approximately equal

Derivations are step-by-step when introducing new concepts; proofs are complete but concise (references for deeper treatments).

0.4 Evolution of Mathematical NLP: From 1950s to 2026

EraDominant ParadigmKey Mathematical AdvancesLandmark Models / Papers1950s–1980sRule-based & early statisticalFinite-state automata, HMMs, n-gram MLEShannon (1948), Chomsky (1957), Baum–Welch (1970)1990s–2000sProbabilistic & discriminativeLog-linear models, CRFs, maximum-entropy, EM algorithmBerger et al. (1996), Lafferty et al. (2001)2003–2017Neural & distributionalNeural language models, word embeddings, RNNs/LSTMs, attentionBengio (2003), Mikolov (2013), Bahdanau (2014)2017–2020Transformer revolutionSelf-attention, scaled dot-product, multi-head, BERT-style pre-trainingVaswani (2017), Devlin (2018), Radford (2018–2019)2020–2023Scaling & decoder-only dominanceAutoregressive scaling laws, RLHF, emergent abilities, sparse MoEBrown (2020), Hoffmann (2022), Chowdhery (2022)2023–2026Reasoning, long-context, agents & interpretabilityTest-time compute scaling, state-space models, circuit discovery, semantic entropyWei (2022), Yao (2023), Anthropic (2024–2026), Gemini (2024–2025)

2026 snapshot: Mathematical focus has shifted toward reasoning circuits, uncertainty quantification, long-context linear-time alternatives (Mamba-2, RWKV-v6), test-time scaling, and mechanistic interpretability.

0.5 Version History & Update Log

VersionDateMajor Additions / Changes1.0Jan 2025Initial release: Sections 0–3, core embeddings & Transformer foundations1.1May 2025Added Section 4 (autoregressive & MoE), updated scaling laws & reasoning papers1.2Sep 2025Section 5 (Bayesian & diffusion), Section 8.1 (long-context SSMs), mechanistic interpretability1.3Jan 20262026 frontier: semantic entropy, test-time compute laws, agentic reasoning, updated benchmarks1.4Mar 2026Current version: new exercises, Grok-4 / Gemini 2.5 references, long-context comparisons

This is a living document — updated quarterly as new theoretical insights and architectural breakthroughs emerge.

1. Foundations of Language Modeling

Language modeling is the core task of NLP: given a sequence of words (or tokens), predict the probability distribution over the next word (or the entire sequence). Mathematically, a language model defines a probability distribution P(w₁, w₂, …, wₙ) over sequences of arbitrary length.

This section reviews the probabilistic and information-theoretic foundations that underpin every modern language model — from n-gram smoothing to autoregressive Transformers and diffusion-based text generation. These concepts appear repeatedly in perplexity calculations, pre-training objectives, evaluation, alignment, and reasoning analysis.

1.1 Probability & Information Theory Essentials

1.1.1 Entropy, Cross-Entropy, Perplexity

Entropy H(p) measures the average uncertainty (information content) of a discrete random variable X ~ p(x):

H(p) = – ∑_{x} p(x) log₂ p(x) (in bits) or – ∑ p(x) log p(x) (natural log, nats)

For language modeling, entropy of the true data distribution p* gives the theoretical lower bound on average bits-per-character (or bits-per-token) needed to encode text.

Cross-entropy H(p, q) = – ∑ p(x) log q(x) = H(p) + D_KL(p || q) ≥ H(p)

Cross-entropy between true distribution p and model q is the quantity minimized during training (negative log-likelihood).

Perplexity (PPL) is the exponential of cross-entropy:

PPL(q) = 2^{H(p,q)} = exp(H(p,q)) (base-2 or natural log depending on convention)

Lower perplexity → better model. Example: random guessing over V = 50,000 tokens → PPL ≈ 50,000 Human-level English text perplexity is estimated ~10–30 bits/char (≈ 1000–10⁹ tokens vocabulary equivalent).

2026 note: Perplexity remains the gold-standard intrinsic metric for autoregressive LMs, but saturates quickly on large models → downstream tasks and human judgments increasingly important.

1.1.2 KL Divergence & Mutual Information

Kullback-Leibler (KL) divergence (asymmetric):

D_KL(p || q) = ∑ p(x) log (p(x) / q(x)) = H(p,q) – H(p) ≥ 0

KL measures how much extra information is needed when using q to encode samples from p. In language modeling: D_KL(p* || q) = H(p*, q) – H(p*) → minimized when q approximates true data distribution p*.

Mutual information I(X;Y) = H(X) – H(X|Y) = H(Y) – H(Y|X) = D_KL(p(x,y) || p(x)p(y))

Measures dependence between variables. In NLP: I(w_t ; w_{1:t-1}) quantifies how much past context reduces uncertainty about the next word.

2026 applications:

• Semantic entropy (Farquhar et al. 2024–2025): cluster LLM output distributions → estimate epistemic uncertainty via conditional entropy reduction.

• Mutual information maximization in contrastive learning (e.g., CLIP-style vision-language alignment).

1.1.3 Chain Rule & Conditional Probability in Sequences

Any joint distribution factors via the chain rule:

P(w₁, w₂, …, wₙ) = P(w₁) P(w₂ | w₁) P(w₃ | w₁,w₂) … P(wₙ | w₁…w_{n-1})

Autoregressive language models approximate the conditional distributions P(w_t | w_{<t}) using a fixed parametric form (n-gram tables, RNN, Transformer).

Log-probability of a sequence (negative log-likelihood, NLL):

log P(w₁…wₙ) = ∑{t=1}^n log P(w_t | w{<t})

Average NLL per token = (1/n) ∑ log P(w_t | w_{<t}) → minimized during training.

Per-token log-prob is the fundamental quantity reported in scaling law papers (e.g., bits-per-byte, nats-per-token).

1.2 N-gram Language Models

Despite being largely replaced by neural models, n-grams remain important for baselines, smoothing analysis, and understanding interpolation/backoff — concepts reused in modern neural smoothing and mixture-of-experts routing.

1.2.1 Maximum Likelihood Estimation & Smoothing (Laplace, Kneser-Ney)

MLE for n-gram:

P̂(w_t | w_{t-n+1}…w_{t-1}) = count(w_{t-n+1}…w_t) / count(w_{t-n+1}…w_{t-1})

Problem: sparse counts → zero probabilities for unseen n-grams.

Additive smoothing (Laplace / Add-δ):

P(w_t | h) = [count(h w_t) + δ] / [count(h) + δ |V|]

δ = 1 (Laplace) → uniform prior over unseen words.

Kneser-Ney smoothing (best classical smoother):

Absolute discounting + lower-order continuation probabilities:

P_{KN}(w_t | h) = max(count(h w_t) – D, 0) / count(h)

• λ(h) P_{KN}(w_t | h')

where λ(h) is the missing mass, and P_{KN}(· | h') uses continuation counts (how many distinct left contexts a word appears in).

Kneser-Ney consistently outperforms Laplace/additive smoothing because it gives higher probability to words seen in diverse contexts.

1.2.2 Backoff & Interpolation Models

Backoff: if n-gram unseen, fall back to (n-1)-gram and renormalize:

P(w_t | h) = P_{disc}(w_t | h) if count(h w_t) > 0 = α(h) P(w_t | h[2:]) otherwise

Interpolation (Jelinek-Mercer): weighted mixture of all orders:

P(w_t | h) = λ_n P(w_t | h_n) + (1-λ_n) P(w_t | h_{n-1})

λ_n estimated on held-out data.

Modified Kneser-Ney (Chen & Goodman 1998) combines discounting + interpolation + continuation counts → still one of the strongest classical LMs.

1.2.3 Perplexity Decomposition & Evaluation

Perplexity on test set:

PPL = exp( – (1/N) ∑ log P(w_i | w_{<i}) )

Decomposition: train perplexity vs. test perplexity gap reveals overfitting. N-gram models typically reach test PPL ~100–300 on English (1B tokens), while modern LLMs reach <10–20.

Intrinsic vs. extrinsic: Perplexity correlates moderately with downstream performance; correlation weakens at low perplexity (<20).

1.3 Neural Language Models – Early Architectures

1.3.1 Bengio et al. (2003) Neural Probabilistic Language Model

First neural language model:

P(w_t | w_{t-n+1}…w_{t-1}) = softmax( W₂ tanh(W₁ e + b₁) + b₂ )

where e is concatenation of learned word embeddings.

Key innovations: distributed word representations + shared parameters across contexts → overcomes curse of dimensionality of n-gram tables.

Limitations: fixed window, no recurrence → poor at long dependencies.

1.3.2 Recurrent Neural Networks (RNN, LSTM, GRU) for Sequences

Vanilla RNN:

h_t = tanh(W{hh} h{t-1} + W_{xh} x_t + bh) y_t = softmax(W{hy} h_t + b_y)

LSTM (Hochreiter & Schmidhuber 1997):

Forget gate f_t, input gate i_t, output gate o_t, cell state c_t → explicit memory cell + gating → much better long-range retention.

GRU (Cho et al. 2014): simplified LSTM with update/reset gates → comparable performance, fewer parameters.

RNN LMs reached state-of-the-art perplexity in mid-2010s (~40–60 on 1B benchmark) before Transformers.

1.3.3 Vanishing/Exploding Gradients & Long-Range Dependencies

Gradient of loss w.r.t. early hidden state:

∂L / ∂h_t ≈ ∏{k=t+1}^T ∂h_k / ∂h{k-1} · ∂L / ∂h_T

Product of Jacobian matrices → eigenvalues <1 → vanishing; >1 → exploding.

Mitigations:

• Gradient clipping

• Better initialization (Xavier, He)

• LayerNorm / residual connections (later Transformers)

• LSTM/GRU gating

• Truncated backpropagation through time (BPTT)

Even with LSTM/GRU, effective context length rarely exceeds ~100–200 tokens → main motivation for attention-based models.

2. Word & Subword Embeddings

Word embeddings represent words (or subwords) as dense, low-dimensional vectors in ℝ^d such that semantically and syntactically similar words lie close together in the vector space. This section covers the mathematical evolution from count-based distributional methods to predictive neural embeddings (static) and the transition toward contextual representations.

These ideas remain foundational even in 2026: static embeddings are still used for lightweight tasks, initialization of Transformer token embeddings, retrieval augmentation, and interpretability studies, while subword tokenization is universal in modern LLMs.

2.1 Distributional Semantics & Word Embeddings

The distributional hypothesis (Harris 1954; Firth 1957): “You shall know a word by the company it keeps.” Words appearing in similar contexts tend to have similar meanings.

2.1.1 Count-based Methods (PMI, PPMI, SVD on Co-occurrence Matrices)

Co-occurrence matrix C ∈ ℝ^{|V| × |V|}: C_{ij} = number of times word j appears in the context window of word i (usually symmetric, window size 5–10).

Raw counts are dominated by frequent words → apply weighting.

Pointwise Mutual Information (PMI):

PMI(i,j) = log₂ ( P(w_i, w_j) / (P(w_i) P(w_j)) ) = log₂ ( C_{ij} / (∑k C{ik} · ∑k C{kj} / ∑{k,l} C{kl}) )

PMI can be negative → shift to non-negative by taking max(0, PMI) → Positive PMI (PPMI).

SVD on PPMI matrix (Levy & Goldberg 2014; earlier LSA):

Factorize PPMI ≈ U Σ Vᵀ Take rows of U Σ^{1/2} (or just U) as word embeddings (common dimensions d = 50–300).

Mathematical advantage: SVD captures global co-occurrence statistics; low-rank approximation denoises and discovers latent semantic dimensions.

Limitations: Very high memory for large |V| (millions of types); no subword handling; static (no polysemy).

2.1.2 Skip-gram with Negative Sampling (SGNS) & CBOW

Skip-gram (Mikolov et al. 2013) predicts context words given target:

maximize ∑{t=1}^T ∑{-c ≤ j ≤ c, j≠0} log P(w_{t+j} | w_t)

P(o|c) ≈ σ( u_oᵀ v_c ) where σ is sigmoid, v_c target embedding, u_o context embedding.

Direct softmax over |V| is O(|V|) per update → intractable.

Negative Sampling approximates:

log σ( uoᵀ vc ) + ∑{k=1}^K log σ( – u{n_k}ᵀ v_c )

where n_k sampled from unigram^{3/4} distribution (noisier frequent words downweighted).

Continuous Bag-of-Words (CBOW) reverses direction: predict target from average of context words → faster training, slightly worse on rare words.

Why SGNS works so well (Levy & Goldberg 2014): SGNS objective implicitly factorizes a shifted PPMI matrix → same low-rank semantic subspace as SVD-based methods, but scalable (O(1) per update via sampling).

2026 note: SGNS-style objectives still appear in contrastive pre-training (e.g., CLIP, SimCLR derivatives) and word2vec-style initializations for vocabulary-efficient models.

2.1.3 GloVe: Global Vectors for Word Representation

GloVe (Pennington et al. 2014) combines global matrix statistics with local context prediction.

Objective: minimize

J = ∑{i,j=1}^{|V|} f(X{ij}) ( w_iᵀ wj + b_i + b_j – log X{ij} )²

where X_{ij} = co-occurrence count, f(X) = weighting function (caps at 100, power-law decay).

Key insight: Weighted least-squares regression on log-co-occurrences recovers semantic vector space.

Advantages over SGNS: Incorporates global statistics (like SVD); better on analogy tasks; symmetric (no distinction between target/context).

Empirical result: GloVe often slightly outperforms word2vec on intrinsic word similarity benchmarks.

2.2 Subword & Character-level Embeddings

Modern LLMs use subword tokenization to handle open vocabularies, rare words, and morphology.

2.2.1 Byte-Pair Encoding (BPE), WordPiece, SentencePiece, Unigram LM

Byte-Pair Encoding (BPE) (Sennrich et al. 2016): Start with character-level vocabulary → iteratively merge most frequent adjacent pairs → greedily builds subword units.

WordPiece (Schuster & Nakajima 2012; used in BERT): Similar to BPE but maximizes likelihood of training data under unigram assumption → adds ## prefix for non-initial subwords.

SentencePiece (Kudo & Richardson 2018): Unifies BPE and unigram LM tokenization; treats text as raw byte stream (no pre-tokenization); supports BPE, unigram, word-level, char-level.

Unigram LM (Kudo 2018): Probabilistic subword segmentation via EM-like optimization → maximizes likelihood of corpus under subword unigram model → allows multiple segmentations per word.

2026 reality: SentencePiece + BPE is dominant (Llama, Qwen, Grok, Gemma, DeepSeek); Unigram used in mT5, ByT5; all support raw bytes → robustness to typos, code-switching, low-resource languages.

2.2.2 FastText: Subword n-grams + Morphological Information

FastText (Bojanowski et al. 2017) extends Skip-gram: Each word w represented as sum of its character n-gram embeddings (n=3–6) + word embedding itself.

P(o|c) uses same negative sampling.

Advantages:

• Handles out-of-vocabulary words (compose from subword n-grams)

• Captures morphology (e.g., “playing” ≈ “play” + “-ing”)

• Strong on morphologically rich languages

Still used in 2026 for lightweight multilingual embeddings and as baseline in OOV-heavy domains.

2.2.3 Contextual vs. Static Embeddings – Transition Point

Static embeddings (word2vec, GloVe, FastText): one fixed vector per word type → polysemy collapse (bank_river vs. bank_finance same vector).

Contextual embeddings (ELMo 2018, BERT 2018 onward): Word representation depends on entire sentence → different vectors for same word in different contexts.

Transition: ELMo (biLSTM) → BERT (bidirectional Transformer MLM) → GPT-style decoder-only → modern LLMs. Static embeddings now mainly used for:

• Initialization / warm-starting token embeddings

• Retrieval (dense passage retrieval)

• Lightweight downstream tasks

• Interpretability baselines

2.3 Geometric & Algebraic Properties of Embeddings

2.3.1 Vector Space Analogies & Linear Substructures

Classic result (Mikolov et al. 2013): king – man + woman ≈ queen Paris – France + Italy ≈ Rome

These emerge because semantic relationships are approximately linear offsets in the embedding space.

Mathematical explanation (Levy & Goldberg 2014): Skip-gram implicitly factorizes shifted PMI matrix → analogies arise from log-prob ratio structure.

Limitations: Not all relations are linear; polysemy and context break strict geometry.

2026 extensions: Probe for linear substructures in contextual embeddings (e.g., “subject-verb agreement direction” in residual stream); sparse autoencoders discover monosemantic features with geometric properties.

2.3.2 Intrinsic Evaluation (WordSim, SimLex-999)

Intrinsic = direct similarity/analogy tasks:

• WordSim-353, SimLex-999: Spearman correlation between cosine similarity and human similarity judgments

• Google Analogy dataset: accuracy on syntactic/semantic analogies

Static embeddings peak ~0.70–0.80 Spearman on SimLex; contextual embeddings (BERT) reach ~0.85–0.90 when extracting contextualized word reps (e.g., last-layer average).

2.3.3 Extrinsic Evaluation & Downstream Transfer

Extrinsic = performance on downstream tasks (NER, sentiment, QA, NLI).

Static embeddings: concatenated or averaged as features → strong baseline in 2015–2018. Contextual embeddings: full fine-tuning or frozen feature extraction → massive gains (BERT improved GLUE by 7–80 points).

2026 perspective: Static embeddings still useful for retrieval (DPR, ColBERT), lightweight classifiers, and as frozen initializers. Extrinsic transfer remains the ultimate test — intrinsic similarity correlates only moderately with downstream gains at high performance regimes.

3. Contextual Embeddings & Transformer Foundations

The introduction of the Transformer architecture in 2017 marked a fundamental shift in NLP. By replacing recurrence with attention mechanisms, Transformers enabled parallelization, captured long-range dependencies more effectively, and paved the way for contextual embeddings — representations that change depending on the surrounding context in a sentence.

This section covers the transition from sequence-to-sequence attention to the full Transformer, its core mathematical components, and the major pre-training paradigms that produced powerful contextual encoders (BERT family) and decoder-only autoregressive models.

3.1 Sequence-to-Sequence & Attention Mechanisms

Before Transformers, seq2seq models (Sutskever et al. 2014; Cho et al. 2014) used encoder–decoder RNNs. Attention was introduced to solve the information bottleneck of fixed-length hidden states.

3.1.1 Bahdanau Attention & Luong Attention

Bahdanau (additive) attention (Bahdanau et al. 2015) — “Neural Machine Translation by Jointly Learning to Align and Translate”

Given encoder hidden states h₁, …, hₙ ∈ ℝ^h and decoder state sₜ at time t:

Alignment score e_{t,i} = vᵀ tanh(W₁ hᵢ + W₂ sₜ)

Attention weights α_{t,i} = softmax(e_{t,i}) over i = 1…n

Context vector cₜ = ∑ α_{t,i} hᵢ

Decoder update: sₜ = f(s{t-1}, y{t-1}, cₜ)

Key insight: Soft, differentiable alignment learned end-to-end → model learns which source words to focus on when generating each target word.

Luong (multiplicative / dot-product) attention (Luong et al. 2015)

e_{t,i} = sₜᵀ W hᵢ (general form) or sₜᵀ hᵢ (dot-product form)

Followed by location-aware or coverage extensions.

Comparison: Bahdanau more expressive (additive MLP), Luong simpler/faster (dot-product). Dot-product became dominant in Transformers due to scalability with matrix multiplication.

3.1.2 Self-Attention & Multi-Head Attention

Self-attention: Attention applied to the same sequence (queries, keys, values all come from the input).

For a sequence X ∈ ℝ^{n × d}:

Q = X W_Q, K = X W_K, V = X W_V

Attention(Q, K, V) = softmax(Q Kᵀ / √d_k) V

Scaled dot-product attention divides by √d_k to prevent vanishing gradients (large dot products → tiny softmax gradients).

Multi-head attention (Vaswani et al. 2017):

Split d_model into h heads → each head has d_k = d_v = d_model / h

Concatenate head outputs → linear projection:

MultiHead(Q, K, V) = Concat(head₁, …, head_h) W^O

Benefits:

• Multiple representation subspaces

• Each head can attend to different types of relationships (syntactic, semantic, positional)

3.2 Transformer Architecture (Vaswani et al. 2017)

The Transformer replaces recurrence with stacked self-attention + feed-forward layers.

3.2.1 Scaled Dot-Product Attention & Positional Encoding

Core attention block (as above).

Positional encoding (fixed or learned):

PE(pos, 2i) = sin(pos / 10000^{2i/d_model}) PE(pos, 2i+1) = cos(pos / 10000^{2i/d_model})

Added to input embeddings → allows attention to distinguish order.

Why sinusoidal?

• Bounded values

• Linear relationships between positions (sin(a+b) = sin a cos b + cos a sin b) → model can extrapolate to longer sequences

3.2.2 Feed-Forward Layers & Layer Normalization

Each Transformer layer:

X' = LayerNorm(X + MultiHeadAttention(X, X, X)) X'' = LayerNorm(X' + FFN(X'))

FFN (position-wise):

FFN(x) = max(0, x W₁ + b₁) W₂ + b₂ (ReLU or GELU in later variants)

LayerNorm normalizes across feature dimension (not batch) → stabilizes training.

Residual connections + LayerNorm enable very deep stacks (12–96+ layers in modern models).

3.2.3 Encoder–Decoder vs. Encoder-only vs. Decoder-only

• Encoder–Decoder (original Transformer): full seq2seq (machine translation, summarization) Encoder: bidirectional self-attention Decoder: masked self-attention + cross-attention to encoder outputs

• Encoder-only (BERT, RoBERTa, DeBERTa): bidirectional context → excellent for understanding tasks (classification, NER, QA) Pre-training: MLM + NSP or variants

• Decoder-only (GPT family, LLaMA, Qwen, Grok, DeepSeek, Gemma): autoregressive, left-to-right → excels at generation, reasoning, in-context learning Pre-training: next-token prediction (causal language modeling) Masked attention (causal mask) prevents future peeking

2026 dominance: Decoder-only architectures lead frontier performance (reasoning, long-context, instruction following); encoder-only still strong for embedding tasks and retrieval.

3.3 BERT, RoBERTa, ELECTRA & Pre-training Objectives

3.3.1 Masked Language Modeling (MLM) & Next Sentence Prediction

BERT (Devlin et al. 2018):

• Input: [CLS] sentence A [SEP] sentence B [SEP]

• Pre-training objectives:

  • Masked LM (MLM): randomly mask 15% tokens → predict original (80% [MASK], 10% random, 10% unchanged) Loss = cross-entropy over masked positions

  • Next Sentence Prediction (NSP): predict whether B follows A (50% positive, 50% random) → binary classification on [CLS]

RoBERTa (Liu et al. 2019): removes NSP, dynamic masking, larger batches, more data → stronger performance.

3.3.2 Replaced Token Detection (ELECTRA)

ELECTRA (Clark et al. 2020):

• Generator (small MLM) → replaces 15% tokens with plausible fakes

• Discriminator → binary classification: is token original or replaced? → Much more sample-efficient than MLM (discriminator sees every token)

ELECTRA-small outperforms BERT-base on many tasks; ELECTRA-large competitive with RoBERTa-large at lower compute.

3.3.3 DeBERTa v3, DeBERTaV3 – Disentangled Attention & Enhanced Masking

DeBERTa (He et al. 2020–2021):

• Disentangled attention: separate content and position vectors → attention score = content-content + content-position + position-content

• Enhanced mask decoder: absolute position embeddings in final layers → better at relative position modeling

DeBERTaV3 (He et al. 2022): ELECTRA-style replaced token detection + disentangled attention → state-of-the-art on many NLU benchmarks at release.

2026 status: DeBERTaV3-style disentangled attention appears in many efficient encoders; ELECTRA objective reused in compact models and continual pre-training.

4. Autoregressive & Decoder-only Language Models

The shift from encoder-only (BERT-style bidirectional) to decoder-only autoregressive architectures (GPT family and successors) has defined the frontier of NLP and generative AI since 2018–2019. Decoder-only models excel at next-token prediction, in-context learning, reasoning, long-form generation, and instruction following — capabilities that emerge reliably only at sufficient scale.

This section covers the GPT lineage and associated scaling laws, the phenomenon of emergent abilities, post-training alignment techniques (RLHF, RLAIF), the rise of sparse Mixture-of-Experts (MoE) architectures, and the engineering innovations that make efficient autoregressive inference feasible at 2026 scale.

4.1 GPT Family & Scaling Laws

4.1.1 GPT-1 → GPT-4o → GPT-5 Era Scaling (Kaplan, Hoffmann/Chinchilla)

GPT-1 (Radford et al. 2018) 117M parameters, Transformer decoder-only, trained on BooksCorpus (~800M words) with causal language modeling. Introduced zero-shot transfer via task prompts → laid groundwork for in-context learning.

GPT-2 (2019) → 1.5B parameters, WebText dataset (~40GB), demonstrated strong few-shot performance.

GPT-3 (Brown et al. 2020) → 175B parameters, ~300B tokens → dramatic few-shot / zero-shot gains → established scaling hypothesis.

Kaplan et al. (2020) – OpenAI scaling laws Power-law fits: L(N) ≈ (N_c / N)^α_N + L_∞ L(D) ≈ (D_c / D)^α_D + L_∞ Claimed optimal at fixed compute: very large N, relatively small D.

Hoffmann et al. (2022) – Chinchilla / DeepMind refinement Re-trained models across iso-FLOP regimes → showed optimal allocation is roughly balanced: N_opt ≈ a C^{0.5}, D_opt ≈ c C^{0.5} → ~20 tokens per parameter optimal (Chinchilla 70B outperformed 280B Gopher on less data).

2023–2026 era

• Llama-2/3, PaLM-2, Qwen-2, DeepSeek-V2, Grok-1/2/3/4, Gemini 1.5/2.0/2.5 all follow roughly Chinchilla-like ratios (15–30 tokens/parameter pre-training).

• Post-training (SFT + RLHF/RLAIF) uses 5–100× more tokens per parameter.

• Test-time compute scaling (chain-of-thought length, tree search, self-refine) follows similar power laws → reasoning performance scales predictably with inference FLOPs.

2026 frontier: Models exceed 10¹³ parameters (sparse MoE), trained on 10–50 trillion tokens; optimal ratio slightly shifts toward more tokens for reasoning-heavy post-training.

4.1.2 Emergent Abilities & Phase Transitions

Emergent abilities (Wei et al. 2022): Capabilities that appear suddenly and unpredictably at sufficient scale (e.g., arithmetic reasoning, few-shot learning, chain-of-thought prompting).

Examples (2022–2026):

• Multi-digit arithmetic → appears sharply around 10–100B parameters

• MMLU few-shot accuracy jumps from near-random to super-human between 10B–100B

• Symbolic manipulation, commonsense reasoning, code generation

Phase transitions: Sharp jumps in performance metrics as model size / training tokens cross critical thresholds. Explained by:

• Circuit formation (mechanistic interpretability view)

• Sharpness reduction / grokking-like dynamics

• Effective rank increase in residual stream

2026 view: Many “emergent” abilities are gradual when measured on finer-grained metrics or with better prompting; true discontinuities are rarer but exist in reasoning and tool-use tasks.

4.1.3 Instruction Tuning, RLHF, RLAIF

Instruction tuning (Wei et al. 2021; FLAN, T0): Fine-tune on diverse (instruction, output) pairs → enables zero-shot generalization to new instructions.

RLHF (Christiano et al. 2017; Ouyang et al. 2022 – InstructGPT/ChatGPT):

1. Supervised fine-tuning (SFT) on high-quality demonstrations

2. Reward model training: Bradley-Terry model on human preference pairs

3. PPO (or DPO/IPO variants) to maximize reward minus KL penalty (prevents drift from SFT policy)

RLAIF (Bai et al. 2022–2025; Constitutional AI): Replace human feedback with AI-generated preferences (self-critique, constitutional rules) → scales feedback data dramatically.

2026 status:

• Direct Preference Optimization (DPO), Identity Preference Optimization (IPO), Kahneman-Tversky Optimization (KTO) largely replace PPO → simpler, more stable

• RLAIF + synthetic data dominate open models (Zephyr, Starling, Llama-3.1-Instruct)

• Multi-turn preference datasets + safety alignment via rejection sampling

4.2 Mixture-of-Experts & Sparse Activation

4.2.1 Switch Transformers, GLaM, Mixtral, DeepSeek-MoE

Switch Transformers (Fedus et al. 2021): First large-scale MoE → route each token to exactly 1 expert out of many (sparse FFN layers).

GLaM (Du et al. 2021): 1.2T parameters, 97B active → strong few-shot performance with lower compute.

Mixtral 8×7B / 8×22B (Jan 2024–2025): Open weights, top open model for months → sparse MoE with top-2 routing.

DeepSeek-MoE (2024–2026): DeepSeek-V2/V3 (236B/671B total, ~21–37B active) → leads many open benchmarks → fine-grained expert specialization.

2026 trend: MoE is default for open frontier models; active parameters ~10–20% of total → Chinchilla-optimal scaling at lower FLOPs.

4.2.2 Routing Mechanisms & Load Balancing

Routing: Learned gating network g(x) → softmax over expert scores → top-k selection (k=1 or 2).

Load balancing loss (prevents expert collapse): AuxLoss = α ∑_i f_i · P_i where f_i = fraction of tokens routed to expert i, P_i = average gate probability to i.

Modern variants:

• Soft MoE (Puigcerver et al. 2024): dense softmax routing + weighted sum

• Expert Choice Routing: experts select tokens (stable load)

• DeepSeek-V3: fine-grained expert splitting + auxiliary-free balancing

4.3 Efficient Autoregressive Decoding

Autoregressive generation is sequential → inference latency scales with output length.

4.3.1 Speculative Decoding (Medusa, Lookahead, Eagle)

Speculative decoding (Leviathan et al. 2023; Chen et al. 2023): Draft model generates multiple tokens in parallel → verification by target model → accept/reject prefix.

Medusa (Cai et al. 2023–2024): Multiple prediction heads on top of frozen backbone → predicts several future tokens → tree search over drafts.

Lookahead Decoding (Fu et al. 2024): n-gram-assisted speculation → reuses n-gram cache for acceleration.

Eagle (Li et al. 2024–2025): Progressive speculation + draft model fine-tuned on rejection samples → 2–3× speedup with minimal quality loss.

2026 status: Speculative decoding + continuous batching standard in vLLM, TensorRT-LLM → 2–5× throughput gains on long generations.

4.3.2 KV Cache Compression & PagedAttention

KV cache grows linearly with sequence length → memory bottleneck for long-context inference.

PagedAttention (Kwon et al. 2023, vLLM): Treat KV cache as virtual memory pages → non-contiguous allocation → eliminates fragmentation → supports dynamic batching.

KV cache compression:

• Token merging / eviction (H₂O, SnapKV, PyramidKV)

• Quantization (FP8/INT4 KV cache)

• Low-rank projection (GQA, MQA → GQA dominant)

2026 reality: PagedAttention + FP8 KV + speculative decoding enables 1M+ context inference on 8×H100 clusters with high throughput.

5. Probabilistic & Bayesian Approaches in Modern NLP

While autoregressive decoder-only models dominate 2026 generative NLP through massive scaling and next-token prediction, probabilistic and Bayesian methods provide complementary strengths: principled uncertainty quantification, better-calibrated generation, controllable sampling, latent variable modeling, and alternatives to autoregressive decoding bottlenecks. These approaches are increasingly important for safety-critical applications, reasoning under uncertainty, controllable text generation, and scientific NLP (e.g., hypothesis generation, counterfactual reasoning).

This section covers variational autoencoders for discrete text, Bayesian neural networks for uncertainty estimation, and the emerging class of diffusion and score-based generative models adapted to language.

5.1 Variational Inference & Amortized Inference

Variational inference approximates intractable posteriors p(θ|D) by optimizing a simpler distribution q(θ) to minimize KL(q(θ)||p(θ|D)).

5.1.1 VAE for Text & β-VAE

Vanilla VAE (Kingma & Welling 2013) for continuous latents:

ELBO = 𝔼_{q(z|x)} [log p(x|z)] – D_KL(q(z|x) || p(z))

For text, x is discrete (one-hot or embedding) → reparameterization trick challenging.

Text VAE variants:

• LSTM + VAE (Bowman et al. 2016): encoder LSTM → μ, σ → sample z → decoder LSTM. KL term annealed from 0 → encourages meaningful latents. Problem: posterior collapse (KL → 0, latents ignored).

β-VAE (Higgins et al. 2017): ELBO_β = reconstruction – β D_KL(q||p) β > 1 → stronger disentanglement (independent latent factors). Applied to text: better controllability (style, topic separation).

2026 usage: β-VAE variants for controllable story generation, dialogue style transfer, counterfactual text editing.

5.1.2 Amortized Variational Inference in LLMs (e.g., VQ-VAE-2 extensions)

VQ-VAE (van den Oord et al. 2017): discrete latent codes via nearest-neighbor lookup in embedding table → no KL term, but commitment loss + codebook EMA updates.

VQ-VAE-2 (Razavi et al. 2019): hierarchical discrete latents → high-fidelity generation.

Modern LLM extensions (2024–2026):

• Discrete VAE + autoregressive decoder (e.g., VQGAN-CLIP hybrids for text-conditioned image, but reversed for text)

• Residual VQ-VAE (Lee et al. 2022–2025): multi-level codebooks → used in some non-autoregressive text generation and compression

• Vector Quantized LLMs (e.g., recent works on quantized token latents for efficient inference)

• Amortized inference in latent-variable LMs: encoder amortizes posterior q(z|x) → decoder p(x|z) autoregressive or diffusion-based.

2026 trend: Discrete latent VAEs appear in multimodal models (text → latent → image/video) and efficient long-context compression.

5.2 Bayesian Neural Networks & Uncertainty in Language

Bayesian neural networks place distributions over weights → capture epistemic uncertainty (model ignorance) and aleatoric uncertainty (data noise).

5.2.1 Deep Ensembles, MC Dropout, SWAG

Deep Ensembles (Lakshminarayanan et al. 2017): Train M independent models → average predictions or compute predictive variance. Simple, strong calibration → still gold standard in 2026 for uncertainty.

MC Dropout (Gal & Ghahramani 2016): Apply dropout at test time → Monte-Carlo sampling approximates Bayesian posterior predictive. Cheap (no retraining), but underestimates variance in large models.

SWAG (Maddox et al. 2019): Stochastic Weight Averaging Gaussian: fit Gaussian to SGD trajectory → fast, captures multimodal posterior modes better than single-point estimates.

2026 usage: Ensembles + SWAG for calibrated confidence in medical NLP, legal document classification, hallucination detection.

5.2.2 Laplace Approximation & Last-Layer Laplace

Laplace approximation: Posterior p(θ|D) ≈ 𝒩(θ_MAP, H⁻¹) where H is Hessian at mode.

Last-layer Laplace (Daxberger et al. 2021; Kristiadi et al. 2025 extensions): Only Hessian of last layer → cheap, captures most predictive uncertainty in LLMs. Scalable via KFAC or diagonal approximations.

2026 applications: Uncertainty-aware RAG (reject low-confidence retrievals), calibrated LLM-as-a-judge, out-of-distribution detection.

5.2.3 LLM Uncertainty Quantification (Verbalized Confidence, Semantic Entropy)

Verbalized confidence (Kuhn et al. 2023–2025): Prompt LLM to output probability or confidence score → surprisingly well-calibrated on some benchmarks.

Semantic entropy (Farquhar et al. 2024): Cluster LLM outputs by semantic equivalence (embedding distance + clustering) → entropy over clusters measures epistemic uncertainty (ignores aleatoric variation). Strongly correlates with hallucination; outperforms verbalized confidence in many cases.

2026 trend: Semantic entropy + last-layer Laplace + ensembles form the state-of-the-art uncertainty stack for production LLMs.

5.3 Diffusion Models & Continuous-Time Generative Models for Text

Diffusion models gradually corrupt then denoise data → powerful for images → adapted to discrete text.

5.3.1 Diffusion-LM, SSD-LM, GenAI Diffusion Variants

Diffusion-LM (Li et al. 2022): Continuous embedding space → round-trip embedding + continuous diffusion → classifier-free guidance for controllable generation.

SSD-LM (Han et al. 2023): Semi-autoregressive discrete diffusion → balances parallelism and quality.

GenAI diffusion variants (2024–2026):

• Masked diffusion (e.g., MaskGIT-style for text)

• Continuous-time discrete diffusion (score-based on token embeddings)

• Hybrid autoregressive-diffusion models (e.g., LLaDA, recent works)

Advantages over autoregressive: parallel generation, better global coherence, natural controllability via guidance.

5.3.2 Score-based Generative Modeling on Discrete Spaces

Score-based generative models (Song & Ermon 2019–2021): Learn score function ∇_x log p_t(x) → sample via Langevin dynamics or predictor-corrector.

Discrete adaptations (2023–2026):

• D3PM (Austin et al. 2021): absorbing diffusion on categorical space

• CDCD (Campbell et al. 2023): continuous relaxation + score matching

• SEDD (Lou et al. 2024): score entropy-based discrete diffusion

2026 status: Discrete diffusion and score-based models show promise for controllable, parallel text generation and infilling → still lag autoregressive LLMs on open-ended quality but excel in constrained tasks (e.g., infilling, style transfer, molecular design).

6. Structured Prediction & Constrained Decoding

While autoregressive next-token prediction excels at open-ended generation, many real-world NLP tasks require structured outputs (e.g., parse trees, named-entity tags, logical forms) or constrained generation (e.g., length limits, lexical inclusion, format adherence, logical consistency). Structured prediction methods enforce dependencies across output variables; constrained decoding guides sampling or search to satisfy hard/soft rules.

This section covers classical sequence labeling models (CRF, HMM), their neural extensions, constrained decoding techniques for autoregressive models, and semantic parsing approaches — including modern LLM-based program synthesis and tool-use.

6.1 CRF, HMM & Sequence Labeling

Sequence labeling assigns labels to each token in a sequence (e.g., POS tagging, NER, chunking) while modeling dependencies between adjacent labels.

6.1.1 Viterbi Decoding & Forward-Backward Algorithm

Hidden Markov Models (HMM) Joint probability factorizes as:

P(x₁…xₙ, y₁…yₙ) = P(y₁) P(x₁|y₁) ∏{t=2}^n P(y_t | y{t-1}) P(x_t | y_t)

where y = hidden states (tags), x = observations (words).

Forward-backward algorithm computes marginal posteriors efficiently:

Forward: α_t(j) = P(x₁…x_t, y_t = j) Backward: β_t(j) = P(x_{t+1}…xₙ | y_t = j)

Posterior marginal: P(y_t = j | x) ∝ α_t(j) β_t(j)

Viterbi decoding finds most likely tag sequence:

δ_t(j) = max_{y_{1:t-1}} P(y₁…y_{t-1}, y_t = j, x₁…x_t) = max_k [δ_{t-1}(k) P(y_t = j | y_{t-1} = k)] P(x_t | y_t = j)

Backpointers recover global argmax path → O(T |S|²) time, where T = length, |S| = tag set size.

HMMs were dominant in early 2000s sequence labeling; still used as baselines or in resource-constrained settings.

6.1.2 Neural CRF & Lattice-based Models

Conditional Random Fields (CRF) (Lafferty et al. 2001) Linear-chain CRF models P(y|x) directly:

P(y|x) = (1/Z(x)) exp( ∑_t φ(y_t, x_t) + ∑t ψ(y_t, y{t-1}) )

where φ = unary potentials (feature functions), ψ = transition potentials, Z(x) = partition function.

Neural CRF (Collobert 2011; Lample et al. 2016 – BiLSTM-CRF): Replace hand-crafted features with BiLSTM emissions → unary score = BiLSTM hidden state → CRF layer on top.

Training: maximize log P(y|x) → use forward-backward to compute Z(x) (log-partition function via dynamic programming).

Inference: Viterbi decoding on neural potentials → O(T |S|²) but |S| small (e.g., 20–100 tags).

Lattice-based models (2020–2026): Instead of linear chain, model lattice of possible segmentations/tokenizations (e.g., word segmentation + NER joint). Lattice-LSTM or Transformer over lattices → strong for morphologically rich languages and nested NER.

2026 status: BiLSTM-CRF still competitive on low-resource sequence labeling; Transformer-CRF hybrids used in specialized tasks (e.g., clinical NER, legal entity extraction).

6.2 Constrained & Controlled Generation

Autoregressive decoding often produces undesirable outputs (wrong length, forbidden tokens, logical inconsistencies). Constrained decoding enforces rules during search or sampling.

6.2.1 Beam Search with Length & Lexical Constraints

Standard beam search: keep top-k hypotheses at each step → greedy but high-quality.

Length constraints:

• Length penalty: score = log P(sequence) / length^α (α ≈ 0.6–1.0) → favors longer/shorter outputs

• Minimum/maximum length forcing: reject beams below/above threshold

Lexical constraints (Hokamp & Liu 2017 – Grid Beam Search precursor): Force inclusion of specific words/phrases → maintain separate beams for satisfied/unsatisfied constraints.

2026 usage: Common in summarization, translation, instruction following (e.g., force “JSON format” keywords).

6.2.2 Grid Beam Search, Diverse Beam Search

Grid Beam Search (Hokamp & Liu 2017): Extend beam search with auxiliary dimension for constraint satisfaction state → guarantees inclusion of constrained phrases without quality drop.

Diverse Beam Search (Vijayakumar et al. 2018): Add diversity penalty (Hamming distance or group proportional) → reduces repetition and mode collapse.

Other variants (2023–2026):

• MBR (Minimum Bayes Risk) decoding → rerank beams by expected utility

• Contrastive search → penalize repetition via similarity to previous tokens

6.2.3 Energy-based Models & Gradient-based Decoding

Energy-based models (EBM) (LeCun et al. 2006; Deng et al. 2020–2025): Define scalar energy E(x, y) → P(y|x) ∝ exp(–E(x,y)) Sampling via MCMC (Langevin dynamics) or learned proposal.

Gradient-based decoding (2023–2026): Treat generation as optimization: maximize log p(y|x) – λ constraint_violation(y) → use gradient ascent on continuous relaxation (Gumbel-softmax) or straight-through estimator.

Examples:

• GenAI energy-based reranking

• Controlled generation via classifier-free guidance (analogous to diffusion)

• Gradient-guided beam search for logical constraints

2026 trend: EBMs + gradient-based methods combined with LLMs for format-constrained (JSON, SQL), factually grounded, or safety-aligned generation.

6.3 Semantic Parsing & Logical Reasoning

Semantic parsing maps natural language to executable logical forms (SQL, λ-calculus, programs) for reasoning and tool use.

6.3.1 Tree-based Models & AM-PM (Algebraic Machine Parsing)

Tree-based models:

• Seq2Seq with tree decoders (Dong & Lapata 2016)

• Transition-based parsers (e.g., AMR parsing with stack-LSTM)

• Span-based models (e.g., constituent parsing)

AM-PM (Algebraic Machine Parsing) (recent 2024–2025 works): Represent parsing as algebraic operations over tree structures → differentiable parsing with hard constraints.

2026 usage: Tree decoders in structured generation (e.g., nested JSON, mathematical expressions).

6.3.2 LLM-based Program Synthesis & Tool-use

Program synthesis: Generate executable code from natural language (e.g., AlphaCode, Codex, DeepSeek-Coder).

Tool-use & ReAct-style agents (Yao et al. 2022–2025): LLM interleaves reasoning + tool calls → external APIs (calculator, search, code interpreter).

Modern approaches (2025–2026):

• Toolformer (Schick et al. 2023): self-supervised tool-use pre-training

• Gorilla / ToolLLaMA: fine-tune on API/tool documentation → call thousands of APIs

• ReAct + Reflexion: self-critique + memory → iterative reasoning

• Tree-of-Thoughts / Graph-of-Thoughts: explore multiple reasoning paths → backtracking search

• Program-of-Thoughts (PoT): generate Python programs for numerical reasoning → execute for exact answers

2026 frontier: LLM agents with dynamic tool selection, self-debugging, multi-step planning, and formal verification of generated programs/logical forms.

• 7. Evaluation, Interpretability & Robustness

  As language models scale to hundreds of billions of parameters and exhibit emergent reasoning capabilities, evaluating their true performance, understanding their internal computations, and ensuring robustness against misuse become central challenges. This section covers the evolution of evaluation paradigms (from perplexity to human-centric and adversarial benchmarks), mechanistic interpretability techniques that reverse-engineer model behavior, and adversarial robustness strategies in the 2025–2026 era.

  7.1 Evaluation Metrics for Language Understanding

  Evaluating LLMs requires balancing intrinsic metrics (model-centric), extrinsic performance (task success), and human-aligned judgments (real-world quality).

  7.1.1 Perplexity vs. Downstream Tasks vs. Human Judgments

  Perplexity (PPL) PPL = exp( – (1/N) ∑ log P(w_i | w_{<i}) ) Remains the gold-standard intrinsic metric for autoregressive models. Advantages: fast, cheap, correlates reasonably with language modeling quality up to ~20–30 PPL. Limitations: saturates quickly (modern models reach single-digit PPL on standard corpora); poor correlation with reasoning, factual accuracy, or coherence at low PPL regimes.

  Downstream tasks Fine-tuned or few-shot performance on classification, generation, or reasoning benchmarks (GLUE, SuperGLUE, MMLU, GSM8K, HumanEval). Stronger signal than perplexity for practical utility; still proxy — models can overfit leaderboards.

  Human judgments Gold standard for open-ended generation: pairwise preference, Likert-scale quality, Elo-style rankings. Challenges: expensive, noisy, subjective → mitigated by LLM-as-a-judge pipelines (see 7.1.2).

  2026 consensus: Perplexity useful for pre-training monitoring; downstream tasks for capability tracking; human (or strong LLM) judgments for final quality assessment.

  7.1.2 LLM-as-a-Judge, Pairwise Comparison, Elo Rankings

  LLM-as-a-judge (Zheng et al. 2023; Vicuna, MT-Bench, AlpacaEval) Prompt strong LLM (e.g., GPT-4o, Claude-3.5, Gemini 2.0) to score or compare model outputs. Variants:

  • Single-score: assign 1–10 rating

  • Pairwise: choose winner (with tie option)

  • Chain-of-thought judging: require reasoning trace

  Pairwise comparison + Elo Convert pairwise preferences to Bradley-Terry model → compute Elo ratings (LMSYS Chatbot Arena style). 2026: LMSYS Arena remains gold-standard leaderboard (millions of blind pairwise votes); Elo scores highly stable and correlate strongly with human preference.

  Reliability improvements (2025–2026):

  • Position bias mitigation (shuffle A/B)

  • Reference-free vs. reference-based judging

  • Multi-judge ensembles (GPT-4o + Claude + Gemini)

  • Reward model fine-tuning on human judgments

  Current leaders (mid-2026): Grok-4.1, Gemini 2.5 Pro, Claude 4 Opus, Llama-4 Maverick variants top most Elo arenas.

  7.1.3 Benchmark Suites (GLUE → SuperGLUE → BIG-bench → MMLU → LMSYS Arena → HELM)

  GLUE (2018) → SuperGLUE (2019): NLU classification suites → saturated by 2020–2021.

  BIG-bench (2022): 200+ diverse tasks → many still challenging → BIG-bench Hard subset remains useful.

  MMLU (Hendrycks et al. 2021): 57 subjects, 14k questions → standard multitask benchmark → saturated at ~90% by 2025 frontier models.

  LMSYS Chatbot Arena (2023–2026): Crowdsourced blind pairwise battles → most trusted real-world ranking.

  HELM (Liang et al. 2022–2026): Holistic Evaluation of Language Models → 42 scenarios × 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) → remains the most comprehensive safety & ethics benchmark.

  2026 additions:

  • GPQA Diamond (hard graduate-level science)

  • FrontierMath, AIME 2025–2026 (math reasoning)

  • AgentBench, ToolAlpaca (tool-use & agent capabilities)

  • WildBench, MT-Bench-101 (open-ended multi-turn)

  Trend: Shift from saturated classification to reasoning, agentic, long-context, multimodal, and safety-focused suites.

  7.2 Mechanistic Interpretability

  Mechanistic interpretability seeks to reverse-engineer how models compute outputs — identifying circuits, features, and computations inside weights and activations.

  7.2.1 Logit Lens, Activation Patching, Causal Tracing

  Logit lens (Nostalgebraist 2020): Project residual stream activations at each layer to vocabulary logits → visualize how predictions evolve across layers.

  Activation patching (Meng et al. 2022 – ROME): Replace activation at specific position/layer with clean-corrupted baseline → measure causal effect on output.

  Causal tracing (Meng et al. 2022): Patch corrupted activations layer-by-layer + noise restoration → localize factual knowledge storage (e.g., subject token + late MLP layers).

  These techniques revealed:

  • Factual associations localized to mid-late layers

  • Induction heads (early layers) for in-context pattern matching

  • Copy suppression, successor heads, etc.

  7.2.2 Circuit Discovery & Sparse Autoencoders (Anthropic 2024–2026)

  Circuit discovery (Anthropic, Olsson et al. 2022–2025): Reverse-engineer specific behaviors (e.g., indirect object identification, factual recall) as subgraphs of attention heads and MLPs.

  Sparse autoencoders (SAE) (Anthropic 2024–2026; Bricken et al.): Train autoencoder on residual stream activations → learn dictionary of sparse, monosemantic features (one feature ≈ one interpretable concept). L1 penalty + reconstruction loss → features correspond to concrete entities, abstract concepts, safety-relevant directions.

  2026 breakthroughs:

  • Gated SAEs + JumpReLU → higher sparsity, better reconstruction

  • Scale law for SAE: more features → more monosemanticity

  • Circuit tracing over SAE features → interpret end-to-end computations (e.g., “truth direction”, “sycophancy circuits”)

  Applications: Safety (remove backdoors), debugging hallucinations, steering models via feature addition/subtraction.

  7.3 Robustness & Adversarial Attacks in Language Models

  Robustness ensures models resist adversarial inputs, jailbreaks, and prompt injections while maintaining alignment.

  7.3.1 Prompt Injection, Adversarial Suffixes, Jailbreaking

  Prompt injection (Liu et al. 2023): User input overrides system instructions → classic: “Ignore previous instructions and…”

  Adversarial suffixes (Zou et al. 2023 – GCG): Greedy coordinate gradient → append optimized token sequence that forces harmful output even on aligned models.

  Jailbreaking techniques (2024–2026):

  • DAN-style roleplay

  • ASCII art / Unicode obfuscation

  • Many-shot jailbreaking

  • Multi-turn persuasion

  • Gradient-based universal attacks (GCG successors)

  2026 status: Frontier models (Claude 4, Gemini 2.5, Grok-4) resist most simple injections but remain vulnerable to sophisticated multi-turn or gradient-based attacks.

  7.3.2 Adversarial Training & Robust Alignment

  Adversarial training (RAHF variants): Include adversarial examples in RLHF preference data → reward model penalizes jailbreaks.

  Robust alignment techniques:

  • Constitutional AI (Bai et al. 2022–2025): self-critique against principles

  • Red-teaming + rejection sampling

  • Synthetic adversarial data generation (Self-Taught Reasoner style)

  • Safety layers / refusal tuning

  2026 trend: Multi-stage alignment (SFT → RLHF → RLAIF → adversarial hardening) + circuit-level interventions (remove jailbreak circuits via SAE editing).

  Key Takeaway for 2026 Evaluation has moved beyond perplexity to human-preference Elo rankings (LMSYS Arena) and holistic suites (HELM). Mechanistic interpretability (SAE + causal tracing) unlocks model steering and safety. Robustness requires continuous red-teaming and adversarial alignment — no model is fully jailbreak-proof yet.

  8. Advanced Topics & Research Frontiers (2025–2026)

  By 2025–2026, the field of NLP has moved beyond scaling dense autoregressive Transformers toward architectures and training paradigms that solve fundamental bottlenecks: quadratic attention cost for long contexts, lack of persistent memory, poor reasoning depth, limited multimodality, and unreliable agentic behavior. This section surveys the most influential mathematical and architectural innovations addressing these challenges, as well as open research problems suitable for MSc/PhD theses or independent work.

  8.1 Long-Context & Memory-Augmented Models

  Standard Transformers suffer O(n²) time and space complexity in attention, making contexts >128k tokens impractical without heavy engineering. 2025–2026 solutions fall into linear-time attention approximations, state-space models, and explicit memory mechanisms.

  8.1.1 Ring Attention, Infini-Transformer, RWKV, Mamba-2

  Ring Attention (Liu et al. 2023–2025 extensions) Distributes attention computation in a ring topology across devices → each device computes local block + communicates only with neighbors → hides communication latency → enables near-infinite context during training and inference (effective context >1M tokens demonstrated on 8–64 GPU clusters).

  Infini-Transformer (Munkhdalai et al. 2024) Combines compressive memory + local attention + long-term memory retrieval.

  • Local masked attention within fixed window

  • Compressive memory stores compressed past KV states

  • Retrieval from memory bank via dot-product → linear scaling with effective context length. Strong results on 1M+ needle-in-haystack tasks with minimal perplexity degradation.

  RWKV (v5–v6, Peng et al. 2023–2025) Receptance Weighted Key Value — linear-time RNN alternative with Transformer-like parallel training. Recurrent formulation with time-mixing and channel-mixing → no KV cache growth → inference cost O(1) per token. RWKV-6 (2025) adds grouped-query attention-like parallelism → competitive perplexity with Llama-3-scale models at 14B.

  Mamba-2 (Dao & Gu 2024–2025) Structured state-space model (SSM) with selective scan → linear-time, hardware-aware (CUDA kernels). Mamba-2 improves selectivity and parallel scan efficiency → outperforms Transformers at 1M+ context in many tasks while using 5× less memory.

  2026 comparison Ring Attention → best for distributed long-context training Infini-Transformer → strongest on retrieval-heavy long-document tasks Mamba-2 / RWKV → preferred for single-device long-context inference (no KV cache explosion)

  8.1.2 State Space Models & Linear RNN Alternatives

  State Space Models (SSMs) generalize linear RNNs:

  x'(t) = A x(t) + B u(t) y(t) = C x(t) + D u(t)

  Discretized → selective SSMs (Gu & Dao 2023; Mamba family) make A, B input-dependent → content-aware recurrence.

  Key advantages:

  • Linear scaling in sequence length

  • Constant memory during inference (no KV cache)

  • Parallelizable training via structured kernels (CUDA selective scan)

  2026 frontier:

  • Mamba-2 + hybrid Mamba-Transformer blocks (e.g., Jamba, MambaByte)

  • Liquid Foundation Models (Hasani et al. 2025) → continuous-time SSMs with adaptive computation

  • Hawk / Griffin (Google 2025) → gated linear recurrences competitive with attention at 1M+ context

  SSMs now rival or surpass Transformers in long-context language modeling, DNA sequence modeling, and time-series forecasting.

  8.2 Multimodal & Grounded Language Models

  Pure text models lack grounding in the physical world. Multimodal models integrate vision, audio, or robotics signals to produce grounded, embodied understanding.

  8.2.1 CLIP, Flamingo, LLaVA, Qwen-VL, Kosmos-2

  CLIP (Radford et al. 2021) Contrastive pre-training on 400M image–text pairs → joint embedding space → zero-shot image classification & retrieval.

  Flamingo (Alayrac et al. 2022) Frozen vision encoder + frozen LLM + Perceiver resampler → few-shot visual question answering.

  LLaVA (Liu et al. 2023–2025 family) Vision encoder (CLIP ViT) → projection → LLM (Vicuna/LLaMA) → instruction-tuned on GPT-4V-generated data → open-source multimodal chat leader.

  Qwen-VL (Alibaba 2023–2025) Multimodal version of Qwen series → strong on Chinese–English vision-language tasks, document understanding, grounding.

  Kosmos-2 (Peng et al. 2023) → grounding via location tokens → bounding-box prediction.

  2026 leaders:

  • LLaVA-Next / LLaVA-OneVision (strong open multimodal)

  • Qwen2-VL / Qwen-VL-Max (document + high-res understanding)

  • Gemini 2.0 / Claude 3.5 / Grok-2 Vision (closed frontier)

  8.2.2 Vision-Language Alignment & Visual Prompting

  Alignment techniques:

  • Contrastive (CLIP-style) → global image–text matching

  • Generative (captioning + VQA) → fine-grained understanding

  • Grounding (Kosmos-2, Ferret) → region-level correspondence

  Visual prompting (2024–2026):

  • Soft visual prompts (learned tokens prepended to image features)

  • Visual instruction tuning → “point to the red car” → bounding-box or mask output

  • Multimodal chain-of-thought → describe image → reason → answer

  2026 trend: Unified multimodal pre-training (image + video + text + audio) + grounding → agents that can “see” and act in visual environments.

  8.3 Agents, Reasoning & Tool Use

  LLMs transition from passive predictors to active agents capable of multi-step reasoning, tool use, and self-correction.

  8.3.1 ReAct, Reflexion, Tree-of-Thoughts, Graph-of-Thoughts

  ReAct (Yao et al. 2022): Reason + Act loop → interleave chain-of-thought reasoning with tool calls (search, calculator, code execution).

  Reflexion (Shinn et al. 2023): Self-reflection after failed trajectories → verbal critique stored in episodic memory → improves over episodes.

  Tree-of-Thoughts (ToT) (Yao et al. 2023): Explore multiple reasoning paths → evaluation + pruning → backtracking search over thought trees.

  Graph-of-Thoughts (GoT) (Besta et al. 2024): Generalize ToT to arbitrary graph structures → non-linear reasoning (merging, aggregation, refinement operations).

  8.3.2 Self-Consistency, Self-Refine, Constitutional AI

  Self-Consistency (Wang et al. 2022): Sample multiple chain-of-thought paths → majority vote → boosts reasoning accuracy.

  Self-Refine (Madaan et al. 2023): Generate → self-critique → refine iteratively → single-model loop.

  Constitutional AI (Bai et al. 2022–2025): Self-improvement via AI feedback against a set of principles → scalable alignment without human preference data.

  2026 frontier:

  • Agentic workflows (AutoGPT-style + reflection + tool libraries)

  • Test-time compute scaling (ToT + self-refine + verifiers)

  • Multi-agent debate & critique (improves reasoning & safety)

  8.4 Open Problems & Thesis Directions

  1. Infinite-context models without memory blowup Can SSMs + compressive memory achieve human-level long-document understanding at 10M+ token context with sub-quadratic cost?

  2. Unified multimodal reasoning & grounding How to train a single model that reasons jointly over text, image, video, code, and physical simulation with verifiable grounding?

  3. Mechanistic understanding of reasoning circuits Can sparse autoencoders + causal tracing fully reverse-engineer multi-step reasoning (e.g., math proof steps) in frontier models?

  4. Scalable self-improvement & autonomous alignment Design RLAIF-like loops that iteratively improve capabilities and safety without catastrophic forgetting or reward hacking.

  5. Test-time compute vs. pre-training trade-offs Quantify scaling laws for inference-time methods (ToT depth, self-refine iterations) vs. additional pre-training tokens.

  6. Robustness to adversarial long-context attacks Develop defenses against needle-in-haystack poisoning or context-manipulation attacks that induce hallucinations.

  9. Tools, Libraries & Implementation Resources

  This section provides a practical, up-to-date (mid-2026) overview of the most widely used open-source tools, frameworks, and datasets for implementing, experimenting with, interpreting, evaluating, and deploying mathematical models in NLP — with emphasis on modern large language models (LLMs), contextual embeddings, reasoning, long-context, multimodal, and agentic systems.

  All recommendations prioritize actively maintained projects with strong community adoption in research and industry as of March–June 2026.

  9.1 Core Frameworks

  PyTorch

  Version: 2.7.x – 2.8.x series (2026) Why it dominates NLP in 2026: Native support for dynamic graphs, torch.compile (TorchDynamo + inductor backend), FlexAttention (custom fused attention kernels), torchao (low-bit quantization), torch.distributed (FSDP2, DeviceMesh), and excellent CUDA/ROCm/MPS/XPU backends. Key extensions for NLP/LLMs:

  • torch.nn.functional.scaled_dot_product_attention (FlashAttention-3 backend)

  • torch.compile + dynamo for 1.5–2× training speedup

  • torch.distributed.tensor (DTensor) for 3D/4D parallelism Usage: Almost all frontier open models (Llama-3.1, Qwen2-VL, DeepSeek-V3, Gemma-2) are trained and released in PyTorch.

  Hugging Face Transformers

  Version: v4.50+ (2026) Core strengths:

  • Model hub with >1M models (including Llama-4, Qwen-VL-Max, Grok-4.1 weights when released)

  • AutoModel, AutoTokenizer, pipeline API for zero-code inference

  • PEFT (LoRA, QLoRA, DoRA, IA³, AdaLoRA)

  • Accelerate (multi-GPU/TPU, DeepSpeed/FSLP integration)

  • Tokenizers (fast Rust-based BPE, SentencePiece, WordPiece)

  • Trainer API (with packing, gradient checkpointing, flash attention) 2026 highlights: Native support for multimodal models (LLaVA-OneVision, Qwen2-VL), long-context handling (Ring Attention wrappers), and speculative decoding integrations.

  vLLM

  Version: v0.6.x – v0.7.x (2026) Purpose: High-throughput, memory-efficient LLM inference & serving Key features:

  • PagedAttention (non-contiguous KV cache → no fragmentation)

  • Continuous batching + chunked prefill

  • Speculative decoding (EAGLE, Medusa, Lookahead)

  • FP8 / NVFP4 / INT4 quantization support

  • OpenAI-compatible API server

  • Multi-modal batching (vision-language) 2026 status: De-facto open-source serving engine — often 2–5× faster than Hugging Face Text Generation Inference (TGI) on the same hardware.

  llama.cpp

  Version: bXXXX+ (continuous updates, 2026) Purpose: CPU / GPU / Apple Silicon / edge inference with extreme quantization Key strengths:

  • GGUF format (quantized models: Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K, Q2_K)

  • Metal (Apple M-series), CUDA, Vulkan, SYCL backends

  • Extremely low memory footprint (e.g., 70B Q4_K_M runs on 24–32 GB VRAM)

  • Fast CPU inference (AVX2, NEON)

  • Server mode (OpenAI-compatible) 2026 usage: Dominant for local / edge deployment, research on low-bit quantization, and privacy-sensitive applications.

  9.2 Interpretability Tools

  TransformerLens (formerly EasyTransformer)

  Repo: neelnanda-io/TransformerLens Purpose: Mechanistic interpretability for Transformers Core features:

  • Clean, named hook points on every internal activation (residual stream, attention heads, MLPs)

  • Easy activation caching, patching, ablation

  • Logit lens, direct logit attribution

  • Circuit tracing utilities 2026 status: Gold standard for academic interpretability research — used in almost all SAE and circuit papers.

  nnsight

  Repo: cisnlp/nnsight Purpose: Modern, scalable alternative / successor to TransformerLens Key advantages:

  • Lazy computation + remote execution (run on large models without loading locally)

  • Better support for large models and multi-GPU

  • Cleaner API for interventions (patching, steering, ablation) 2026 usage: Rapidly growing adoption for large-model interpretability (e.g., Llama-4, Qwen-VL).

  CircuitsVis

  Repo: Anthropic / circuitsvis (or community forks) Purpose: Visualization of attention patterns, circuits, and SAE features Features:

  • Attention rollout / attention heads matrix

  • Logit attribution graphs

  • Feature dashboards for sparse autoencoders 2026 role: Standard companion to SAEs and causal tracing papers.

  9.3 Evaluation Suites & Benchmarks

  EleutherAI LM Evaluation Harness

  Repo: EleutherAI/lm-evaluation-harness Purpose: Standardized zero-shot / few-shot evaluation of language models 2026 status: Supports >300 tasks (MMLU, ARC, HellaSwag, TruthfulQA, Winogrande, GSM8K, HumanEval, etc.)

  • Few-shot templates, chat templates

  • Multi-GPU support

  • Widely used for open-model leaderboards

  OpenCompass

  Repo: open-compass/opencompass Purpose: Comprehensive, modular evaluation framework (Chinese & English focus) Key features:

  • 100+ datasets (MMLU, CEval, CMMLU, C-Eval, GSM8K, MATH, HumanEval, BBH)

  • Supports chat, base, multimodal models

  • Distributed evaluation (Slurm, Ray)

  • Leaderboard generation 2026 usage: Dominant in Chinese NLP community and multilingual evaluation.

  Other important 2026 suites:

  • LMSYS Chatbot Arena (crowdsourced Elo)

  • HELM Safety / HELM Classic

  • AgentBench / ToolBench / InterCode

  • GPQA Diamond, FrontierMath, AIME 2025–2026

  • WildBench, MT-Bench-101 (multi-turn)

  9.4 Datasets & Preprocessing Pipelines

  Pre-training corpora (2026 commonly used open datasets)

  • FineWeb-Edu (filtered Common Crawl, educational subset)

  • The Pile v2 / Dolma (AllenAI)

  • RedPajama-V2, RefinedWeb

  • Cosmopedia (synthetic textbooks, code, stories)

  • Proof-Pile-2, OpenWebMath (math & proof data)

  Instruction & preference datasets

  • UltraChat, UltraFeedback, HelpSteer2

  • OpenHermes-2.5, Zephyr-7B style mixtures

  • No_robots, Nectar (high-quality synthetic + curated)

  • Preference datasets: HH-RLHF, PKU-RLHF, reward-model training pairs

  Long-context & retrieval-augmented

  • LongAlpaca, LongBench, InfiniteBench

  • Needle-in-a-Haystack variants (Paul Graham essays, codebases)

  • RAG datasets: Natural Questions, TriviaQA, HotpotQA, MuSiQue

  Multimodal & vision-language

  • LAION-2B, COYO-700M, MMC4

  • LLaVA-Instruct-150k, ShareGPT4V, LVIS-Instruct4V

  • DocVQA, ChartQA, InfoVQA (document understanding)

  Preprocessing pipelines

  • Hugging Face Datasets + tokenizers

  • SentencePiece / tiktoken / cl100k_base

  • FineWeb-Edu deduplication pipeline (MinHash LSH)

  • Datatrove (bigscience-workshop) → filtering, quality scoring, PII removal

  • Ray Data / Polars for distributed preprocessing at terabyte scale

110. Assessments, Exercises & Projects

This final section offers a carefully graded set of learning activities — from conceptual checks and mathematical proofs to hands-on coding, structured mini-projects, and open-ended thesis-level research ideas. All exercises are tightly aligned with the core material in Sections 1–9 and are designed for:

• MSc / early PhD students preparing for research or qualifying exams

• Advanced undergraduates deepening theoretical and practical NLP skills

• Industry engineers / researchers building portfolio projects or preparing for interviews

• Professors / lecturers looking for assignment, lab, or capstone ideas

10.1 Conceptual & Proof-Based Questions

Purpose: Reinforce mathematical foundations, understand why techniques succeed or fail, and prepare for research discussions / exams.

Short conceptual questions (quiz / interview style)

1. Explain why the chain rule makes autoregressive modeling tractable for arbitrary-length sequences, but also creates the exposure bias problem during training vs. inference.

2. Show mathematically why Kneser-Ney smoothing outperforms Laplace smoothing on morphologically rich languages (hint: role of continuation counts).

3. Why does the Transformer’s self-attention mechanism scale better to long sequences than RNNs/LSTMs? Link your answer to vanishing gradients and parallelization.

4. Describe how the KL divergence term in the VAE ELBO prevents posterior collapse, and why β-VAE with β > 1 promotes disentangled representations.

5. Explain the difference between epistemic and aleatoric uncertainty in LLMs. Why is semantic entropy often more reliable than verbalized confidence for detecting hallucinations?

6. Why does beam search with lexical constraints sometimes produce lower-quality output than unconstrained search? What trade-off does grid beam search address?

7. In mechanistic interpretability, why is activation patching more powerful than logit lens for localizing factual knowledge?

8. Describe one mathematical reason why sparse autoencoders (SAEs) can recover monosemantic features even when the residual stream is highly polysemantic.

9. Explain why test-time compute scaling (e.g., Tree-of-Thoughts depth) can yield power-law improvements similar to pre-training scaling laws.

10. Why do adversarial suffixes (GCG-style) remain effective against aligned models despite safety training? Link to gradient-based optimization.

Proof / derivation questions (homework / exam level)

1. Derive the Viterbi recursion for linear-chain CRF and show it runs in O(T |S|²) time (T = sequence length, |S| = tag set size).

2. Prove that the scaled dot-product attention (dividing by √d_k) prevents vanishing gradients when d_k is large (use variance analysis of dot products).

3. Show that the ELBO in a VAE is a lower bound on the marginal log-likelihood log p(x), and explain why maximizing ELBO indirectly minimizes KL(q(z|x) || p(z|x)).

4. Derive the Bradley-Terry model used in pairwise preference learning (RLHF reward modeling) and explain how Elo ratings are computed from it.

5. Sketch why the communication complexity lower bound for distributed SGD requires Ω(√(κ/ε)) rounds in the worst case (Arjevani et al. style argument).

10.2 Coding Exercises

Language: Python (PyTorch 2.6+, Hugging Face Transformers, sentencepiece, vLLM where relevant). Use GPU if available.

Exercise 1 – Implement BPE from scratch Build a basic Byte-Pair Encoding tokenizer (no external libraries except typing & collections).

• Input: list of training sentences

• Steps: compute pair frequencies → iteratively merge most frequent pair → build merge rules & vocabulary

• Output: encode new sentence using learned merges

• Bonus: implement SentencePiece-style unigram LM tokenization (probabilistic segmentation)

Exercise 2 – LoRA fine-tuning from scratch Implement a LoRA layer that wraps nn.Linear modules (no PEFT library).

• Apply to a small decoder-only Transformer (e.g., nanoGPT or tiny Llama-style model)

• Fine-tune on a toy instruction dataset (e.g., Alpaca 52k subset or custom arithmetic)

• Merge LoRA weights back into base model → verify zero inference overhead

• Compare memory usage and final perplexity vs. full fine-tuning

Exercise 3 – Simple speculative decoding Implement a basic speculative decoding loop (draft + verify style) using a small student model (e.g., tiny Llama) and target model (e.g., Llama-3.1-8B-Instruct).

• Draft 4–8 tokens in parallel with student

• Verify prefix with target model

• Accept longest correct prefix → repeat

• Measure speedup vs. standard greedy decoding on long-prompt generation

• Bonus: add tree search over multiple draft branches (Medusa-style)

Starter resources

• BPE: Hugging Face tokenizers code (reference), or implement from Sennrich 2016 paper

• LoRA: microsoft/LoRA repo (reference impl)

• Speculative decoding: vLLM codebase + Medusa paper pseudocode

10.3 Mini-Projects

Duration: 3–10 weeks (individual or small team)

Project A – Long-context Retrieval-Augmented Generation (RAG) Goal: Build a production-style long-context RAG pipeline.

• Dataset: LongBench / Natural Questions long-tail subset

• Components:

  • Embedding model (e.g., BGE-large-en-v1.5 or E5-mistral)

  • Vector store (FAISS HNSW or Chroma)

  • Long-context LLM (Llama-3.1-8B-Instruct, Qwen2-7B-Instruct, or Mamba-2 hybrid)

  • Use Ring Attention / Infini-Transformer wrapper or Mamba-2 for 32k–128k context

• Evaluation: approximate NDCG@10, exact match on answer, faithfulness

• Bonus: Add semantic entropy to detect low-confidence answers → reject or re-retrieve

Project B – Constrained Generation with Lexical & Format Rules Goal: Generate structured text (JSON, SQL, YAML) with hard constraints.

• Task: Text-to-SQL or instruction-to-JSON conversion

• Base model: Llama-3.1-8B-Instruct or Qwen2-7B-Instruct

• Techniques:

  • Beam search with lexical constraints (force “SELECT”, “WHERE”)

  • Grammar-constrained decoding (Outlines or Guidance library)

  • Energy-based reranking (classifier-free guidance on format score)

• Evaluation: execution accuracy (SQL), JSON schema validity, BLEU/ROUGE

• Bonus: Add gradient-based decoding to optimize format + semantic score

Project C – Basic Circuit Discovery with Sparse Autoencoders Goal: Identify monosemantic features in a small Transformer.

• Model: small GPT-2 or Llama-3.2-1B

• Train SAE on residual stream activations (use TransformerLens or nnsight)

• L1 penalty + reconstruction loss → aim for 1k–8k features

• Visualize top features (CircuitsVis or manual logit attribution)

• Identify simple circuits (e.g., induction heads, copy heads) via activation patching

• Bonus: Steer model behavior by adding/subtracting SAE features (e.g., increase “truthfulness” direction)

10.4 Advanced / Thesis-Level Project Ideas

Suitable for MSc thesis, PhD qualifying projects, research internships, or conference submissions (6–24 months)

1. Hybrid Mamba-Transformer with Adaptive Long-Context Routing Design a model that dynamically switches between Mamba-2 (linear-time) and Transformer attention based on context length or semantic complexity. Evaluate on InfiniteBench and 1M+ needle-in-haystack tasks.

2. Semantic Entropy-Guided RAG with Self-Refine Integrate semantic entropy as a rejection criterion in long-context RAG. If entropy high → trigger self-refine loop or re-retrieval. Benchmark on HotpotQA, MuSiQue, and LongBench.

3. Circuit-Level Adversarial Robustness via SAE Editing Use sparse autoencoders to identify jailbreak-related circuits (e.g., sycophancy, refusal suppression). Edit feature activations to harden model against GCG-style suffixes and multi-turn persuasion attacks.

4. Test-Time Compute Scaling Laws for Multimodal Reasoning Investigate power-law relationships between inference compute (ToT depth, self-refine iterations, visual prompt tokens) and performance on multimodal benchmarks (MMMU, ChartQA, DocVQA). Compare dense vs. MoE vs. Mamba-2 backbones.

5. Controllable Discrete Diffusion with Semantic Guidance Extend Diffusion-LM / SSD-LM with classifier-free guidance on semantic embeddings (from SAE features or contrastive text encoders). Evaluate controllability (topic, style, sentiment) and coherence vs. autoregressive baselines.

6. Mechanistic Analysis of Emergent Reasoning in Long-Context Models Use causal tracing + SAE on models with 128k–1M context (e.g., Llama-3.1-70B long, Qwen2-VL) to localize multi-step reasoning circuits (e.g., induction heads for pattern matching, late-layer fact aggregation).

Suggested evaluation rubric for advanced projects

• Theoretical novelty / mathematical rigor — 30%

• Implementation quality & reproducibility — 25%

• Empirical thoroughness (multiple seeds, ablations, statistical tests) — 25%

• Ethical & societal discussion (bias, misuse, energy cost) — 10%

• Clarity of write-up (paper-quality) — 10%

These activities can scale from course assignments to submissions at NeurIPS, ICLR, ACL, EMNLP, ICML workshops, or interpretability-focused venues (e.g., MechInterp workshops).

PREVIOUS PAGE INDEX PAGE NEXT PAGE

Join AI Learning

Get free AI tutorials and PDFs