TSM 2025–2026 Course Summary 1
Time Series Models: Course Summary
Mathematical Concepts & Key Intuitions, Weeks 1–6
TSM 2025–2026
The Course in One Sentence
Given noisy observations yt , estimate the unobserved signal αt using the state space framework:
the Kalman filter/smoother solves the linear Gaussian case exactly; linearization (EKF), mode
estimation, importance sampling, and particle filtering extend the approach to nonlinear and
non-Gaussian models.
Complete concept map (forward references to relevant sections):
LGM (Weeks 1–2) nLGM (Week 3) nLnGM (Weeks 4–5)
Model specification
Observation eq. yt = dt +Zt αt +εt , yt = Zt (αt ) + εt , yt ∼ p(yt |θt ),
εt Gaussian εt Gaussian arbitrary density
State equation αt+1 = ct + Tt αt + Rt ηt , or nonlinear Tt (αt ) for nLGM same as LGM
Signal θt dt + Zt αt (linear) Zt (αt ) (nonlinear) dt + Zt αt (linear)
Posterior p(θ|Yn ) Gaussian (exact) ≈ Gaussian non-Gaussian
Signal extraction
Filtering E[αt |Yt ] KF (§1.3.1, §2.3.1) EKF (§3.3.2) Bootstrap filter (§5.4)
Smoothing KS (§1.3.2, §2.3.2) EKS (§3.3.2) Mode est. + IS (§4.5, §5.2.4)
E[αt |Yn ]
Nonlinear Sim. smoothing (§3.2.4) — IS (§5.2.4)
E[h(α)|Yn ]
Mode θ̌ = θ̂ (mean=mode) ≈ θ̂ Newton–Raphson (§4.5)
Parameter estimation
PED: −21 (log Ft + vt2/Ft)
P
Log-likelihood Approx. PED via EKF p(Yn ) = g(Yn )·Eg [w] (§5.2.4)
ℓ(ψ) (§1.3.3)
Diffuse ℓd Drop first d terms (§1.3.3) same same
Initialization Stat./diffuse/mixed (§2.4.1) same same
√ ?
Diagnostics et = vt / Ft ∼ N (0, 1) same (approx.) —
Key mathematical tools
Core derivation Lemmas I & II (§1.5) Taylor / Jacobian Żt Newton–Raphson (§4.5)
(§3.3.2)
Simulation tool Lemma III (§3.2.4) — Lemma III via SPDK (§5.2)
Approximation exact (no approx.) Linearise: Zt → Żt Surrogate LGM: zt , At
strategy (§3.3.2) (§4.5)
Importance — — w = p(Yn |θ)/g(Yn |θ)
weights (§5.2.4)
Example models
LLM, AR(1)+noise, LLT, SV model via QML Poisson, Bernoulli (§4.2.1)
TV regression (§2.2.2) (§3.3.3)
Computational efficiency (Week 6, multivariate LGM)
Ht diagonal: univariate
treatment O(pm2 ) (§6.3.2)
Ht non-diag: transform
— —
+ UT (§6.3.3)
p ≫ m (DFM): collapsing
O(pm + m3 ) (§6.3.4)
,TSM 2025–2026 Course Summary 2
1 Week 1: Local Level Model & Signal Extraction
1.1 Overview
The Central Problem
Every observation is signal plus noise: yt = αt + εt . How do we recover the unobserved signal αt
from the noisy data? The local level model is the simplest state space model, and the Kalman
filter, smoother, and MLE developed here generalize directly to all models in the course.
1.2 The Local Level Model
Definition 1.1 (Local Level Model (LLM)).
yt = αt + εt , εt ∼ N (0, σε2 ), (1)
αt+1 = αt + ηt , ηt ∼ N (0, ση2 ), (2)
for t = 1, . . . , n, with α1 ∼ N (a1 , P1 ). Noise terms are mutually and serially independent.
The signal-to-noise ratio q = ση2 /σε2 controls behavior: q ↑ means the KF tracks data closely; q ↓
means smooth estimates.
The LLM is equivalent to ARIMA(0,1,1): ∆yt ∼ MA(1) with ρ1 = −1/(q + 2) ∈ [− 21 , 0).
Key Intuition: Signal + Noise Decomposition
Every observation is signal plus noise: yt = αt + εt . The entire course is about separating these
two, first in this simple model, then in progressively richer ones. Every technique developed for
the LLM generalizes directly to the general LGM in Week 2.
1.3 Inference Tools
1.3.1 The Kalman Filter
The KF is a forward recursion (t = 1, . . . , n) computing the predicted state (αt | Yt−1 ) ∼ N (at , Pt )
(note: at is a prediction, not a filtered estimate; the filtered estimate at|t = at +Kt vt is an intermediate
quantity):
vt = yt − at (prediction error) (3)
Ft = Pt + σε2 (prediction error variance) (4)
Kt = Pt /Ft (Kalman gain) (5)
at+1 = at + Kt vt (predicted state) (6)
Pt+1 = Kt σε2 + ση2 (predicted state variance) (7)
Key Intuition: Kalman Gain as Trust Allocation
The update at+1 = (1 − Kt ) at + Kt yt is a weighted average of prior and data. The Kalman
gain Kt = Pt /(Pt + σε2 ) is state uncertainty as a share of total uncertainty. When Pt ≫ σε2 : trust
the data (Kt ≈ 1). When Pt ≪ σε2 : trust the prior (Kt ≈ 0).
Directional intuition. Think of Kt as a discount factor controlling how aggressively at+1 gets
pulled toward yt . Shaky state estimate (Pt large) → the filter corrects strongly toward the data.
Noisy observations (σε2 large) → the new data point carries little information, so the filter largely
ignores it. Both forces feed into the same ratio Kt = Pt /Ft ; it is their relative magnitude that
determines which way the filter leans.
Pt converges to a steady-state value P̄ within ∼ 20 observations; after that, Kt → K̄ and Ft → F̄ ,
so the diffuse initialization effect fades quickly.
,TSM 2025–2026 Course Summary 3
1.3.2 The Kalman Smoother
The KS is a backward recursion (t = n, . . . , 1) using all data Yn . Initialize rn = 0, Nn = 0:
rt−1 = Ft−1 vt + (1 − Kt ) rt (8)
Nt−1 = Ft−1 + (1 − Kt ) Nt2
(9)
α̂t = at + Pt rt−1 = E[αt | Yn ] (10)
Vt = Pt − Pt2 Nt−1 = Var(αt | Yn ) (11)
Key Intuition: KF Estimate + Future Correction
The smoother adds a single correction to the filter’s answer: α̂t = at|t + Pt|t · rt , where rt is a
backward running total of precision-weighted future surprises vs /Fs , each discounted by (1 − Ks ):
how surprised the filter was
z }| {
vs ys − a s
= .
Fs Ps + σε2
| {z }
total variation in ys
Nt accumulates variance reductions from future data. The filter extrapolates from the past
(estimates lag behind level shifts); the smoother interpolates using past and future (estimates
are centered and smooth).
Kalman smoother adjusts at|t at t =1898 using future data
1400
yt (observations)
at|t (KF)
1200 α̂t (KS)
Pt|t ·rt
1000
Nile flow
800
600
1880 1900 1920 1940 1960
Year
Weight functions: the KF implicitly assigns weights wt,j to past observations. These decay
1
exponentially into the past (recent data matters most). The KS assigns bell-shaped weights centered
at t: observations both before and after t contribute, with influence fading symmetrically in both
directions.
1.3.3 Maximum Likelihood Estimation
MLE estimates the hyperparameters ψ = (σε2 , ση2 ), not the states αt . The states are estimated by
the KF/KS given ψ; MLE finds the ψ that makes the observed data most likely. The prediction error
, TSM 2025–2026 Course Summary 4
decomposition gives the log-likelihood directly from KF output:
n
vt2
n 1X
ℓ(ψ) = − log(2π) − log Ft + . (12)
2 2 Ft
t=1
Each term penalizes two things: log Ft penalizes uncertainty, vt2 /Ft penalizes surprise. Good parame-
ters balance both.
With diffuse initialization (P1 = κ → ∞), the first d terms become parameter-independent: F1 ≈
P1 → ∞, so log F1 → log κ (constant) and v12 /F1 → 0. The diffuse log-likelihood drops these terms:
n
vt2
n 1 X
ℓd (ψ) = − log(2π) − log Ft + . (13)
2 2 Ft
t=d+1
The only difference is the lower summation index: t = 1 vs t = d + 1 (where d = number of diffuse
states, typically d = 1 for the LLM). Parameter estimates are unaffected because the dropped terms
carry no information about ψ.
Key Intuition: The KF Already Computes the Likelihood
Q
By the chain rule, p(Yn ) = t p(yt |Yt−1 ). The KF computes each predictive density (yt |Yt−1 ) ∼
N (at , Ft ) as a byproduct. The KF is just the “inner loop” that evaluates the objective function:
2 (0) 2 (0) ′
1. Start with an initial guess ψ̂ (0) = (σε , ση ).
2. For each iteration k:
(a) Run the KF with current ψ̂ (k) to obtain vt , Ft for t = 1, . . . , n.
(b) Evaluate ℓd (ψ̂ (k) ).
(c) A numerical optimizer (e.g. BFGS) proposes ψ̂ (k+1) .
3. Repeat until convergence. The result is ψ̂ = arg maxψ ℓd (Yn ).
1.4 Practical Extensions
Normal KF Missing yj Forecasting (h steps)
Idea Update state with yt No data; skip update Treat future as missing
Kalman Kt = Pt /Ft Kj = 0 K = 0 at each step
gain
State at+1 = at + Kt vt aj+1 = aj an+h = an+1
Variance Pt+1 = Kt σε2 + ση2 Pj+1 = Pj + ση2 Pn+h = Pn+1 + (h−1)ση2
Effect Uncertainty reduced by data Uncertainty grows (+ση2 ) Uncertainty grows linearly
1.5 Mathematical Foundation: Lemmas I & II
X
µX
ΣXX ΣXY
Lemma 1.2 (Lemma I). If Y ∼N µY , , then:
ΣY X ΣY Y
E[X | Y = y] = µX + ΣXY Σ−1
Y Y (y − µY ), (14)
Var(X | Y = y) = ΣXX − ΣXY Σ−1
Y Y ΣY X . (15)
Time Series Models: Course Summary
Mathematical Concepts & Key Intuitions, Weeks 1–6
TSM 2025–2026
The Course in One Sentence
Given noisy observations yt , estimate the unobserved signal αt using the state space framework:
the Kalman filter/smoother solves the linear Gaussian case exactly; linearization (EKF), mode
estimation, importance sampling, and particle filtering extend the approach to nonlinear and
non-Gaussian models.
Complete concept map (forward references to relevant sections):
LGM (Weeks 1–2) nLGM (Week 3) nLnGM (Weeks 4–5)
Model specification
Observation eq. yt = dt +Zt αt +εt , yt = Zt (αt ) + εt , yt ∼ p(yt |θt ),
εt Gaussian εt Gaussian arbitrary density
State equation αt+1 = ct + Tt αt + Rt ηt , or nonlinear Tt (αt ) for nLGM same as LGM
Signal θt dt + Zt αt (linear) Zt (αt ) (nonlinear) dt + Zt αt (linear)
Posterior p(θ|Yn ) Gaussian (exact) ≈ Gaussian non-Gaussian
Signal extraction
Filtering E[αt |Yt ] KF (§1.3.1, §2.3.1) EKF (§3.3.2) Bootstrap filter (§5.4)
Smoothing KS (§1.3.2, §2.3.2) EKS (§3.3.2) Mode est. + IS (§4.5, §5.2.4)
E[αt |Yn ]
Nonlinear Sim. smoothing (§3.2.4) — IS (§5.2.4)
E[h(α)|Yn ]
Mode θ̌ = θ̂ (mean=mode) ≈ θ̂ Newton–Raphson (§4.5)
Parameter estimation
PED: −21 (log Ft + vt2/Ft)
P
Log-likelihood Approx. PED via EKF p(Yn ) = g(Yn )·Eg [w] (§5.2.4)
ℓ(ψ) (§1.3.3)
Diffuse ℓd Drop first d terms (§1.3.3) same same
Initialization Stat./diffuse/mixed (§2.4.1) same same
√ ?
Diagnostics et = vt / Ft ∼ N (0, 1) same (approx.) —
Key mathematical tools
Core derivation Lemmas I & II (§1.5) Taylor / Jacobian Żt Newton–Raphson (§4.5)
(§3.3.2)
Simulation tool Lemma III (§3.2.4) — Lemma III via SPDK (§5.2)
Approximation exact (no approx.) Linearise: Zt → Żt Surrogate LGM: zt , At
strategy (§3.3.2) (§4.5)
Importance — — w = p(Yn |θ)/g(Yn |θ)
weights (§5.2.4)
Example models
LLM, AR(1)+noise, LLT, SV model via QML Poisson, Bernoulli (§4.2.1)
TV regression (§2.2.2) (§3.3.3)
Computational efficiency (Week 6, multivariate LGM)
Ht diagonal: univariate
treatment O(pm2 ) (§6.3.2)
Ht non-diag: transform
— —
+ UT (§6.3.3)
p ≫ m (DFM): collapsing
O(pm + m3 ) (§6.3.4)
,TSM 2025–2026 Course Summary 2
1 Week 1: Local Level Model & Signal Extraction
1.1 Overview
The Central Problem
Every observation is signal plus noise: yt = αt + εt . How do we recover the unobserved signal αt
from the noisy data? The local level model is the simplest state space model, and the Kalman
filter, smoother, and MLE developed here generalize directly to all models in the course.
1.2 The Local Level Model
Definition 1.1 (Local Level Model (LLM)).
yt = αt + εt , εt ∼ N (0, σε2 ), (1)
αt+1 = αt + ηt , ηt ∼ N (0, ση2 ), (2)
for t = 1, . . . , n, with α1 ∼ N (a1 , P1 ). Noise terms are mutually and serially independent.
The signal-to-noise ratio q = ση2 /σε2 controls behavior: q ↑ means the KF tracks data closely; q ↓
means smooth estimates.
The LLM is equivalent to ARIMA(0,1,1): ∆yt ∼ MA(1) with ρ1 = −1/(q + 2) ∈ [− 21 , 0).
Key Intuition: Signal + Noise Decomposition
Every observation is signal plus noise: yt = αt + εt . The entire course is about separating these
two, first in this simple model, then in progressively richer ones. Every technique developed for
the LLM generalizes directly to the general LGM in Week 2.
1.3 Inference Tools
1.3.1 The Kalman Filter
The KF is a forward recursion (t = 1, . . . , n) computing the predicted state (αt | Yt−1 ) ∼ N (at , Pt )
(note: at is a prediction, not a filtered estimate; the filtered estimate at|t = at +Kt vt is an intermediate
quantity):
vt = yt − at (prediction error) (3)
Ft = Pt + σε2 (prediction error variance) (4)
Kt = Pt /Ft (Kalman gain) (5)
at+1 = at + Kt vt (predicted state) (6)
Pt+1 = Kt σε2 + ση2 (predicted state variance) (7)
Key Intuition: Kalman Gain as Trust Allocation
The update at+1 = (1 − Kt ) at + Kt yt is a weighted average of prior and data. The Kalman
gain Kt = Pt /(Pt + σε2 ) is state uncertainty as a share of total uncertainty. When Pt ≫ σε2 : trust
the data (Kt ≈ 1). When Pt ≪ σε2 : trust the prior (Kt ≈ 0).
Directional intuition. Think of Kt as a discount factor controlling how aggressively at+1 gets
pulled toward yt . Shaky state estimate (Pt large) → the filter corrects strongly toward the data.
Noisy observations (σε2 large) → the new data point carries little information, so the filter largely
ignores it. Both forces feed into the same ratio Kt = Pt /Ft ; it is their relative magnitude that
determines which way the filter leans.
Pt converges to a steady-state value P̄ within ∼ 20 observations; after that, Kt → K̄ and Ft → F̄ ,
so the diffuse initialization effect fades quickly.
,TSM 2025–2026 Course Summary 3
1.3.2 The Kalman Smoother
The KS is a backward recursion (t = n, . . . , 1) using all data Yn . Initialize rn = 0, Nn = 0:
rt−1 = Ft−1 vt + (1 − Kt ) rt (8)
Nt−1 = Ft−1 + (1 − Kt ) Nt2
(9)
α̂t = at + Pt rt−1 = E[αt | Yn ] (10)
Vt = Pt − Pt2 Nt−1 = Var(αt | Yn ) (11)
Key Intuition: KF Estimate + Future Correction
The smoother adds a single correction to the filter’s answer: α̂t = at|t + Pt|t · rt , where rt is a
backward running total of precision-weighted future surprises vs /Fs , each discounted by (1 − Ks ):
how surprised the filter was
z }| {
vs ys − a s
= .
Fs Ps + σε2
| {z }
total variation in ys
Nt accumulates variance reductions from future data. The filter extrapolates from the past
(estimates lag behind level shifts); the smoother interpolates using past and future (estimates
are centered and smooth).
Kalman smoother adjusts at|t at t =1898 using future data
1400
yt (observations)
at|t (KF)
1200 α̂t (KS)
Pt|t ·rt
1000
Nile flow
800
600
1880 1900 1920 1940 1960
Year
Weight functions: the KF implicitly assigns weights wt,j to past observations. These decay
1
exponentially into the past (recent data matters most). The KS assigns bell-shaped weights centered
at t: observations both before and after t contribute, with influence fading symmetrically in both
directions.
1.3.3 Maximum Likelihood Estimation
MLE estimates the hyperparameters ψ = (σε2 , ση2 ), not the states αt . The states are estimated by
the KF/KS given ψ; MLE finds the ψ that makes the observed data most likely. The prediction error
, TSM 2025–2026 Course Summary 4
decomposition gives the log-likelihood directly from KF output:
n
vt2
n 1X
ℓ(ψ) = − log(2π) − log Ft + . (12)
2 2 Ft
t=1
Each term penalizes two things: log Ft penalizes uncertainty, vt2 /Ft penalizes surprise. Good parame-
ters balance both.
With diffuse initialization (P1 = κ → ∞), the first d terms become parameter-independent: F1 ≈
P1 → ∞, so log F1 → log κ (constant) and v12 /F1 → 0. The diffuse log-likelihood drops these terms:
n
vt2
n 1 X
ℓd (ψ) = − log(2π) − log Ft + . (13)
2 2 Ft
t=d+1
The only difference is the lower summation index: t = 1 vs t = d + 1 (where d = number of diffuse
states, typically d = 1 for the LLM). Parameter estimates are unaffected because the dropped terms
carry no information about ψ.
Key Intuition: The KF Already Computes the Likelihood
Q
By the chain rule, p(Yn ) = t p(yt |Yt−1 ). The KF computes each predictive density (yt |Yt−1 ) ∼
N (at , Ft ) as a byproduct. The KF is just the “inner loop” that evaluates the objective function:
2 (0) 2 (0) ′
1. Start with an initial guess ψ̂ (0) = (σε , ση ).
2. For each iteration k:
(a) Run the KF with current ψ̂ (k) to obtain vt , Ft for t = 1, . . . , n.
(b) Evaluate ℓd (ψ̂ (k) ).
(c) A numerical optimizer (e.g. BFGS) proposes ψ̂ (k+1) .
3. Repeat until convergence. The result is ψ̂ = arg maxψ ℓd (Yn ).
1.4 Practical Extensions
Normal KF Missing yj Forecasting (h steps)
Idea Update state with yt No data; skip update Treat future as missing
Kalman Kt = Pt /Ft Kj = 0 K = 0 at each step
gain
State at+1 = at + Kt vt aj+1 = aj an+h = an+1
Variance Pt+1 = Kt σε2 + ση2 Pj+1 = Pj + ση2 Pn+h = Pn+1 + (h−1)ση2
Effect Uncertainty reduced by data Uncertainty grows (+ση2 ) Uncertainty grows linearly
1.5 Mathematical Foundation: Lemmas I & II
X
µX
ΣXX ΣXY
Lemma 1.2 (Lemma I). If Y ∼N µY , , then:
ΣY X ΣY Y
E[X | Y = y] = µX + ΣXY Σ−1
Y Y (y − µY ), (14)
Var(X | Y = y) = ΣXX − ΣXY Σ−1
Y Y ΣY X . (15)
