Regression Analysis III: Stationarity, Autocorrelation, and ARMA–ARIMA Models

Ordinary regression relates an outcome to predictors. Time-series modeling also asks how the past predicts the future and whether that relationship remains stable through time. A trend, persistent autocorrelation, and changing volatility can each require a different response.

Learning goals

Distinguish strict stationarity, covariance stationarity, and white noise.
Interpret autocorrelations without equating every negative lag with a profitable reversal.
Explain Wold innovations and their connection to lag-polynomial models.
Derive AR(1) moments and forecasts, and recognize MA and ARMA behavior.
Choose differencing and model orders using diagnostics and comparable information criteria.
Evaluate a financial time-series model chronologically.

1. Stationarity: what remains stable through time?

A discrete-time series {Xₜ} is a stochastic process indexed by observation dates. Its full specification includes joint distributions across time, not just a histogram at one date. Time-series theory often uses all integer dates to describe an indefinitely long history; an observed sample contains only a finite portion.

Strict stationarity

For every finite set of dates t₁,…,tₘ and allowable shift h, strict stationarity means:

(Xₜ₁,…,Xₜₘ) has the same joint distribution as (Xₜ₁₊ₕ,…,Xₜₘ₊ₕ).

A joint density is not required to exist. The definition applies equally to discrete variables or other distributions. Equality of one-date marginal distributions alone is insufficient because dependence across dates must also be invariant.

Covariance or weak stationarity

With finite second moments, weak stationarity requires:

E[Xₜ]=μ, and Cov(Xₜ,Xₜ₊ₕ)=γ(h) independently of t.

Variance is the special case γ(0)=σ_X², constant through time. Strict stationarity plus finite second moments implies weak stationarity. The converse is generally false; for a jointly Gaussian process, the mean and covariance determine all finite-dimensional distributions, so weak stationarity does imply strict stationarity.

Stationarity is a model assumption. One finite sample cannot prove it, and a visually stable mean does not establish stable tails or dependence. Estimating population moments reliably from one path also needs conditions such as appropriate ergodicity; stationarity alone is not enough.

2. Prices, log returns, and trends

For a positive price Yₜ, define the simple price return Rₜ=Yₜ/Yₜ₋₁−1 and log return:

Xₜ=log(Yₜ/Yₜ₋₁)=log(1+Rₜ).

Log returns add across dates. For small simple returns, log(1+R)≈R, but the quantities differ materially for large moves. If dividends or distributions matter, use a consistently defined total-return wealth series rather than treating an unadjusted price ratio as total return. Logs are undefined at zero or for negative values.

A model where differencing works

Let log Yₜ=log Yₜ₋₁+a+ηₜ, with i.i.d. zero-mean innovations of variance σ². Then log returns are a+ηₜ, a stationary series. In contrast, the log-price variance grows with elapsed time from a fixed initial value.

This is a model implication, not a guarantee about all asset returns. Structural breaks, changing unconditional volatility, and evolving distributions can remain after taking returns. Volatility clustering can also arise in a stationary model: a changing conditional variance is compatible with a constant unconditional variance under suitable parameters.

Deterministic versus stochastic trends

Yₜ=a+bt+uₜ, with stationary uₜ, is trend-stationary after subtracting its deterministic trend. A random walk has a stochastic trend and generally needs differencing to obtain stationary increments. These are different mechanisms, and their long-horizon forecasts differ. Near-unit-root behavior and finite samples can make them hard to distinguish.

Unit-root tests such as ADF and stationarity-null tests such as KPSS provide evidence under their own assumptions and deterministic-term specifications. Failing to reject a null is not proof that it is true; breaks and short samples can affect the tests. See Forecasting: Principles and Practice, ARIMA chapter for stationarity and differencing context.

3. Autocovariance, ACF, and PACF

For a covariance-stationary series with positive variance, the autocorrelation function is:

ρ(h)=γ(h)/γ(0).

One common sample estimate at positive lag k is:

ρ̂(k)=[Σₜ₌ₖ₊₁ⁿ(Xₜ−X̄)(Xₜ₋ₖ−X̄)]/[Σₜ₌₁ⁿ(Xₜ−X̄)²].

Other finite-sample normalizations exist, so software estimates can differ slightly. The partial autocorrelation (PACF) at lag k measures the remaining linear association between Xₜ and Xₜ₋ₖ after accounting for intermediate lags. It helps distinguish direct lag dependence from correlation transmitted through intervening observations.

Approximate significance bands

For i.i.d. white noise under suitable moment conditions and fixed nonzero lags, sample autocorrelations have asymptotic variance approximately 1/n. The familiar pointwise reference bands are approximately ±1.96/√n. A lag-dependent n−k denominator can be a rough effective-pair heuristic; it is not an exact universal normal sampling law.

For n=400, the approximate bands are ±0.098. A sample autocorrelation of 0.15 is outside this single-lag reference. Looking at many lags increases the chance of seeing an excursion even for white noise. These are not simultaneous 95% bands, and their simple form need not apply to autocorrelated or conditionally heteroscedastic data or fitted residuals.

What the sign does—and does not—mean

Positive return autocorrelation indicates linear persistence at that lag; negative return autocorrelation indicates a tendency toward reversal at that lag. Neither proves causality, stability, or trading profitability. Bid–ask bounce and nonsynchronous prices can generate apparent patterns.

Mean reversion of a level is a distinct statement. A stationary AR(1) level with a positive coefficient below one has positive level autocorrelation while reverting toward its long-run mean. Negative return autocorrelation is therefore not a necessary definition of level mean reversion. Specify the modeled variable before giving an economic label.

4. White noise and residual checks

A weak white-noise process ηₜ has zero mean, finite constant variance σ_η², and zero autocovariance at every nonzero lag. White noise need not be independent or Gaussian. I.i.d. white noise is a stronger assumption.

A successful conditional-mean model should leave no material predictable mean pattern relative to its information set. For a classical ARMA model, residuals should resemble its assumed innovation process. But uncorrelated residuals alone do not establish constant conditional volatility, independence, or correct tail behavior. Residuals and squared residuals can require separate checks; an ARMA mean model may need a volatility model such as GARCH for interval or risk applications.

The Box–Pierce statistic uses Q_BP=nΣₖ₌₁ᴷρ̂_e(k)². Ljung–Box adjusts finite-lag weighting:

Q_LB=n(n+2)Σₖ₌₁ᴷρ̂_e(k)²/(n−k).

Under an appropriate null, these have approximate chi-square reference laws. For residuals from a fitted nonseasonal ARMA(p,q), a common adjustment uses K−p−q degrees of freedom, with K>p+q; other estimated models need their own treatment. The official R Box.test documentation describes this ARMA adjustment.

Rejecting a portmanteau null suggests remaining linear dependence at the tested lags. Not rejecting does not prove all dependence is absent. Lag selection, estimated parameters, heavy tails, heteroscedasticity, and searching among many models affect interpretation.

5. Wold representation: innovations plus a predictable component

For a zero-mean, finite-variance covariance-stationary process with a full past, Wold’s decomposition separates:

Xₜ=Vₜ+Sₜ, with Sₜ=Σⱼ₌₀∞ψⱼηₜ₋ⱼ, ψ₀=1, and Σⱼψⱼ²<∞.

The sum converges in mean square. V is linearly deterministic: it can be predicted without linear prediction error from the arbitrarily remote past. S is purely nondeterministic, driven by white-noise innovations η. The two components are orthogonal across time. With the innovation normalization and nonzero innovation variance, the representation is unique in the relevant second-order sense.

“Deterministic” here does not mean any chosen trend line can be inserted into a stationary process. A random-amplitude harmonic component can be linearly predictable from its full history while still being a random stationary process. An arbitrary nonconstant deterministic trend would violate the constant-mean requirement.

An innovation is the new component left after the best linear prediction from the past. It is uncorrelated with that past; without additional assumptions it need not be independent of it or Gaussian. The best linear forecast and the full conditional expectation need not coincide outside suitable models.

Wold motivates moving-average representations but does not promise that every stationary process has an exact finite-order ARMA representation. Finite ARMA models are useful structured approximations or additional assumptions. Columbia’s time-series lecture notes discuss the representation and prediction framework.

6. Lag operators and invertibility

The lag operator L shifts back one observation: LXₜ=Xₜ₋₁, LᵏXₜ=Xₜ₋ₖ. Define Δ=1−L, so ΔXₜ=Xₜ−Xₜ₋₁.

Using ψ(L)=1+ψ₁L+ψ₂L²+…, the nondeterministic Wold component is Sₜ=ψ(L)ηₜ. Under additional stability conditions allowing a convergent causal inverse, ψ(L)⁻¹Sₜ=ηₜ gives an infinite autoregressive form. Do not automatically apply this to X without accounting for its deterministic component V.

For a finite MA polynomial θ(z), the usual invertibility condition is that its roots lie outside the unit circle. Wold’s square-summable representation alone should not be mistaken for an unconditional guarantee of an absolutely summable inverse filter. Invertibility recovers shocks from present and past observations; it is not the same condition as stationarity.

7. ARMA models: finite lag descriptions

A nonseasonal ARMA(p,q) model is:

φ(L)(Xₜ−μ)=θ(L)ηₜ,

where φ(L)=1−φ₁L−…−φₚLᵖ and θ(L)=1+θ₁L+…+θ_qLᑫ. The MA plus-sign convention is stated explicitly because software conventions can differ.

For a minimal representation without common polynomial factors, roots of φ(z) outside the unit circle give the usual stable causal stationary solution. Roots of θ(z) outside the unit circle give the usual invertibility condition. Causality means the process can be represented using current and past shocks, not future ones.

The MA terms are past shocks, not rolling averages of past observed prices. When fitting, those shocks are typically unobserved and estimated through likelihood or innovations methods.

8. AR(1): moments, mean reversion, and forecasts

Consider Xₜ−μ=φ(Xₜ₋₁−μ)+ηₜ, with white-noise innovation variance σ_η² and the causal innovation interpretation. For abs(φ)<1:

Xₜ−μ=Σⱼ₌₀∞φʲηₜ₋ⱼ.

Therefore:

Var(Xₜ)=σ_η²/(1−φ²), and ρ(h)=φ^abs(h).

These are stationary-solution moments. Starting a finite simulation from an arbitrary deterministic value introduces a transient before it resembles that distribution. The often-quoted “if and only if abs(φ)<1” concerns the usual causal innovation model; permitting dependence on future shocks changes the mathematical class and is not a forecasting interpretation used here.

Positive, negative, and unit-root coefficients

0<φ<1: deviations persist with the same sign but their expected magnitude shrinks toward zero.
−1<φ<0: expected deviations alternate sign while decaying.
φ=1: the level follows a random walk, with an optional drift depending on specification, and has no constant finite unconditional variance for nondegenerate accumulated shocks.
abs(φ)>1: the usual forward recursion is explosive.

Writing the same model as Xₜ=c+φXₜ₋₁+ηₜ gives c=(1−φ)μ. The intercept c is not generally the unconditional mean μ.

Worked forecast of a spread level

Assume a stationary Gaussian AR(1) spread in basis points with μ=3, φ=0.8, σ_η=2, and current Xₜ=8. Then:

Var(X)=4/(1−0.64)=11.1111 bp², with unconditional standard deviation 3.3333 bp.

The h-step conditional mean is μ+φʰ(Xₜ−μ). Hence the one-step forecast is 7 bp, and the three-step forecast is 5.56 bp.

The h-step forecast-error variance, treating model parameters as known, is:

σ_η²Σⱼ₌₀ʰ⁻¹φ²ʲ=σ_η²(1−φ²ʰ)/(1−φ²).

At h=3 it is 8.1984 bp², giving standard deviation about 2.8633 bp. Gaussian independent innovations yield normal conditional prediction errors; white-noise second moments alone support the linear forecast variance, not automatically a Gaussian interval or full conditional-mean claim. Estimated-parameter uncertainty adds further risk.

For 0<φ<1, the expected-deviation half-life is log(0.5)/log(φ), approximately 3.106 periods for φ=0.8. This describes the fitted model, not a guarantee that a real spread remains stable or tradable.

9. MA(q): finite memory in autocovariance

For Xₜ−μ=ηₜ+θ₁ηₜ₋₁+…+θ_qηₜ₋q, let θ₀=1. Then:

γ(h)=σ_η²Σⱼ₌₀ᑫ⁻ʰθⱼθⱼ₊ₕ for 0≤h≤q, and γ(h)=0 for h>q.

A finite MA of white noise is covariance stationary for finite coefficients. Invertibility imposes a separate restriction.

For Xₜ−μ=ηₜ+0.5ηₜ₋₁, variance is 1.25σ_η², lag-one correlation is 0.5/1.25=0.4, and correlations at higher lags are zero. Since the root of 1+0.5z is −2, this MA(1) is invertible.

ACF–PACF identification is a guide, not an exact sample rule

Population model	ACF	PACF
Causal AR(p)	Generally tails off	Cuts off beyond p
Invertible MA(q)	Cuts off beyond q	Generally tails off
ARMA(p,q)	Generally tails off	Generally tails off

Finite-sample estimates do not cut off perfectly. Seasonal structure, near-canceling AR/MA factors, and parameter uncertainty can obscure these signatures.

10. ARIMA and choosing differences

An ARIMA(p,d,q) model applies an ARMA structure to Wₜ=ΔᵈYₜ=(1−L)ᵈYₜ. Here Y is the modeled level, which could be a log-price, while W is its stationary modeled difference. d counts differences, not AR lags.

For a random walk Yₜ=Yₜ₋₁+c+ηₜ, d=1 yields ΔYₜ=c+ηₜ. That is an ARIMA(0,1,0) with drift c.

Differencing a polynomial trend

Level term	First difference	Second difference
a+bt	b	0
at²	a(2t−1)	2a

One difference turns a linear trend into a constant; two differences turn a quadratic trend into a constant. An additional difference annihilates that constant. Thus “d=1 removes a linear trend” means removing its time-varying part, not necessarily leaving a zero-mean series. For deterministic trends, detrending can be preferable to mechanically imposing integration.

Overdifferencing adds artificial dependence

If Yₜ=ηₜ is already white noise, then ΔYₜ=ηₜ−ηₜ₋₁ is an MA(1) with θ=−1. Its lag-one correlation is −0.5 and its MA root lies on the unit circle. Differencing has created an invertibility-boundary representation rather than discovering a meaningful reversal signal. Use the smallest defensible differencing order, not repeated differencing until an ACF plot looks convenient.

Seasonal differencing 1−Lˢ is another operation and should reflect seasonal integration, not merely the existence of a calendar. Multiple nonstationary financial series may be cointegrated; modeling everything in differences can discard a long-run relation. Those cases call for a richer specification.

11. Model selection: likelihood, parameter counts, and comparability

For maximized log-likelihood ℓ̂ and k estimated parameters, use:

AIC=−2ℓ̂+2k,

BIC=−2ℓ̂+k log n.

Count the fitted AR and MA coefficients, included mean/drift or regressors, and innovation variance as appropriate to the likelihood convention. p+q alone is not always the complete parameter count. Lower values are preferred among comparable candidates, not proofs that the selected model is true.

A commonly used small-sample correction is:

AICc=AIC+2k(k+1)/(n−k−1),

with n>k+1 and parameter-count conventions matched to the ARIMA implementation. The correction is model-context dependent; inspect the software’s treatment of effective sample size, initialization, and included parameters.

The lecture’s log(σ̂²)+2(p+q)/n is a normalized simplified Gaussian criterion: after dividing by n and dropping common likelihood constants, it can rank models equivalently under matching sample and nuisance-parameter conventions. It is not a universal substitute for the fitted likelihood.

A worked comparison

For a common n=100 sample and a consistent likelihood definition, suppose:

Candidate	−2ℓ̂	k	AIC	AICc	BIC
Smaller model	300	2	304	304.124	309.210
Larger model	294	4	302	302.421	312.421

AIC and AICc prefer the larger fit here, while BIC prefers the smaller one. The criteria balance fit and complexity differently. Neither substitutes for residual checks or out-of-sample forecast evaluation.

Use identical observations and compatible response transformations, likelihood methods, and initialization conventions when comparing candidates. Standard ARIMA information criteria generally guide p and q after choosing differencing; naive comparisons across different d can involve noncomparable likelihoods or samples. Avoid selecting d solely by the smallest printed AIC. See the official R ARIMA documentation for its likelihood, mean, and differencing conventions.

12. Connecting ARMA errors back to regression

A time-series regression can have external predictors and serially dependent errors:

Yₜ=β₀+βᵀzₜ+uₜ, with φ(L)uₜ=θ(L)ηₜ.

Predictors explain the conditional relationship, while ARMA structure describes remaining linear error dependence. Simply adding lags of Y is a different specification and changes coefficient interpretation. Lagged dependent variables also create conditions under which the fixed-design OLS formulas require care.

If regression variables are nonstationary, an apparently strong level regression can be spurious unless a long-run relationship is justified. HAC standard errors alone do not turn a spurious regression into a valid model. Forecasting with external predictors also requires those future predictor values to be known, separately forecast, or supplied as scenarios.

13. A finance-oriented workflow

Define the price, total return, yield level, spread, or change being modeled, including units and observation frequency.
Plot levels and transformations; inspect breaks, missing dates, seasonality, and rolling moments.
Choose a defensible stationarity treatment and a simple forecast baseline, such as a random walk for the level or historical mean for returns.
Inspect ACF/PACF and fit a limited, documented set of causal/invertible candidates with consistent likelihood conventions.
Compare AIC/AICc/BIC on a common basis, then check residuals, squared residuals, forecast stability, and economic plausibility.
Evaluate rolling or expanding forecasts using only earlier data for preprocessing, fitting, and model selection. Match the evaluation horizon to the intended decision.
Assess prediction intervals as well as point errors. If forecasts become trades, include bid–ask spreads, turnover, financing, and implementation timing.
Reassess after regime changes rather than treating a once-selected lag order as permanent.

An ARMA fit can describe linear dependence without producing an economically valuable trading signal. A nearly unpredictable mean and a highly predictable volatility process can coexist in financial returns.

14. Practice questions

Must a strictly stationary process have a joint density or finite variance?
Does taking log returns guarantee stationary financial data?
For n=400, what are the common approximate pointwise white-noise ACF bands? Are they simultaneous bands across every plotted lag?
For a causal AR(1) with φ=0.6 and innovation variance 4, find stationary variance and lag-two autocorrelation.
If an AR(1) is written Xₜ=2+0.8Xₜ₋₁+ηₜ, what is its stationary mean?
Does an MA(2) necessarily have independent values three dates apart? What does its covariance do at that lag?
What does differencing an already-white-noise series do to its lag-one autocorrelation?
A fitted model has −2ℓ̂=200, k=3, and n=80. Compute AIC and the stated AICc correction.

Answers

Neither is required. Strict stationarity concerns joint distributions; weak stationarity additionally requires finite second moments.
No. They are stationary under some models, but breaks and evolving distributions can remain.
Approximately ±0.098. No: these are pointwise reference bands under the stated assumptions.
4/(1−0.36)=6.25, and ρ(2)=0.6²=0.36.
μ=2/(1−0.8)=10.
Its covariance is zero beyond lag two. Independence would follow from independent driving shocks for nonoverlapping terms, but weak white noise alone does not guarantee it.
It produces ρ(1)=−0.5, an overdifferencing artifact in this example.
AIC=206. The correction is 2×3×4/(80−3−1)=24/76≈0.316, so AICc≈206.316.