Absorbed period effects

Download the script: absorbed.py

The problem. When shocks are common to a region in a year, year dummies are not enough: the errors share a region-year component, and the instruments are correlated with it. A dummy for every region-year cell would fix that, but with 400 regions it means 2,400 more regressors (the cells with observations), and as many more instruments.

The fix. In difference GMM, | region^year absorbs them. The region must be constant within each panel. The estimates are the same as with the dummies (see how it works).

inst = "gmm(y, 2:4) + gmm(x, 1:3) + iv(w)"
naive = dpd_stream(f"y ~ l(y, 1) + x + w | year | {inst}", df, panel=("id", "year"))
cells = dpd_stream(f"y ~ l(y, 1) + x + w | region^year | {inst}", df, panel=("id", "year"))

year effects only:   [0.3347 0.9283 0.4494]
region-year cells:   [0.4938 0.9966 0.4988] {'term': 'region^year', 'effects': 2400, 'method': 'cells'}

The true values are 0.5, 1 and 0.5.

Everything is the dummies’. The estimates and their variances (Windmeijer-corrected or not), σ², and the Sargan, Hansen, difference-in-Hansen and Arellano–Bond tests are those of the model with the dummies among the regressors and instruments, with any clustering. Windmeijer’s correction and the Arellano–Bond tests take one or two more passes over the data for it (see how it works).

Options. The term is a single g^year, with g constant within panels, in difference GMM. It works with:

  • first differences or orthogonal deviations;

  • any clustering, weights, one-step, two-step or iterated GMM;

  • the identification tests, which partial the effects out.

It does not work with PCA, or with Ahn and Schmidt’s moments, which are nonlinear in the effects.

| year is absorbed by dummies. Time effects alone are few, so | year is not partialled out: it is estimated as a dummy per period, each its own instrument, and the dummies are not reported (fit.absorbed says 'method': 'dummies', against 'cells' for region^year). Everything is then exact, in both estimators, with every option, including PCA and Ahn and Schmidt’s moments.

The script
"""Absorbing high-dimensional period effects in difference GMM.

    python examples/absorbed.py

With region-by-year shocks, year dummies are not enough; a dummy for every
region-year cell is thousands of regressors and instruments. Absorbing
`region^year` gives the same estimates as those dummies would.
"""

import numpy as np
import polars as pl

from dpd_stream import dpd_stream
from dpd_stream.simulate import simulate_dynamic

rng = np.random.default_rng(5)
df = simulate_dynamic(50_000, 8, rho=(0.5,), time_effects=False, seed=5)
n_regions = 400
region = rng.integers(0, n_regions, df["id"].max() + 1)
shock = rng.normal(0, 1, (n_regions, 8 + 1))
df = df.with_columns(pl.Series("region", region[df["id"].to_numpy()]))
yr = (df["year"] - df["year"].min()).to_numpy()
df = df.with_columns(pl.col("y") + shock[df["region"].to_numpy(), yr])  # region-year shocks

P = ("id", "year")
inst = "gmm(y, 2:4) + gmm(x, 1:3) + iv(w)"
naive = dpd_stream(f"y ~ l(y, 1) + x + w | year | {inst}", df, panel=P, verbose=False)
cells = dpd_stream(f"y ~ l(y, 1) + x + w | region^year | {inst}", df, panel=P, verbose=False)
print("year effects only:  ", naive.beta.round(4))
print("region-year cells:  ", cells.beta.round(4), cells.absorbed)