Leave-out variance components

The leave-out variance components section explains how to use these.

hdfe_stream.leave_out_kss(fml, data, workdir=None, *, psi=None, n_draws=250, seed=0, block=None, prune=True, leave_out='match', stayers=None, se=False, se_draws=None, se_trace=True, diagnose=None, diagnose_draws=64, weak_interval=True, confidence=0.95, centering='reference', se_variance='person_year', verbose=True, logger=None, **options)[source]

Kline-Saggio-Solvsten leave-out variance components, end to end.

The plug-in AKM decomposition is biased – upward for the variances, toward zero for the covariance – because the fixed effects are estimated with error. This removes that bias, following Kline, Saggio and Solvsten (2020), without assuming the errors are identically distributed.

Three things happen, in this order, because the order matters:

  1. The panel is pruned to its largest leave-one-out connected set. Leave-out estimation needs every effect to stay estimable when any one match is dropped, and that fails at workers who are articulation points of the worker-firm network; they are removed, repeatedly, until none is left, and workers observed once are dropped too. This changes the estimation sample, often substantially – in KSS’s own application it removes about half the firms. It is the fit’s connected="leave_one_out", so it runs on the rows the model can use, and the fit’s sample() says which rows it dropped ("leave_one_out"); connected_components(..., leave_one_out=True) finds the same set on its own, ahead of time.

  2. The model is fitted on what survives.

  3. Leverages are approximated by random projection, and from them the bias.

Parameters:

  • fml (str) – A formula with at least two fixed effects, e.g. "log_earn ~ age_squared | worker_id + firm_id". One outcome only.

  • data (str, Path, polars.LazyFrame or polars.DataFrame) – As in feols_stream.

  • workdir (str or Path, optional) – As in feols_stream.

  • psi (str, optional) – The non-streamed dimension whose variance is wanted (the firms); the streamed dimension plays the part of alpha (the workers). Both default to the first two fixed effects in the formula, and the streamed dimension is pinned to the first so that the sample that was pruned and the decomposition that follows are about the same pair.

  • n_draws (int, default 250) – Random-projection draws for the leverages, and Hutchinson draws for the bias term. A budget, not a function of the sample size: KSS use 250-500 on panels with millions of effects.

  • seed (int, default 0) – The seed of every random draw. Results are reproducible from it.

  • block (int, optional) – Draws per block (each block is one solve with that many right-hand sides and two passes over the rows).

  • prune (bool, default True) – False skips step 1 and fails if the panel is not already leave-one-out connected, for callers who have pruned upstream and want to be told if they are wrong.

  • leave_out ({"match", "observation"}, default "match") – "match" leaves out a whole worker-firm match, as LeaveOutTwoWay, VarianceComponentsHDFE.jl and xhdfe do: the fit’s partialled-out outcome is collapsed to one row per match and the correction runs on that weighted regression. Stayers’ variances follow LeaveOutTwoWay’s within-match rule. With se=True, standard errors are reported for var(psi) and the covariance, following leave_out_COMPLETE’s match-level option (beta there). "observation" leaves out one row at a time, and all three components get standard errors.

  • stayers (str, optional) – At observation level only: how a stayer’s variance is estimated ("own" by default, its own leave-out estimate).

  • se (bool, default False) – Compute standard errors and intervals for the components.

  • se_draws (int, optional) – Draws for the trace term of the standard errors. Default: n_draws.

  • se_trace (bool, default True) – Include the trace term. Without it the standard errors are conservative, and cost about three fewer solves per draw.

  • diagnose (bool, optional) – Run KSS’s weak-identification diagnostic. Default: when se=True.

  • diagnose_draws (int, default 64) – The diagnostic’s draw budget.

  • weak_interval (bool, default True) – For every component the diagnostic flags (q >= 1), also compute KSS’s q = 1 interval, which stays valid under weak identification.

  • confidence (float, default 0.95) – The level of both intervals.

  • centering ({"reference", "weighted"}, default "reference") – At match level: how sigma2 centers the outcome; see HDFEResult.leave_out_kss.

  • se_variance ({"person_year", "match"}, default "person_year") – At match level with se=True: how each match’s error variance is estimated; see HDFEResult.leave_out_kss.

  • verbose (bool, default True) – Emit progress messages.

  • logger (logging.Logger, optional) – As in StreamingHDFE.

  • **options – Passed to the fit, so weights= and the memory options remain available. fixef_rm defaults to "none" here, because the pruning has settled the sample.

Returns:

LeaveOutComponents – Its pruning holds the counts of step 1 (the fit’s diagnostics["connected"]), or None without it, and fit the regression.

Examples

>>> lo = leave_out_kss("log_earn ~ age_squared | worker_id + firm_id",
...                    "data/*.parquet", workdir="scratch")
>>> print(lo.summary())
class hdfe_stream.leaveout.LeaveOutComponents[source]

KSS leave-out variance components, with the plug-in ones for comparison, as returned by leave_out_kss.

The three components are keyed "var(psi)", "var(alpha)" and "cov(psi, alpha)".

Attributes:

  • plug_in (dict) – The plug-in components, computed from the estimated effects.

  • bias (dict) – The estimated bias of each.

  • leave_out (dict) – The bias-corrected components: plug-in less bias.

  • n_obs (int) – Observations (person-years) in the decomposition.

  • n_movers (int) – Movers’ observations, or movers’ matches when leaving out a match.

  • leverages (JLALeverages) – The random-projection leverages.

  • trace (TraceEstimate) – The bias term’s trace estimate.

  • se (ComponentSE or None) – Standard errors, with se=True.

  • weak_id (WeakIdDiagnostics or None) – The weak-identification diagnostic, when it ran.

  • diagnostics (dict) – Settings and summary statistics (maximum leverage, mean sigma2, …).

  • pruning (dict or None) – Counts of the pruning to the leave-one-out connected set.

  • fit (HDFEResult) – The regression the components are from.

  • match_fit (HDFEResult or None) – When leaving out a match, the collapsed match-level regression.

tidy()[source]

The three components as a Polars DataFrame.

summary()[source]

A report of the components, with the standard errors, intervals and diagnostic when they were computed.

__init__(plug_in, bias, leave_out, n_obs, n_movers, leverages, trace, se=None, weak_id=None, diagnostics=<factory>, pruning=None, fit=None, match_fit=None)
classmethod __new__(*args, **kwargs)
class hdfe_stream.ComponentSE[source]

Standard errors for the three components, and the parts they came from: LeaveOutComponents.se.

Attributes:

  • se (dict) – The standard error of each component; nan where none is reported.

  • variance (dict) – The estimated variance of each component’s estimator.

  • first_term (dict) – The first term of that variance, 4 sum_i sigma2_i (C mu)_i^2.

  • trace_term (dict) – The second, 2 tr(C Omega C Omega).

  • se_conservative (dict) – The standard error from the first term alone, which is conservative.

  • theta (dict) – The estimand recomputed as the quadratic form y'Cy, an independent check on the point estimate.

  • n_draws (int) – Draws for the trace term.

  • weak (dict or None) – For each weakly identified component, the q = 1 interval ("interval") and the pieces it was computed from.

  • diagnostics (dict) – Settings and timings.

summary()[source]

A table of the estimates and their standard errors.

__init__(se, variance, first_term, trace_term, se_conservative, theta, n_draws, diagnostics=<factory>, weak=None)
classmethod __new__(*args, **kwargs)
class hdfe_stream.WeakIdDiagnostics[source]

KSS’s weak-identification diagnostic for the three components: LeaveOutComponents.weak_id.

Each attribute but threshold and diagnostics is a dict keyed by component.

Attributes:

  • eigenvalues (dict) – The leading eigenvalues of S^- Q, largest in magnitude first (signed: the covariance’s can be negative).

  • sum_sq (dict) – The sum of all squared eigenvalues, tr((S^- Q)^2), estimated.

  • sum_sq_se (dict) – The Monte Carlo standard error of that estimate.

  • ratios (dict) – lambda_l^2 / sum_sq for the leading eigenvalues.

  • q (dict) – KSS’s threshold rule; 0 means the normal interval is justified.

  • q_capped (dict) – True when every reported ratio clears the threshold, so the true q may be larger.

  • q_borderline (dict) – True when a ratio lies within two Monte Carlo standard errors of the threshold, so q could go either way at this draw budget; raise diagnose_draws to resolve it.

  • f_stat (dict) – b1^2 / V[b1] for the leading direction: how well its nuisance parameter is identified. Large values mean the delta-method (normal) interval is less at risk even when q >= 1 (KSS Remark 10).

  • bounds (dict) – The Lanczos error bound on each reported eigenvalue.

  • converged (dict) – Whether the leading two eigenvalues met the tolerance.

  • threshold (float) – KSS’s threshold on lambda^2 / sum lambda^2, 0.1.

  • diagnostics (dict) – Settings and timings.

property weakly_identified

Components for which the normal interval is not justified (q >= 1).

summary()[source]

A table of the ratios, q and F for each component.

__init__(eigenvalues, sum_sq, sum_sq_se, ratios, q, q_capped, q_borderline, f_stat, bounds, converged, threshold=0.1, diagnostics=<factory>, lead_vectors=None)
classmethod __new__(*args, **kwargs)