Results

class hdfe_stream.HDFEResult

The result of one model: coefficients, standard errors, fit statistics, and lazy access to the fixed effects and the row-level output.

Returned by feols_stream, fepois_stream, feglm_stream and StreamingHDFE.fit (or within an HDFEMulti). The fixed effects and residuals are Parquet files in the fit’s run directory, which live as long as this object (unless the fit used outputs="keep"): collect or sink what you need from fixef and resid before dropping it.

Attributes:

• fml (str) – The model’s formula.

• depvar (str) – The dependent variable.

• coefnames (list of str) – Coefficient names, as pyfixest names them.

• beta (numpy.ndarray) – The coefficients, in the order of coefnames.

• se (numpy.ndarray) – Their standard errors, under the current vcov.

• vcov (numpy.ndarray) – The variance-covariance matrix of the coefficients.

• vcov_type (str) – Which vcov that is: "iid", "hetero", "CRV1:var", …

• all_vcovs (dict) – Every vcov computed at fit time, for with_vcov.

• n_clusters (dict) – The number of clusters of each clustering computed.

• df_t (int) – Degrees of freedom of the t distribution for p-values.

• n_obs (int) – Observations used (the sum of the weights for frequency weights).

• fe_names (list of str) – The fixed-effect dimensions.

• n_levels (dict) – The number of levels of each dimension.

• k_fe (int) – Fixed-effect parameters, net of the redundant ones (see fe_dof in StreamingHDFE.fit).

• n_components (int) – Connected components of the first two dimensions.

• n_identifying (int) – Groups of the streamed dimension that carry identifying information (movers, in an AKM model).

• rss (float) – Residual sum of squares; nan for GLMs.

• r2 (float) – R-squared; nan for GLMs.

• adj_r2 (float) – Adjusted R-squared; nan for GLMs.

• r2_within (float) – Within R-squared, net of the fixed effects; nan for GLMs.

• adj_r2_within (float) – Adjusted within R-squared; nan for GLMs.

• rmse (float) – Root mean squared error; nan for GLMs.

• family (str or None) – "poisson", "logit" or "probit" for a GLM; None for OLS.

• deviance (float) – For a GLM, the deviance; nan for OLS.

• loglik (float) – For a GLM, the log-likelihood; nan for OLS.

• pseudo_r2 (float) – For a GLM, McFadden’s pseudo R-squared; nan for OLS.

• is_iv (bool) – Whether the model was estimated by 2SLS.

• first_stage (list of HDFEResult) – For IV, the first-stage regressions.

• f_stat_1st_stage (list of float) – For IV, the Wald F of the excluded instruments, under the same vcov.

• collin_vars (list of str) – Covariates dropped as collinear.

• weights (str or None) – The weights column, if any.

• weights_type (str) – "aweights" or "fweights".

• solver_info (dict) – The solver used, its iterations, convergence and timing.

• diagnostics (dict) – Everything else the fit recorded: the streamed dimension and why ("stream"), dropped observations ("singletons", "connected", "separation"), IRLS steps ("irls"), disk and time ("disk_peak_gb", "seconds_total"), and more.

• paths (dict) – The result files.

summary(logger=None, level=20)

Print the report, or write it to logger (one record, at level).

summary_text()

The summary() report as a string.

summary_dict(coefficients=False)

The model-level information of summary() as a dict of plain Python types (JSON-ready; nan is None). Fit statistics not defined for the model (e.g. r2 for a GLM) are absent. coefficients=True adds the tidy() table as a list of row dicts.

summary_json(path=None, indent=2, coefficients=False)

The model-level information of summary() as a JSON string (see summary_dict); with path, also written there as UTF-8. Reload with json.loads or json.load.

tidy()

Coefficient table. p-values use the t distribution with df_t degrees of freedom, or the normal for GLMs, as in pyfixest.

confint(alpha=0.05, keep=None, drop=None, exact_match=False, joint=False, reps=10000, seed=None)

Confidence intervals, one coefficient at a time or simultaneous.

Pointwise intervals use the t distribution with df_t degrees of freedom, or the normal for GLMs, as tidy does. With joint=True, the intervals cover all the selected coefficients at once with probability 1 - alpha: the critical value is the 1 - alpha quantile of the largest absolute t statistic, simulated from the normal with the coefficients’ correlation (the multiplier bootstrap of pyfixest’s confint(joint=True); Chernozhukov et al., Applied Causal Inference Powered by ML and AI, remark 4.4.1).

Parameters:

• alpha (float, default 0.05) – One minus the coverage.

• keep (str or list of str, optional) – Regular expressions selecting coefficients by name, as in pyfixest: those matching a keep pattern, in the order of the patterns (all, by default), less those matching a drop pattern. A joint interval covers exactly these.

• drop (str or list of str, optional) – Regular expressions selecting coefficients by name, as in pyfixest: those matching a keep pattern, in the order of the patterns (all, by default), less those matching a drop pattern. A joint interval covers exactly these.

• exact_match (bool, default False) – Match names exactly instead of as regular expressions.

• joint (bool, default False) – Simultaneous intervals.

• reps (int, default 10000) – Draws for the joint critical value.

• seed (int, optional) – Seed for those draws.

Returns:

polars.DataFrame – Coefficient and the lower and upper bounds, in columns named by their percentiles ("2.5%", "97.5%").

wald_test(R=None, q=None, distribution=None, keep=None, drop=None, exact_match=False)

Wald test of the linear hypothesis R beta = q, under the current vcov (see with_vcov).

By default the hypothesis is that every coefficient is zero; keep and drop narrow it to some of them (the coefficients of an event study’s leads, say), and R and q state any other.

The statistic is W = (R b - q)’ (R V R’)^+ (R b - q), and F = W / m for m restrictions. F is referred to the F distribution with m and df_t degrees of freedom, so that for one coefficient the p-value is the one tidy reports; W, to the chi-squared with m. Two differences from pyfixest’s wald_test: it reports chi-squared whenever R is not the identity or q is not zero, where this test reports what is asked for; and its F’s denominator degrees of freedom, without clustering, subtract every fixed-effect level, including the redundant ones that df_t does not.

Parameters:

• R (array-like, optional) – m x k, one row per restriction, one column per coefficient in the order of coefnames; a 1-d array is one restriction.

• q (float or array-like, optional) – The m right-hand sides; zero by default.

• distribution ({"F", "chi2"}, optional) – Default “F” for OLS and “chi2” for GLMs, whose tidy p-values are normal.

• keep – Without R: test that the coefficients they select are zero (or q), as in confint.

• drop – Without R: test that the coefficients they select are zero (or q), as in confint.

• exact_match – Without R: test that the coefficients they select are zero (or q), as in confint.

Returns:

dict – statistic (F or W), pvalue, df1 (m), df2 (df_t for F, None for chi-squared) and distribution.

wildboottest(param, reps=9999, weights_type='rademacher', impose_null=True, seed=None)

Wild cluster bootstrap test of param = 0.

Needs the fit made with bootcluster= (see StreamingHDFE.fit); the bootstrap clusters by that variable, works from the per-cluster sums the fit kept, and reads no data. A test costs about reps x G x k operations, for G clusters and k coefficients: well under a second with 50 clusters.

The bootstrap is the restricted wild cluster bootstrap (WCR, the “11” variant of pyfixest and fwildclusterboot), or the unrestricted one with impose_null=False. Its draws are those of pyfixest’s wildboottest with the same seed and weights, so the p-values agree with pyfixest’s up to rounding, where pyfixest can fit the model (it puts the fixed effects in as dummy variables). With Rademacher weights and 2^G < reps, every sign pattern is used once instead, as in pyfixest.

A draw that gives every cluster the same weight reproduces the sample (up to scale), and so the t statistic exactly. With few clusters and two-point weights such draws are common (with 12 clusters, about 2% of Mammen draws), and they count as at least as extreme as the t statistic: the conservative choice, and a deterministic one. pyfixest’s comparison counts or skips them depending on rounding.

Parameters:

• param (str) – The coefficient tested.

• reps (int, default 9999) – Bootstrap draws.

• weights_type ({"rademacher", "mammen", "webb", "norm"}, default "rademacher") – The distribution of the cluster weights.

• impose_null (bool, default True) – Impose the null hypothesis on the bootstrap data-generating process (WCR), or not (WCU).

• seed (int, optional) – Seed for the draws.

Returns:

dict – param; t value, the t statistic under CRV1 by the bootstrap clustering (the one tidy reports under that vcov); Pr(>|t|), the bootstrap p-value, the share of draws whose absolute t statistic is at least as large; reps, the draws used; weights_type, impose_null and cluster.

coef()

The coefficients, as {name: estimate}.

with_vcov(vcov)

A copy of this result with another of the vcovs computed at fit time: "iid", "hetero", or a clustering given as {"CRV1": var} or "CRV1:var" (likewise CRV3, when it was the fit’s vcov). Ask for clusterings at fit time with vcov= or cluster=.

fixef(name)

Lazy scan of one dimension’s estimated fixed effects: one row per level, with its effect (fe_<name>, and a column per slope with varying slopes), its number of observations and, for the first non-streamed dimension, its connected component. The file lives as long as this result object (outputs="auto"): collect or sink what you need before dropping the result.

resid()

Lazy scan of the row-level output: the outcome, xb, each row’s fixed effects and the residual (for GLMs also eta, fitted and resid_working), plus any keep= columns. Same lifetime caveat as fixef. The row_id column is each row’s position in the source, so the residuals join back onto sample (or onto the source with with_row_index("row_id")) on it. The file is in the estimator’s own order, not the source’s; sort by row_id to restore it.

sample(check=True)

The source, lazily, with three columns added saying what the fit did with each row.

row_id

The row’s position in the source (the first column); the same column resid carries.

in_sample

True for the rows the model was estimated on.

dropped_because

Null for those rows. Otherwise "missing" (a null in any column the fit uses, a NaN in a fixed effect or cluster, or a non-finite outcome, covariate or weight), "min_degree" (a level with too few distinct partners, min_degree), "disconnected" (outside the largest connected set, connected="largest"), "leave_one_out" (outside the leave-one-out connected set, connected="leave_one_out"), "singleton" (fixef_rm) or "separation" (GLMs).

Nothing is stored per row: the frame is the source plus an expression and the small tables of levels the fit dropped, so it costs a scan of the source when collected, and a join with those tables.

The source must be the same rows in the same order as at the fit. A DataFrame is, and so is a file or glob (Polars keeps the file list and metadata in the plan, and reading a rewritten file fails); a LazyFrame is only if its plan is, which a group_by, an unordered join or a random sample is not.

Parameters:

check (bool, default True) – Compare the source’s row count with the count at the fit. This catches a plan that returns a different number of rows each time it runs, not one that returns the same rows in a different order.

Returns:

polars.LazyFrame

predict(newdata=None, type='link', name='prediction')

Predictions on new data, or on the source, as a lazy frame.

Returns newdata, lazily and in its own order, with one column added: the linear predictor (each row’s covariates times the coefficients, plus its fixed effects and, for a GLM, its offset), or with type="response", the mean. Nothing is computed until the frame is collected or sunk, so newdata can be larger than memory. The fixed effects come from the fit’s files, which live as long as this result (see fixef): collect or sink the predictions before dropping it.

A row gets a null prediction if any of its fixed-effect levels has no estimated effect (a level the estimation sample does not contain: new, or dropped, e.g. as a singleton), as pyfixest gives NaN, or if a covariate is null.

The fixed effects are identified only within a connected set of the first two dimensions (workers and firms, say), so a prediction that combines levels from different components, such as a worker from one and a firm from another, depends on the normalization and means little. fixef(...) reports each level’s component for the first non-streamed dimension.

Parameters:

• newdata (DataFrame, LazyFrame, or Parquet path or glob, optional) – The rows to predict for, with every column the model uses. A categorical covariate’s indicators are those of the fit, so a level the fit did not see counts as the base level. Default: the fit’s own source, every row of it (the estimation sample’s rows then reproduce the fitted values in resid).

• type ({"link", "response"}, default "link") – For a GLM, the linear predictor, or the mean (exp for Poisson, the logistic or normal CDF for logit and probit). The same for OLS.

• name (str, default "prediction") – The name of the added column.

Returns:

polars.LazyFrame

to_pyfixest()

A pyfixest Feols (Feiv for IV) view of this result, for pf.etable, pf.summary, pf.coefplot and pf.iplot. Only the stored results are available; methods that need the data (recomputing the vcov, the wild bootstrap) are not, and predict is this result’s own predict.

leave_out_kss(n_draws=250, seed=0, block=None, psi=None, leave_out='match', stayers=None, se=False, se_draws=None, se_trace=True, diagnose=None, diagnose_draws=64, weak_interval=True, confidence=0.95, centering='reference', se_variance='person_year')

Kline-Saggio-Solvsten leave-out variance components for this fit.

This is the secondary way in, for when you want the regression in its own right as well. The top-level hdfe_stream.leave_out_kss is the usual one, and does the whole sequence.

The fit must be on a panel that is already leave-one-out connected (fit with connected="leave_one_out", or on data pruned with connected_components(..., leave_one_out=True)). If you have not pruned, use the top-level function: pruning changes the estimation sample, so a fit made before pruning is a fit of a different model.

Parameters:

• n_draws – As in hdfe_stream.leave_out_kss.

• seed – As in hdfe_stream.leave_out_kss.

• block – As in hdfe_stream.leave_out_kss.

• psi – As in hdfe_stream.leave_out_kss.

• se – As in hdfe_stream.leave_out_kss.

• se_draws – As in hdfe_stream.leave_out_kss.

• se_trace – As in hdfe_stream.leave_out_kss.

• diagnose – As in hdfe_stream.leave_out_kss.

• diagnose_draws – As in hdfe_stream.leave_out_kss.

• weak_interval – As in hdfe_stream.leave_out_kss.

• confidence – As in hdfe_stream.leave_out_kss.

• leave_out ({"match", "observation"}, default "match") – The unit left out. "match" is a whole worker-firm spell, as LeaveOutTwoWay, VarianceComponentsHDFE.jl and xhdfe do by default. It is robust to errors correlated within a spell. Everything except the worker and firm effects is partialled out first and the data collapsed to one row per match, so this fit is the regression, not the leave-out one. Standard errors for var(psi) and the covariance, following LeaveOutTwoWay’s leave_out_COMPLETE, whose match-level option is marked beta there. "observation" is a single person-year, and needs this fit made with keep_intermediates=True.

• stayers (str, optional) – At observation level only ("own" by default); at match level the reference’s within-match rule is used.

• centering ({"reference", "weighted"}, default "reference") – At match level only. "reference" is LeaveOutTwoWay’s: sqrt(w) ybar less its mean over matches. "weighted" centers ybar at its weighted mean first. They coincide when spells are of equal length or the outcome is centered near zero; otherwise both are unbiased but the reference’s carries the outcome’s level into sigma2, and with log earnings its standard deviation was 1.7 to 3 times the weighted one’s (docs/kss/differences.md, 3.4).

• se_variance ({"person_year", "match"}, default "person_year") – At match level with se=True: how each match’s error variance is estimated. "person_year" is the reference implementation’s (leave_out_COMPLETE, matches, beta there), from the person-year residuals, pooling the variation within the spell; it assumes errors independent within a spell and needs an unweighted fit. "match" goes beyond the reference: the collapsed leave-match-out estimate alone, which stays right when errors are correlated within a spell, at the cost of more noise (docs/kss/differences.md, 5.7).

Returns:

LeaveOutComponents

cleanup()

Delete this model’s result files (and its first-stage files for IV); the run directory goes once no model files remain.

property files_dir

Directory holding this model’s result files.

__init__(fml, depvar, coefnames, fe_names, beta, vcov, vcov_type, df_t, n_obs, n_levels, n_identifying, n_components, k_fe, rss, r2_within, solver_info, paths, collin_vars=<factory>, diagnostics=<factory>, all_vcovs=<factory>, r2=nan, adj_r2=nan, adj_r2_within=nan, rmse=nan, is_iv=False, first_stage=<factory>, f_stat_1st_stage=<factory>, n_clusters=<factory>, weights=None, weights_type='aweights', family=None, deviance=nan, loglik=nan, pseudo_r2=nan, _run=None, _estimator=None, _sample=None, _boot=None)

classmethod __new__(*args, **kwargs)

class hdfe_stream.HDFEMulti

Results of a formula that expands to several models.

Iterate over it, or take one model with fetch_model, to get each HDFEResult; len() counts them. The models are in the formula’s order, and keyed by formula in all_fitted_models.

__init__(results)

fetch_model(i)

The i-th model, in the formula’s order.

to_pyfixest()

A pyfixest view of each model (see HDFEResult.to_pyfixest).

sample(check=True)

{formula: HDFEResult.sample()} for each model. Models fitted together on the same fixed effects share one sample, so their frames are the same; only a stepwise change in the fixed effects (or, for GLMs, in the outcome) changes it. Every frame uses the same row_id.

cleanup()

Delete the result files of all models.

tidy()

Every model’s coefficient table, with a column fml first.

summary_text()

Every model’s report, as one string.

summary_dict(coefficients=False)

{formula: HDFEResult.summary_dict(...)} for each model.

summary_json(path=None, indent=2, coefficients=False)

JSON object keyed by formula, each value as HDFEResult.summary_json; with path, also written there.

summary(logger=None, level=20, per_model=False)

Print all reports, or write them to logger: one record in total, or one per model with per_model=True.

classmethod __new__(*args, **kwargs)