Changelog¶
0.4.0¶
New
mlogit_stream(own_covariates=...): covariates that enter only their own level’s equation. For example, each group’s baseline share can predict that group alone, leaving out the other groups’ shares. This drops the off-diagonal coefficients, which are often poorly determined where the groups are segregated, and fits fewer parameters.logit_streamandprobit_streamhave their own engine for no fixed effect or one, about 2 times faster than hdfe-stream’s. They fit the two-level ordered logit or probit with this package’s Newton iterations and return aBinaryLogitResultorBinaryProbitResult(new classes;predict(type="class")is new). Estimates are at the maximum to machine precision (Newton’s decrement judges convergence;iwls_tolandiwls_maxiterstill work). Coefficients, variances, degrees of freedom, fit statistics,fixef,predict,residandsampleagree withfeglm_stream’s; the probit’s variances use the expected information, as its iteratively reweighted fit does. Several fixed effects,bias_correction=True,bootcluster=, CRV3,separation_check=Falseandprofile=Truestill runfeglm_streamand return itsHDFEResult.
Faster (estimates unchanged)
Multinomial logit without fixed effects: about 5 times on dense designs, 1.3-1.8 times on designs of indicators. Ordered logit and probit without fixed effects: about 4 times. Each pass over the data is now one fused parallel kernel.
Multinomial and ordered logit with an absorbed fixed effect: 2-4 times. The first pass writes the rows grouped by the fixed effect, so each thread owns whole groups. Fixed effects with many small groups are a further 1.5-2 times faster, and the ordered logit and probit no longer slow down with many groups (100,000 groups: 47 s, now 2 s).
Robust and clustered variances: 3-18 times faster for the multinomial and ordered logit, and 2.5-7 times for the conditional logit and
cfelogit_stream, from a fused kernel that forms each score in a thread.fit.diagnostics["information"]says how the information matrix was formed.
Fixed
Cluster counts
Gand degrees of freedom count only clusters with rows used, and analytic weights are rescaled over the rows used after a fixed effect’s separation drops some, in the binary, multinomial and ordered logit, as hdfe-stream does. Before, the multinomial and ordered logit counted clusters whose rows were all dropped (variances slightly small) and scaled the weights over all rows read (iid variance and log-likelihoods).Sorting rows by a fixed effect respects
max_memory_gb: rows not already grouped are partitioned in memory to sort them, and when that would not fit they stay in their order and the passes are slower.resid()of a result fitted with an offset works.The within pseudo R-squared is nan for an ordered or multinomial fit with an offset and an absorbed fixed effect (the fixed-effect-only log-likelihood it came from ignored the offset).
0.3.1¶
Works with pyfixest 0.60 and 0.70. Formulas are read through hdfe-stream’s own formula parser, so either pyfixest version can be installed.
Requires hdfe-stream 0.10.0 or later.
0.3.0¶
The conditional logit is 2.3-2.8 times faster. Its log-likelihood, gradient and information are computed in one parallel numba pass over each block’s choice sets, giving the same estimates (4 million sets: 13 s, now 5 s; 8 million: 26 s, now 9-12 s). Designs of more than 256 parameters keep the previous BLAS-based pass, which is faster there.
The fixed-effects logit (
cfelogit_stream) balances work across threads better. Panels are dealt out in small pieces to more parts than threads, so a run of long panels no longer leaves threads idle: 0.55 of the time on panels sorted by length, 0.8-1.0 on shuffled ones, with the same results.Lower memory for conditional, mixed and fixed-effects logit. Choice sets are now checked and counted a block at a time instead of from a table of every set, so memory no longer grows with the number of sets (8 million sets: peak 1.71 GB down to 1.33 GB). Same errors, same results.
Memory tests.
pytest -m resourcefits the multinomial, ordered and conditional logit on data larger than a fixed memory limit; CI runs them on Linux, macOS and Windows. The defaultpytestrun leaves them out.Requires hdfe-stream 0.9.0 or later, which reads Parquet in less memory.
0.2.0¶
Fit resources (CPUs and memory) are constrained to your job’s limits automatically. On a cluster (PBS, Slurm, or a container with cgroup limits), importing
logit_streamcaps the threads of Polars, numba, BLAS and pyarrow at the job’s CPU allocation, and with Polars 2.x gives Polars a memory budget so its streaming operations spill to disk instead of growing. Caveats: the Polars budget is a target for its buffers, not a hard cap on the process, and Polars 1.x has no such option, so there only the threads are capped. The caps must be in place before Polars is imported: importlogit_streamfirst, or callstream_limits.set_limits()at the top of the script. With no limit found nothing changes. See “Resource limits and profiling” in the documentation.profile=True: estimate a fit’s memory and disk before running it. On every estimator, profiling returns ahdfe_stream.Profile– the estimated memory and disk, from one scan of the data – instead of fitting. P comes from the model and the outcome’s levels; the information matrix, its per-thread copies, the fixed effect’s cross blocks and the clusters’ score sums are sized from it. The multinomial logit with several fixed effects (fitted byfepois_streamon data expanded J-fold) is profiled as that regression, from counts worked out from one scan of the original data. Not modeled: random intercepts, and the mixed and nested logit’s workspaces beyond the per-thread accumulators.max_memory_gbandmemory_reserve: the fixed effect’s cross-block guard and the random intercepts’ guard (both a fixed 8 GB) and CRV3’s (25% of physical memory) now ask whether their exact allocation fits in the memory left (the limit less a reserve, less what the process holds). The limit defaults to the job’s allocation (cgroup,STREAM_LIMITS_MEMORY_GB, or on PBSqstat).n_threads=on every estimator: numba and BLAS threads for one fit, restored afterwards (Polars’ pool is fixed at import; see hdfe-stream’s “Resource limits and profiling”).diagnostics["peak_rss_gb"]in every fit.Depends on
stream-limits, and needs hdfe-stream 0.8.0 or later (the<0.7upper bound on hdfe-stream is gone).
0.1.0¶
The first release: logit models by streamed Newton’s method, for data that do not fit in memory. Formulas are pyfixest-style (hdfe-stream’s front end); data are Parquet files, Polars LazyFrames or DataFrames. Pass 0 writes the evaluated design as blocks of arrays, and each Newton step is one read of them.
Models¶
Binary:
logit_streamandprobit_stream(hdfe-stream’sfeglm_stream), with any number of absorbed fixed effects, fit statistics (loglik,aic,bic,pseudo_r2_within) and the incidental-parameter bias correction (bias_correction=).Multinomial and ordered:
mlogit_stream,ologit_streamandoprobit_stream.Conditional (McFadden):
clogit_stream; the conditional logit drops a choice set whole when an alternative has missing values (Stata’scmclogit), or keeps the complete alternatives withaltwise=True.Panels:
cfelogit_stream, Chamberlain’s conditional fixed-effects logit (Stata’sxtlogit, fe); matches statsmodels’ConditionalLogit.Nested logit (experimental):
nlogit_stream, a tree of nests of any depth, in the utility-consistent normalization (Stata’snlogitdefault). Matches R’s mlogit.Mixed logit (experimental prototype):
mixlogit_stream, by streamed maximum simulated likelihood, with normal and lognormal coefficients, panels, and Halton or pseudo-random draws regenerated in every read.Firth’s penalized likelihood (
firth=True) for the multinomial and binary logit, matching brglm2 and logistf; an optional ridge penalty (l2=).
Fixed and random effects¶
One absorbed fixed effect for the multinomial and ordered logit, solved exactly in each Newton step (
"y ~ x | puma"); several for the multinomial logit ("y ~ x | a + b"), fitted as the equivalent Poisson regression on the data expanded by outcome level (fepois_stream).Random intercepts for the multinomial and ordered logit (
random=; experimental): Laplace’s approximation (matchingglmerand ordinal’sclmm) or the joint mode, with an unstructured or diagonal covariance over the outcome levels. Nested groupings at any depth, optionally within a fixed effect, or two crossed.varcorr()andranef()report the covariance and the predicted effects.With a fixed effect,
loglik_fe_nullandpseudo_r2_within.
Inference and results¶
Variances: iid, heteroskedasticity-robust and clustered (one-way, multi-way, or by an interaction), with Stata’s small-sample factors for maximum likelihood; the cluster jackknife (
{"CRV3": var}). Frequency weights, and analytic weights rescaled to average 1 as Stata does.Results subclass hdfe-stream’s
RegressionResult:summary,tidy,confint(pointwise or simultaneous),wald_test,with_vcov,to_pyfixestandetable, and lazypredict,resid(including the ordered models’ probability-scale and generalized residuals),sampleandfixef.offset=and warm starts (start=).Convergence by the Newton decrement, with step-halving and a Levenberg–Marquardt fallback, collinearity detection and warnings for separated coefficients.
Scale and speed¶
GPU (
device="cuda", experimental): the Newton passes and the variances’ scores pass of every model, on an NVIDIA GPU through CuPy, with the blocks kept on the device when they fit;polars_engine="gpu"runs pass 0 on Polars’ GPU engine. Tested on a numpy stand-in; the GPU benchmark covers the hardware.0/1 design columns are stored as one byte each, and the multinomial information is computed from each row’s nonzeros when the design is mostly indicators.
Numba kernels that accumulate per thread (the nested, mixed and conditional fixed-effects logit).
storage=,dtype=,block_mb=andchunk_mb=control the block store.
Tools¶
logit_stream.simulate:simulate_people,simulate_choices(with nests, or tasks and a random coefficient) andsimulate_panel.examples/: a standalone script for each page of the guide.benchmarks/: ACS microdata, the Swissmetro survey, simulated panels and a GPU suite, against scikit-learn, statsmodels, mord, pyfixest, xlogit, jaxlogit, Biogeme, and R’s glm, fixest, nnet, MASS, ordinal, survival, mlogit, logitr and alpaca. Resumable.