Resource limits and profiling

Quickstart for cluster jobs

Two habits cover most of it.

1. Profile your fits before you run them. profile=True reads the data once and returns the estimated memory and disk instead of fitting, so you can request the right resources:

from dpd_stream import dpd_stream

p = dpd_stream("y ~ l(y, 1) + x | year | gmm(y, 2:.) + iv(x)", "data/*.parquet",
               panel=("id", "year"), profile=True)
print(p.recommend())      # ncpus=8  mem=14gb  disk=40gb
print(p.recommend().pbs)  # select=1:ncpus=8:mem=14gb

2. If your script imports Polars, run stream_limits.set_limits() before the import.

import stream_limits
stream_limits.set_limits()     # caps threads and Polars' memory at the job's allocation

import polars as pl            # only now
from dpd_stream import dpd_stream

Polars fixes its thread pool and its memory budget when it is imported, and nothing can change them afterwards, so a limit set after the import cannot reach it (you get a warning that says so). On a cluster, importing dpd-stream applies the limits for you; the line above matters when something imports Polars before dpd-stream does, such as your own import polars at the top of the script.

The rest of this page is the short version of hdfe-stream’s Resource limits and profiling, the full reference.

What the profile models

dpd-stream’s memory is set by the number of instrument columns jf, which the formula and the number of periods fix before any data is read (jf is an upper bound: columns empty in the sample are dropped after pass 1), and by the thread count: S1 and, with the identification tests on (idtests, the default "auto"), Z'Z each take n_threads x jf^2 x 8 bytes. collapse=True, tighter lag limits, idtests=False and fewer threads all shrink it, and when the accumulators would not fit in the memory left, pass 1 uses fewer of them (fewer threads, same answer).

Thread and memory limits

Polars, numba, BLAS and pyarrow each default to every core of the node. dpd-stream caps all of them at a scheduler’s allocation automatically when it finds one, and gives Polars 2.x a memory budget past which it spills to disk. This is done by the standalone stream-limits package; its documentation covers thread limits, memory limits and schedulers and containers (PBS, Slurm, cgroups, and a qstat fallback for sites that set no memory variable). python -m stream_limits shows what was found, from inside a job. The n_threads= option holds numba and BLAS to that many threads for one fit.

Choices that follow the memory left

Choices that trade speed for memory follow what is left of the job’s memory: the max_memory_gb option sets the limit and memory_reserve the share held back (default 0.15), and what cannot shrink is refused up front with the numbers. Every fit records diagnostics["peak_rss_gb"]. The full explanation, with the profile’s accuracy and what each package does with the headroom, is in hdfe-stream’s documentation.