solprop.driver.executors.queue#
Drain durable point work across controllers sharing a filesystem.
A Queue directory holds an immutable manifest (the scan, its retention
settings and its builder’s identity), one state row per point and the published
artifacts. Each controller claims points under a lock with a refreshed lease
(_Claim, _Heartbeat; the lock and row helpers are
solprop.driver.executors._queue_state), runs them in its own
Pool, and publishes each result as a checked
artifact only while its claim token still holds. collect() then validates the
artifacts and writes them, idempotently, into the collected scan file through one
locked writer. Leases, takeover and the publication order are in docs/dev/scans.md
(“Durable executor implementation”).
Functions
|
Idempotently collect validated artifacts through one locked HDF5 writer. |
Classes
|
Drain a shared durable queue using a local persistent process pool. |
- class solprop.driver.executors.queue.Queue(path, stale_age=600.0, pool=<factory>, chunk=None)[source]#
Bases:
objectDrain a shared durable queue using a local persistent process pool.
- Attributes:
- pathpathlib.Path, str or None
Queue directory. None derives it from the collected output path for the CLI.
- stale_agefloat
Positive remote lease duration, s. Live local owners remain protected.
- poolPool
Resource policy for workers within this controller.
- chunktuple of int or None
Optional zero-based round-robin chunk number and count.
- prepare(request)[source]#
Create or validate the manifest under an automatically released lock.
- Parameters:
- requestScanRequest
Scan identity, retention settings and worker to bind before dispatch.
- run(points, work)[source]#
Run claimed points while a separate thread refreshes their leases.
- Parameters:
- pointsIterable[T]
Lazily supplied point descriptions retaining global indices.
- workCallable[[T], R]
Pickleable point worker with its memory probe.
- Yields:
- object
Completed result from the prepared execution policy.
- solprop.driver.executors.queue.collect(path, output)[source]#
Idempotently collect validated artifacts through one locked HDF5 writer.
- Parameters:
- pathpathlib.Path or str
Queue directory containing its manifest and published point artifacts.
- outputpathlib.Path or str
Collected HDF5 destination.
- Returns:
- ScanOutput
Reader masking unfinished points.
- Raises:
- OutputError
If a scan, point, configuration or artifact digest conflicts.