Shared Memory¶

Provides assets for transferring data between multiple Python processes via a shared NumPy array memory buffer.

class ataraxis_data_structures.shared_memory.SharedMemoryArray(name, shape, datatype, buffer, *, auto_connect=True)¶

Bases: object

Wraps a NumPy n-dimensional array object and exposes methods for accessing the array’s data from multiple different Python processes.

During initialization, this class creates a persistent memory buffer to which it connects from different Python processes. The data inside the buffer is accessed via an n-dimensional NumPy array with optional locking to prevent race conditions.

Notes

Supports instantiation inside the main runtime thread via the create_array() method alone. Do not attempt to instantiate the class manually. The create_array() method returns an instance already connected to the shared memory buffer. Each process that receives the instance connects as part of the transfer, so no process calls connect() to reach the array data.

The creating process destroys the shared memory buffer when its instance is garbage-collected. Every process that receives the instance only disconnects from the buffer, which leaves the creating process solely responsible for the buffer’s lifetime.

Shared memory buffers are garbage-collected differently depending on the host Operating System. On Windows, garbage collection is handed off to the OS and cannot be enforced manually. On Unix (macOS and Linux), the buffer can be garbage-collected by calling the destroy() method.

Parameters:

name (str) – The unique name to use for the shared memory buffer.
shape (tuple[int, ...]) – The shape of the NumPy array used to access the data in the shared memory buffer.
datatype (dtype[Any]) – The datatype of the NumPy array used to access the data in the shared memory buffer.
buffer (SharedMemory) – The SharedMemory buffer that stores the shared data.
auto_connect (bool, default: True) – Determines whether the processes that receive this instance connect to the shared memory buffer as part of unpickling it.

_name¶: Stores the name of the shared memory buffer.

_shape¶: Stores the shape of the NumPy array used to access the buffered data.

_datatype¶: Stores the datatype of the NumPy array used to access the buffered data.

_buffer¶: Stores the Shared Memory buffer object.

_lock¶: Stores the Lock object used to prevent multiple processes from working with the shared data at the same time.

_array¶: Stores the NumPy array used to interface with the data stored in the shared memory buffer.

_connected¶: Determines whether the instance is connected to the shared memory buffer.

_destroy_buffer¶: Determines whether the shared memory buffer is destroyed when this instance is garbage-collected.

_auto_connect¶: Determines whether the processes that receive this instance connect to the shared memory buffer automatically.

array(*, with_lock=True)¶

Returns a context manager for accessing the managed shared memory array with optional locking.

Notes

When with_lock is True (default), the lock is held for the entire duration of the context. Keep operations concise to avoid blocking other processes. When with_lock is False, ensure no other processes are writing to avoid race conditions and data corruption.

The returned array is the actual shared array, not a copy. All modifications to the array are immediately visible to other processes.

Parameters:: with_lock (bool, default: True) – Determines whether to acquire the multiprocessing Lock before accessing the array. Acquiring the lock prevents collisions with other Python processes trying to simultaneously access the array’s data.
Yields:: The shared NumPy array that can be directly manipulated using any NumPy operations. Changes made to this array directly affect the data stored in the shared memory buffer.
Raises:: ConnectionError – If the class instance is not connected to the shared memory buffer.

connect()¶

Guarantees that the instance is connected to the shared memory buffer by the time the call returns.

Calling this method is optional for an array created through create_array(), as the creating process and every process that receives the instance are connected already, which makes the call a no-op. Calling it anyway establishes the connection at a point the caller chooses rather than leaving it implicit, and it reconnects an instance that called disconnect().

Return type:: None

Notes

An array created with the auto_connect flag disabled relies on each receiving process calling this method before it accesses the array data.

Raises:: FileNotFoundError – If the shared memory buffer with the instance’s name does not exist. This typically indicates that the buffer has already been destroyed via the destroy() method.

classmethod create_array(name, prototype, *, exists_ok=False, auto_connect=True)¶

Creates a SharedMemoryArray instance using the input prototype NumPy array.

Notes

Applies only when the array is first created in the main runtime thread (scope). The returned instance is already connected to the shared memory buffer, and every process it is passed to connects as part of the transfer, so no process calls connect() to reach the array data.

The calling process destroys the shared memory buffer when the returned instance is garbage-collected. The instance therefore has to stay referenced for as long as any process still uses the buffer, and calling destroy() ends the buffer’s lifetime ahead of that point.

Parameters:

name (str) – The unique name to use for the shared memory buffer.
prototype (NDArray[Any]) – The prototype NumPy array instance for the created SharedMemoryArray.
exists_ok (bool, default: False) – Determines how the method handles the case where the shared memory buffer with the same name already exists. If False, the method raises an exception. If True, the method unlinks the existing buffer and creates a new buffer using the input name and prototype data.
auto_connect (bool, default: True) – Determines whether the processes that receive the created instance connect to the shared memory buffer as part of unpickling it. Disabling this defers the connection to each receiving process’s own connect() call, which keeps a buffer that is destroyed between the spawn and the worker’s first access from failing the worker’s startup.

Return type:

SharedMemoryArray

Returns:

The created and connected SharedMemoryArray instance.

Raises:

TypeError – If the input prototype is not a NumPy array.
ValueError – If the input prototype is an empty NumPy array, as a shared memory buffer cannot be created with a size of zero bytes.
FileExistsError – If a shared memory object with the same name as the input name argument value already exists and the exists_ok flag is False. Also raised if the exists_ok flag is True and the buffer name is still claimed by an open handle after the existing buffer is unlinked, which is the state Windows leaves behind while any handle to the buffer remains open.
IndexError – If the input prototype is a zero-dimensional NumPy array, which passes the type check above and then fails the slice assignment that fills the buffer. The buffer is released before the error propagates, so the buffer name is left free for a later call to claim.

property datatype: dtype[Any]¶: Returns the datatype used by the shared memory array.

destroy()¶

Requests the instance’s shared memory buffer to be destroyed.

Applies only to a single call issued from the highest runtime scope. Calling this method while having SharedMemoryArray instances connected to the buffer leads to undefined behavior.

Return type:: None

Notes

On Windows, the underlying unlink() call has no effect, as the Operating System destroys the buffer only after every handle to it is closed. The method still disconnects this instance from the buffer and releases its local handle on all platforms, so the instance cannot access the array data until connect() is called again.

Unlinking addresses the buffer by name. Recreating a name through the exists_ok flag therefore strips the destruction right from the instance that held it, which confines this method to the buffer the instance created. Two processes creating the same name concurrently fall outside that protection, so buffer names have to stay unique across every process that runs at the same time.

disconnect()¶

Disconnects from the shared memory buffer, preventing the instance from accessing and manipulating the shared data.

Applies to each Python process that no longer requires shared buffer access, and to a process’s shutdown sequence. A process that exits releases its handle regardless, so the call bounds the handle’s lifetime to the work that needs it rather than to the process.

Return type:: None

Notes

Releases the local reference to the shared memory buffer without destroying it, potentially enabling the buffer to be garbage-collected by the Operating System. Use the destroy() method on Unix-based Operating Systems to destroy the buffer.

property is_connected: bool¶: Returns True if the instance is connected to the shared memory buffer that stores the array data.

property name: str¶: Returns the name of the shared memory buffer.

property shape: tuple[int, ...]¶: Returns the shape of the shared memory array.

Data Structures¶

Provides the YAML-serializable dataclass base and the file-locked processing pipeline state tracker.

class ataraxis_data_structures.data_structures.JobState(job_name, specifier='', status=ProcessingStatus.SCHEDULED, executor_id=None, error_message=None, started_at=None, completed_at=None)¶

Bases: object

Stores the metadata and the current runtime status of a single job in the processing pipeline.

completed_at: int | None¶: The UTC timestamp (microsecond-precision epoch) when the job completed (succeeded or failed).

error_message: str | None¶: An optional error message describing why the job failed.

executor_id: str | None¶: An optional identifier for the executor running the job (e.g., a SLURM job ID, a process PID, or any user-defined string).

job_name: str¶: The descriptive name of the job.

specifier: str¶: An optional specifier that differentiates instances of the same job, for example, when running the same job over multiple batches of data.

started_at: int | None¶: The UTC timestamp (microsecond-precision epoch) when the job started running.

status: ProcessingStatus¶: The current status of the job.

class ataraxis_data_structures.data_structures.ProcessingStatus(*values)¶

Bases: IntEnum

Defines the status codes used by the ProcessingTracker instances to communicate the runtime state of each job making up the managed data processing pipeline.

FAILED = 3¶: Indicates the job encountered a runtime error and was not completed.

RUNNING = 1¶: Indicates the job is currently being executed.

SCHEDULED = 0¶: Indicates the job is scheduled for execution.

SUCCEEDED = 2¶: Indicates the job has been completed successfully.

class ataraxis_data_structures.data_structures.ProcessingTracker(file_path, jobs=<factory>)¶

Bases: YamlConfig

Tracks the state of a data processing pipeline and provides tools for communicating this state between multiple processes and host machines.

Notes

All modifications to the tracker file require the acquisition of the .lock file, which ensures exclusive access to the tracker’s data, allowing multiple independent processes (jobs) to safely work with the same tracker file.

align_jobs(jobs, universe=None)¶

Aligns the tracker’s job registry with the jobs requested for the current pipeline invocation.

Notes

Foreign entries are detected against universe, the full set of jobs the pipeline could produce, rather than against the requested subset. That distinction lets an invocation run part of a pipeline while its siblings keep their recorded state. A registry holding entries outside the universe means the pipeline’s own definition has changed since the tracker was written, so those entries alone are discarded and reported through a warning, and every in-universe job keeps its recorded state.

Otherwise, the method additively registers any requested job the registry is missing, and is a no-op when every requested job is already present.

Parameters:

jobs (list[tuple[str, str]]) – The (job_name, specifier) tuples the current invocation intends to execute.
universe (list[tuple[str, str]] | None, default: None) – The (job_name, specifier) tuples enumerating every job the pipeline could produce for its current definition, used only to detect foreign entries. Defaults to jobs, which is correct for a pipeline whose requested set is always its full universe.

Return type:

list[str]

Returns:

A list of job IDs corresponding to the requested jobs.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
ValueError – If the requested job list is empty, or if any requested job is not part of the resolved universe.

property complete: bool¶

Returns True when the tracked pipeline has jobs and all of them have been marked as succeeded.

Raises:: TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.

complete_job(job_id)¶

Marks a target job as successfully completed.

Clears the error message recorded by any previous attempt at the job.

Parameters:

job_id (str) – The unique identifier of the job to mark as complete.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
ValueError – If the specified job ID is not found in the managed tracker file.

Return type:

None

classmethod discover(root_directory, tracker_name)¶

Discovers every processing tracker with the target filename stored anywhere under the root directory.

Notes

Each returned instance is bound to the file it was discovered at and holds no job state until one of its readers loads that file, so surveying a large tree costs one traversal rather than one read per tracker.

Parameters:

root_directory (Path) – The root directory whose tree is searched.
tracker_name (str) – The exact filename every discovered tracker .yaml file carries.

Return type:

list[Self]

Returns:

A tracker bound to every matching file found anywhere under the root directory, sorted by file path.

Raises:

OSError – If the root directory does not exist, is not a directory, or cannot be read, or if any directory beneath it cannot be read.

property encountered_error: bool¶

Returns True when any of the tracked pipeline’s jobs has been marked as failed.

Raises:: TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.

fail_job(job_id, error_message=None)¶

Marks the target job as failed.

Parameters:

job_id (str) – The unique identifier of the job to mark as failed.
error_message (str | None, default: None) – An optional error message describing why the job failed.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
ValueError – If the specified job ID is not found in the managed tracker file.

Return type:

None

file_path: Path¶: The path to the .YAML file used to cache the tracker’s data on disk. Excluded from the serialized document, since it records where the tracker lives rather than the pipeline state the tracker holds.

find_jobs(job_name=None, specifier=None)¶

Searches for jobs matching the given name and/or specifier patterns.

Supports partial matching (substring search) on job names and specifiers. If both parameters are provided, jobs must match both patterns.

Parameters:

job_name (str | None, default: None) – A substring to match against job names. If None, matches any job name.
specifier (str | None, default: None) – A substring to match against specifiers. If None, matches any specifier.

Return type:

dict[str, tuple[str, str]]

Returns:

A dictionary mapping matching job IDs to (job_name, specifier) tuples. Calling the method without arguments matches every tracked job.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.

static generate_job_id(job_name, specifier='')¶

Generates a unique hexadecimal job identifier based on the job’s name and optional specifier using the xxHash64 checksum generator.

Notes

A colon joins the two components inside the hashed string, so neither component may contain one. Were a colon allowed, the pairs (’data:batch’, ‘’) and (‘data’, ‘batch’) would join to the same string and therefore to the same identifier, collapsing two declared jobs onto one registry entry.

Parameters:

job_name (str) – The descriptive name for the processing job (e.g., ‘process_data’). Cannot contain a colon.
specifier (str, default: '') – An optional specifier that differentiates instances of the same job (e.g., ‘batch_101’). Cannot contain a colon.

Return type:

str

Returns:

The unique hexadecimal identifier for the target job.

Raises:

ValueError – If the job name or the specifier contains a colon.

get_job_info(job_id)¶

Returns the full JobState object for the specified job.

Notes

The returned state is a copy, so mutating it does not affect the tracker. Use the dedicated mutators to change job state.

Parameters:

job_id (str) – The unique identifier of the job to query.

Return type:

JobState

Returns:

A copy of the JobState object containing all metadata for the job.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
ValueError – If the specified job ID is not found in the managed tracker file.

get_job_status(job_id)¶

Queries the current runtime status of the target job.

Parameters:

job_id (str) – The unique identifier of the job for which to query the runtime status.

Return type:

ProcessingStatus

Returns:

The status the tracker currently records for the target job.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
ValueError – If the specified job ID is not found in the managed tracker file.

get_jobs_by_status(status)¶

Returns all job IDs that have the specified status.

Parameters:

status (ProcessingStatus | str) – The status to match, given as a ProcessingStatus member or its member name string.

Return type:

list[str]

Returns:

The identifiers of every tracked job currently holding the requested status.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
KeyError – If status is a string that does not name a valid ProcessingStatus member.

get_summary()¶

Returns a summary of job counts by status.

Return type:: dict[ProcessingStatus, int]
Returns:: The number of tracked jobs currently holding each status, with every status present even when its count is zero.
Raises:: TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.

initialize_jobs(jobs)¶

Configures the tracker with the list of one or more jobs to be executed during the pipeline’s runtime.

Notes

If the job already has a section in the tracker, this method emits a warning and does not duplicate or modify the existing job entry. Use the reset() method to clear all cached job states.

Parameters:

jobs (list[tuple[str, str]]) – A list of (job_name, specifier) tuples defining the jobs to track. Each tuple contains the descriptive job name and an optional specifier string. Use an empty string for jobs without a specifier.

Return type:

list[str]

Returns:

A list of job IDs corresponding to the input jobs.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
ValueError – If any job name or specifier contains a colon.

jobs: dict[str, JobState]¶: Maps the unique identifiers of the jobs that make up the processing pipeline to their current state and metadata.

lock_path: str¶: The path to the .LOCK file used to ensure process-safe access to the tracker’s data. Excluded from the serialized document, since it is derived from the file path.

reset()¶

Resets the tracker file to the default state.

Raises:: TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
Return type:: None

reset_jobs(job_ids)¶

Resets the specified jobs back to SCHEDULED status, leaving every other job’s state untouched.

Clears the error_message, started_at, completed_at, and executor_id fields of each targeted job.

Notes

Every job outside job_ids keeps its recorded status, executor, and timestamps.

A job ID the tracker does not know means the caller’s view of the registry disagrees with the registry itself, so the method rejects the whole request. Changing which jobs a tracker holds is the job of align_jobs. The membership check completes before any job is modified, so a rejected request leaves the tracker untouched.

Parameters:

job_ids (list[str]) – The unique identifiers of the jobs to reset.

Return type:

list[str]

Returns:

A list of the reset job IDs, in registry order.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
ValueError – If any of the specified job IDs is not found in the managed tracker file.

resolve_job(job_id, universe)¶

Resolves the job that a hexadecimal identifier names within the pipeline’s declared job universe.

Notes

An invocation handed the identifier of the single job it is to run has to recover that job’s name and specifier before it is able to execute it. Resolving against the universe rather than against the tracker’s own registry lets the invocation reject an unknown identifier before the tracker holds any entry for it, which keeps a mistyped identifier from registering a job the pipeline cannot produce.

Parameters:

job_id (str) – The unique hexadecimal identifier of the job to resolve.
universe (list[tuple[str, str]]) – The (job_name, specifier) tuples enumerating every job the pipeline could produce for its current definition.

Return type:

tuple[str, str]

Returns:

The name and the specifier of the job the identifier names.

Raises:

ValueError – If the declared universe is empty, or if the identifier names no job within it.

static resolve_status(summary)¶

Resolves the high-level progress label that a tracker summary’s job counts describe.

Notes

Applies a fixed priority. A summary counting a failed job resolves to FAILED whatever else it counts, one whose every job succeeded resolves to COMPLETED, one counting a running job resolves to PROCESSING, and one whose every job is still scheduled resolves to NOT_STARTED. Every remaining case resolves to IN_PROGRESS, which covers a summary counting no jobs at all, since the COMPLETED and NOT_STARTED branches each require at least one counted job.

Counts aggregated across several trackers resolve the same way, so a caller reporting on a directory of trackers labels the group with the same priority it labels each member.

Parameters:: summary (Mapping[str, int]) – The job counts to resolve, carrying the ‘total’, ‘succeeded’, ‘failed’, ‘running’, and ‘scheduled’ keys that summarize() produces. An absent key counts as zero.
Return type:: TrackerStatus
Returns:: The label matching the highest-priority condition the counts satisfy.

classmethod restore_excluded_fields(data, file_path)¶

Reattaches the reconstructed tracker to the file it was read from.

Notes

This method overrides the YamlConfig implementation and runs only as part of that class’s deserialization machinery, which calls it from from_yaml() between reading the document and building the instance. Nothing calls it directly.

The tracker marks both of its path fields with YAML_EXCLUDE_METADATA, so the document it writes holds the job registry alone and offers the constructor no path to take. Supplying the path here keeps every instance from_yaml() returns bound to a real file.

Parameters:

data (dict[Any, Any]) – The top-level mapping read from the tracker .yaml file.
file_path (Path) – The path the mapping was read from.

Return type:

dict[Any, Any]

Returns:

The mapping extended with the tracker’s own file path.

retry_failed_jobs()¶

Resets all failed jobs back to SCHEDULED status for retry.

Clears the error_message, started_at, completed_at, and executor_id fields for each failed job.

Return type:: list[str]
Returns:: A list of job IDs that were reset for retry.
Raises:: TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.

run_job(job_id, executor_id=None)¶

Runs a single tracked job, recording its start, its completion, and its failure on the tracker.

Notes

Owns the job’s state transitions and leaves the work itself to the wrapped block. The guard spans both the block and the completion call, so an Exception raised by either marks the job failed, records the exception’s message as the failure reason, and re-raises the exception unchanged.

A BaseException such as KeyboardInterrupt propagates with the job left running, since an interrupted job did not fail on its own terms and its executor is the authority on what became of it.

Parameters:

job_id (str) – The unique identifier of the job to run.
executor_id (str | None, default: None) – An optional explicit identifier for the executor running the job. When None (default), the identifier is resolved automatically from the runtime environment.

Yields:

None. The tracker holds the job in its running state for the duration of the block.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
ValueError – If the specified job ID is not found in the managed tracker file.

snapshot()¶

Returns a point-in-time copy of the tracker’s complete job registry.

Notes

Reads the whole registry under a single lock acquisition, so the returned states are consistent with each other.

The returned states are copies, so mutating them does not affect the tracker. Use the dedicated mutators to change job state.

A tracker file that does not exist yields an empty registry and is left uncreated, so probing a pipeline that has never run leaves its output directory unchanged.

Return type:: dict[str, JobState]
Returns:: A dictionary mapping every tracked job ID to a copy of its JobState, or an empty dictionary when the tracker file does not exist.
Raises:: TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.

start_job(job_id, executor_id=None)¶

Marks the target job as running and records the identifier of the executor running it.

Clears the error message and completion timestamp recorded by any previous attempt at the job.

Parameters:

job_id (str) – The unique identifier of the job to mark as started.
executor_id (str | None, default: None) – An optional explicit identifier for the executor running the job. When None (default), the identifier is resolved automatically from the runtime environment, preferring a recognized job scheduler’s job ID and falling back to the process ID, each tagged with its scheme.

Raises:

TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.
ValueError – If the specified job ID is not found in the managed tracker file.

Return type:

None

summarize()¶

Returns the tracker’s job registry as per-job details, aggregate job counts, and a progress label.

Notes

Reports every field JobState carries, so a consumer serializing the returned details preserves the registry rather than a projection of it. A job that recorded no failure reason omits the ‘error_message’ key instead of carrying it as an empty value.

Return type:: dict[str, Any]
Returns:: A dictionary carrying the per-job details under ‘jobs’, the tracked job total alongside the count of the jobs holding each status under ‘summary’, and the label those counts resolve to under ‘status’.
Raises:: TimeoutError – If the .LOCK file for the tracker .YAML file cannot be acquired within the timeout period.

class ataraxis_data_structures.data_structures.TrackerStatus(*values)¶

Bases: StrEnum

Defines the high-level progress labels that summarize the job registry of a ProcessingTracker instance.

Notes

A label describes the pipeline the tracker follows, while a ProcessingStatus member describes one job inside that pipeline.

COMPLETED = 'completed'¶: Indicates every tracked job succeeded.

FAILED = 'failed'¶: Indicates at least one tracked job failed.

IN_PROGRESS = 'in_progress'¶: Indicates the tracked jobs have mixed outcomes with none running and none failed, which also covers a tracker holding no jobs at all.

NOT_STARTED = 'not_started'¶: Indicates every tracked job is still scheduled.

PROCESSING = 'processing'¶: Indicates at least one tracked job is currently running.

class ataraxis_data_structures.data_structures.YamlConfig¶

Bases: object

Extends the standard Python dataclass with methods to save and load its data from a .yaml (YAML) file.

Notes

This class is designed to be subclassed by custom dataclasses so that they inherit the YAML saving and loading functionality. Serialization automatically converts Path instances to strings, Enum members to their raw values, and tuples to lists. Deserialization reverses these conversions based on the dataclass’s type annotations.

classmethod from_yaml(file_path)¶

Instantiates the class using the data loaded from the provided .yaml (YAML) file.

Notes

Deserialization automatically converts YAML-native types back to the annotated Python types: strings to Path instances, raw values to Enum members, and lists to tuples where applicable. Type hooks are derived from the dataclass’s field annotations, so no manual conversion boilerplate is needed in subclasses.

Parameters:

file_path (Path) – The path to the .yaml file that stores the instance’s data.

Return type:

Self

Returns:

A new class instance that stores the data read from the .yaml file.

Raises:

ValueError – If the provided file path does not point to a .yaml or .yml file, or if the file does not contain a top-level mapping.
FileNotFoundError – If no file exists at the provided file path.
YAMLError – If the file does not contain a well-formed YAML document.
MissingValueError – If the document omits a field the class requires and the class supplies no default for it.

classmethod restore_excluded_fields(data, file_path)¶

Returns the loaded mapping extended with the values of any field this class excludes from serialization.

Notes

This method exists to be replaced by subclasses. The implementation here excludes no field and returns the mapping untouched, which is correct for every class that serializes all of its fields.

A subclass marking a constructor-required field with YAML_EXCLUDE_METADATA overrides this method to supply that field’s value. The written document carries no entry for such a field, so deserialization cannot build the instance without one. The path the document was read from is passed in, because a field excluded this way usually records where the instance lives.

from_yaml() calls this method between reading the document and building the instance, so it belongs to the deserialization machinery rather than to the API a caller invokes.

Parameters:

data (dict[Any, Any]) – The top-level mapping read from the .yaml file.
file_path (Path) – The path the mapping was read from.

Return type:

dict[Any, Any]

Returns:

The mapping to build the instance from, which is the input unchanged when no field is excluded.

to_yaml(file_path)¶

Saves the instance’s data as the specified .yaml (YAML) file.

Notes

Path fields are serialized as strings, Enum fields as their raw values, and tuples as lists. This keeps YAML files human-readable while preserving type fidelity on round-trip via from_yaml() for concretely annotated fields. A field whose annotation unions Path with str is not reliably discriminated on load, so annotate such a field concretely when the restored type matters.

The file is written through a temporary file and renamed into place, so a process killed mid-write leaves the previously saved file intact. Both reading and writing use UTF-8 regardless of the host locale. The written file carries the permissions the process umask allows, matching the built-in open().

Parameters:: file_path (Path) – The path to the .yaml file to write.
Raises:: ValueError – If file_path does not point to a file with a ‘.yaml’ or ‘.yml’ extension.
Return type:: None

ataraxis_data_structures.data_structures.yaml_config.YAML_EXCLUDE_METADATA: MappingProxyType = mappingproxy({'yaml_exclude': True})¶

The dataclass field metadata that keeps a field out of the serialized document.

Notes

A field declared as field(metadata=YAML_EXCLUDE_METADATA) is skipped when the instance is written, which suits a field describing where the instance lives rather than what it holds. A class excluding a field that its constructor requires supplies the value back through restore_excluded_fields().

Data Loggers¶

Provides assets for saving (logging) various forms of data to disk and for reading the resulting log archives.

class ataraxis_data_structures.data_loggers.DataLogger(output_directory, instance_name, thread_count=5, poll_interval=5)¶

Bases: object

Manages the runtime of a data logger that saves serialized data collected from multiple concurrently active sources.

The logger runs in a separate process and uses multiple concurrent threads to optimize the I/O operations associated with saving the data to disk.

Notes

The start() method must complete before any data is submitted for logging.

Use the multiprocessing Queue exposed via the input_queue property to send the data to the logger. The data must be packaged into the LogPackage class instance before it is submitted to the queue.

Submitting data to the input queue does not confirm that the data reached the disk, since the logger process writes the entries asynchronously. A write that fails while the logger is running terminates the logger process, which the watchdog thread reports as a ChildProcessError. A write that fails during the shutdown sequence is reported by stop() as a warning.

Parameters:

output_directory (Path) – The directory in which to save the logged data. The data is saved under a subdirectory named ‘{instance_name}_data_log’.
instance_name (str) – The name of the logger instance. This name has to be unique across all concurrently active DataLogger instances.
thread_count (int, default: 5) – The number of threads to use for saving the data to disk. It is recommended to use multiple threads to parallelize the I/O operations associated with writing the logged data to disk. Values below 1 are clamped to 1.
poll_interval (int, default: 5) – The interval, in milliseconds, between polling the input queue. Primarily, this is designed to optimize the CPU usage during light workloads. Setting this to 0 disables the polling delay mechanism. Negative values are clamped to 0.

_started¶: Tracks whether the logger process is running.

_shutdown_lock¶: Stores the lock that serializes the shutdown sequence between stop() and the watchdog thread, so exactly one of the two retires the instance.

_multiprocessing_context¶: Stores the spawn-based multiprocessing context used to create the manager and the logger process.

_multiprocessing_manager¶: Stores the manager object used to instantiate and manage the multiprocessing Queue.

_thread_count¶: Stores the number of concurrently active data saving threads.

_poll_interval¶: Stores the data queue poll interval, in milliseconds.

_name¶: Stores the name of the data logger instance.

_output_directory¶: Stores the directory where the data is saved.

_input_queue¶: Stores the multiprocessing Queue used to buffer and pipe the data to the logger process.

_logger_process¶: Stores the Process object that runs the data logging cycle.

_terminator_array¶: Stores the shared memory array used to terminate (shut down) the logger process.

_watchdog_thread¶: Stores the thread used to monitor the runtime status of the remote logger process.

property alive: bool¶: Returns True if the instance’s logger process is currently running.

property input_queue: Queue¶: Returns the multiprocessing Queue used to buffer and pipe the data to the logger process.

property name: str¶: Returns the name of the instance.

property output_directory: Path¶: Returns the path to the directory where the data is saved.

start()¶

Starts the remote logger process and the assets used to control and monitor the logger’s uptime.

Return type:: None

stop()¶

Stops the logger process once it saves all buffered data and releases reserved resources.

Return type:: None

Notes

A logger process that failed to save one of its buffered entries is reported through a warning rather than an exception. This method only reaches that check during the shutdown sequence, where raising would mask the shutdown work of the caller that is already unwinding, and where the data is lost either way.

The shutdown is claimed under a lock the watchdog thread takes as well, so exactly one of the two performs the teardown. The lock is released before this method joins that thread, since the watchdog acquires the same lock and holding it across the join would leave each side waiting on the other.

class ataraxis_data_structures.data_loggers.LogArchiveReader(archive_path, onset_us=None)¶

Bases: object

Reads and iterates through .npz log archives generated by DataLogger instances.

Notes

Each .npz archive contains messages from a single source (producer). Messages are stored with the structure: [source_id (1 byte)][timestamp (8 bytes)][payload (N bytes)].

The first message with a timestamp value of 0 contains the onset timestamp as its payload. This onset timestamp is the UTC epoch reference used to convert elapsed microseconds to absolute timestamps.

Parameters:

archive_path (Path) – The path to the .npz log archive file to read.
onset_us (uint64 | None, default: None) – The pre-discovered onset timestamp in microseconds since epoch. If provided, skips onset discovery.

_archive_path¶: Stores the path to the log archive file.

_onset_us¶: Stores the onset timestamp if pre-provided.

_message_keys¶: Caches the list of message keys (excluding the onset message).

Raises:

FileNotFoundError – If the specified archive file does not exist.
ValueError – If the archive lacks a valid onset timestamp message when the onset timestamp is accessed.

get_batches(workers=-1, batch_multiplier=4)¶

Divides message keys into batches optimized for parallel processing.

Notes

Uses over-batching (creating more batches than workers) to improve load balancing when message processing times vary. The batch_multiplier parameter controls the degree of over-batching.

For archives with fewer messages than PARALLEL_PROCESSING_THRESHOLD, returns a single batch containing all message keys, or an empty list when the archive contains no data messages.

Parameters:

workers (int, default: -1) – The number of worker processes to optimize batching for. A value less than 1 uses all available CPU cores minus 2.
batch_multiplier (int, default: 4) – The over-batching factor. Creates up to (workers * batch_multiplier) batches for better load distribution.

Return type:

list[list[str]]

Returns:

The message key batches, each of which is accepted as the keys argument of iter_messages().

iter_messages(keys=None)¶

Iterates through messages in the archive, yielding LogMessage instances.

Notes

Opens the archive once and keeps it open for the duration of iteration, decoding each requested entry on access.

If keys is provided, only iterates through the specified messages.

Parameters:: keys (list[str] | None, default: None) – The message keys to iterate. If None, iterates through every data message in the archive.
Yields:: LogMessage instances containing the absolute timestamp and payload for each message.

property message_count: int¶: Returns the number of data messages in the archive, excluding the onset message.

property onset_timestamp_us: uint64¶

Returns the onset timestamp in microseconds since epoch, taken from the pre-provided value when available or discovered by scanning the archive for the first message with a timestamp of 0.

Raises:: ValueError – If the archive does not contain a valid onset timestamp message.

read_all_messages()¶

Reads all messages from the archive and returns them as arrays.

Notes

Loads every message into memory at once.

Return type:: tuple[NDArray[uint64], list[NDArray[uint8]]]
Returns:: The absolute message timestamps in microseconds, paired with a list of the per-message payload arrays.

class ataraxis_data_structures.data_loggers.LogMessage(timestamp_us, payload)¶

Bases: object

Stores a single message extracted from a log archive.

Notes

The structure of the payload is domain-specific and must be parsed by the consumer.

payload: NDArray[uint8]¶: The message payload as a byte array.

timestamp_us: uint64¶: The absolute UTC timestamp of when the message was logged, in microseconds since epoch.

class ataraxis_data_structures.data_loggers.LogPackage(source_id, acquisition_time, serialized_data)¶

Bases: object

Stores the data and ID information to be logged by the DataLogger class and exposes methods for packaging this data into the format expected by the logger.

acquisition_time: uint64¶: The timestamp of when the data was acquired. This value typically communicates the number of microseconds elapsed since the onset of the data acquisition runtime.

property data: tuple[str, NDArray[uint8]]¶: Returns the filename and the serialized data package to be processed by a DataLogger instance.

serialized_data: NDArray[uint8]¶: The serialized data to be logged, stored as a one-dimensional byte array.

source_id: uint8¶: The ID code of the source that produced the data. Has to be unique across all systems that send data to the same DataLogger instance during runtime.

ataraxis_data_structures.data_loggers.assemble_log_archives(log_directory, max_workers=None, *, remove_sources=True, memory_mapping=True, verbose=False, verify_integrity=False)¶

Consolidates all .npy files in the target log directory into .npz archives, one for each unique source.

Notes

Log entries are grouped into archives by their source, and the entries within each archive are sorted by their acquisition timestamp value before consolidation. The consolidated archive names include the ID code of the source that generated the original log entries.

Discovery covers the target directory itself and does not descend into its subdirectories, since one DataLogger instance is the unit of serialization and owns exactly one log directory. Entry names carry the source ID and the acquisition timestamp alone, so entries from two logger instances that share a source ID would collide on name if a single call consolidated both.

Parameters:

log_directory (Path) – The path to the directory that stores the log entries of one DataLogger instance as .npy files, which is the directory the instance exposes through its output_directory property.
max_workers (int | None, default: None) – Determines the number of worker processes and threads used to process the data in parallel. A positive value is honored exactly, capped at the logical core count. If set to None, 0, or a negative value, the function uses the number of CPU cores minus 2, clamped to at least 1.
remove_sources (bool, default: True) – Determines whether to remove the .npy files after consolidating their data into .npz archives.
memory_mapping (bool, default: True) – Determines whether to memory-map or load the processed data into RAM during processing. Due to Windows not releasing memory-mapped file handles, this function always loads the data into RAM when running on Windows.
verbose (bool, default: False) – Determines whether to communicate the log assembly progress via the terminal.
verify_integrity (bool, default: False) – Determines whether to verify the integrity of the created archives against the original log entries before removing sources.

Return type:

None

ataraxis_data_structures.data_loggers.discover_log_archives(log_directory)¶

Discovers every log archive stored directly inside the target logger output directory.

Notes

Only the directory’s own entries are inspected, since one logger writes every archive it assembles side by side. Searching a tree that spans several loggers instead resolves each source with find_log_archive().

Parameters:: log_directory (Path) – The logger output directory whose archives are discovered.
Return type:: dict[str, Path]
Returns:: The path to every discovered archive, keyed by the identifier of the source that produced it.
Raises:: FileNotFoundError – If the log directory does not exist or is not a directory.

ataraxis_data_structures.data_loggers.find_log_archive(log_directory, source_id)¶

Searches for the single log archive holding the entries of the target source under the log directory.

Notes

The whole tree beneath the log directory is searched, so an archive nested at any depth is found. A source writes one archive per logger output directory, so a tree holding several matches spans several loggers and is ambiguous rather than merely redundant.

Parameters:

log_directory (Path) – The root directory whose tree is searched.
source_id (str) – The identifier of the source whose archive is resolved, matching the archive filename ahead of the log archive suffix.

Return type:

Path

Returns:

The path to the discovered log archive.

Raises:

FileNotFoundError – If the log directory does not exist, is not a directory, or holds no archive for the requested source.
OSError – If any directory beneath the log directory cannot be read.
ValueError – If the log directory holds more than one archive for the requested source.

ataraxis_data_structures.data_loggers.find_log_archives(log_directory, source_ids)¶

Searches for the log archive of every target source under the log directory, in a single pass.

Notes

One traversal resolves every requested source, so a caller resolving several sources pays the same walk as a caller resolving one.

A source that resolves to no archive, or to several, fails the whole call, so a caller receives either every requested archive or none of them.

Parameters:

log_directory (Path) – The root directory whose tree is searched.
source_ids (Iterable[str]) – The identifiers of the sources whose archives are resolved, each matching the archive filename ahead of the log archive suffix.

Return type:

dict[str, Path]

Returns:

The path to the discovered archive of each requested source, keyed by that source identifier.

Raises:

FileNotFoundError – If the log directory does not exist, is not a directory, or holds no archive for any requested source.
OSError – If the log directory or any directory beneath it cannot be read, or if the kind of an entry carrying an archive name cannot be determined.
ValueError – If the log directory holds more than one archive for any requested source.

ataraxis_data_structures.data_loggers.read_archive_message_count(archive_path)¶

Reads the number of data messages the target log archive holds without decoding any of them.

Notes

Reads the archive’s zip directory alone, so the cost tracks the number of messages rather than their size.

Parameters:: archive_path (Path) – The path to the .npz log archive to read.
Return type:: int
Returns:: The number of data messages the archive holds, excluding the onset message.
Raises:: FileNotFoundError – If the archive does not exist or is not a file.

ataraxis_data_structures.data_loggers.serialized_data_logger.LOG_DIRECTORY_SUFFIX: str = '_data_log'¶

The name suffix of the output directory each DataLogger instance creates for its log entries and archives.

Notes

Every directory is named {instance_name}{LOG_DIRECTORY_SUFFIX} after the logger instance that owns it, so a consumer resolves a logger’s output directory from the logger’s name alone.

ataraxis_data_structures.data_loggers.serialized_data_logger.LOG_ARCHIVE_SUFFIX: str = '_log.npz'¶

The filename suffix of the .npz log archives assemble_log_archives() writes.

Notes

Every archive is named {source_id}{LOG_ARCHIVE_SUFFIX} after the source whose entries it holds, so a consumer resolves an archive from a source ID alone.

ataraxis_data_structures.data_loggers.log_archive_reader.PARALLEL_PROCESSING_THRESHOLD: int = 2000¶

The number of data messages a log archive has to hold before its messages are worth processing in parallel.

Notes

An archive holding fewer messages costs more in worker startup and message transfer than the parallel decode saves, so LogArchiveReader.get_batches() stops dividing such an archive and hands back every message it holds as one batch.

Processing¶

Provides utilities for data integrity verification, directory transfer and deletion, data asset discovery, atomic and direct file writing, data interpolation, and worker thread limiting.

ataraxis_data_structures.processing.atomic_write(file_path, *, binary=False)¶

Opens a temporary file that replaces the target path in one step once the caller finishes writing to it.

Notes

Suits a destination that already exists and that another process may read. A file nothing has opened yet is written through direct_write() instead, which pays neither the flush nor the rename this function pays.

A reader of the target path observes either the previous file or the complete new one, never a partial write. Writing the destination directly instead truncates it first, so a writer killed mid-write leaves a truncated file where a complete one used to be.

The temporary file is created in the destination’s own directory, so the rename that publishes it stays within one filesystem and is therefore atomic. A destination whose parent directory does not exist yet has it created.

The written file carries 0o644 on a host using the default 0o022 umask, since the temporary file requests the same bits the built-in open() requests and the umask narrows them. The permissions a destination carried before the write are NOT preserved, so a caller needing anything other than the default applies it with chmod() after this context exits.

The contents reach the disk before the rename publishes them, so a host losing power immediately afterwards finds a complete file rather than a partial one. The rename itself is not synced, so the file that survives such a loss may be the previous one. A caller writing a large number of small files pays that flush per file.

A failure anywhere inside the context removes the temporary file and propagates, leaving the destination as it was.

Parameters:

file_path (Path) – The path to the file to write. The file is created when it does not exist and replaced when it does.
binary (bool, default: False) – Determines whether the yielded file object accepts bytes instead of text.

Yields:

The file object to write the contents to.

Raises:

OSError – If the temporary file cannot be created or written.
PermissionError – If the destination stays locked by another process for every publishing attempt.

ataraxis_data_structures.processing.calculate_directory_checksum(directory, num_processes=None, *, progress=False, save_checksum=True, excluded_files=None)¶

Calculates the xxHash3-128 checksum for the input directory.

Notes

The xxHash3 checksum is not suitable for security purposes and is only used to ensure data integrity.

The returned checksum accounts for the contents of each file and for each file’s path relative to the input directory. A directory contributes only through the relative paths of the files stored beneath it, so a directory with no file anywhere beneath it contributes nothing.

Parameters:

directory (Path) – The path to the directory for which to generate the checksum.
num_processes (int | None, default: None) – The number of processes to use for parallelizing checksum calculation. If set to None, the function uses all available CPU cores minus 2 reserved cores (via resolve_worker_count).
progress (bool, default: False) – Determines whether to track the checksum calculation progress using a progress bar.
save_checksum (bool, default: True) – Determines whether to write the checksum to the ax_checksum.txt file at the top level of the input directory.
excluded_files (set[str] | None, default: None) – The set of filenames to exclude from the checksum calculation. If set to None, defaults to {"ax_checksum.txt"}.

Return type:

str

Returns:

The xxHash3-128 checksum for the input directory as a hexadecimal string.

Raises:

ValueError – If the input directory holds no file for the checksum to cover.
OSError – If the directory does not exist, is not a directory, or cannot be read, if any directory beneath it cannot be read, or if the kind of an entry beneath it cannot be determined. Also raised if a discovered file cannot be opened, or if the checksum file cannot be written while save_checksum is enabled. The digest covers the whole tree or the call fails, since a digest computed over the readable subset would certify a subset as the whole.

ataraxis_data_structures.processing.delete_directory(directory_path)¶

Deletes the target directory and all its subdirectories, unlinking the files within each directory in parallel.

Notes

A symlink is removed as a link, whatever it points at, so the tree behind a symlinked subdirectory is left untouched and only entries living inside the target directory are deleted. The rule covers the target itself, so a link passed as directory_path is unlinked in place and the tree it names is left whole. Every entry that is not a real directory is unlinked in place, which additionally covers the sockets and FIFOs that a file check skips.

Removal of each emptied directory is attempted up to five times, with a 500 millisecond delay between attempts, as some operating systems are slow to release file handles. If every attempt fails, the function reports a warning and returns, leaving the directory in place. The path is verifiable with Path.exists() when the removal has to be guaranteed.

Parameters:: directory_path (Path) – The path to the directory to delete.
Raises:: OSError – If the directory, or any directory beneath it, cannot be read, if the metadata of an entry inside it cannot be read, or if an entry inside it cannot be unlinked.
Return type:: None

ataraxis_data_structures.processing.direct_write(file_path, *, binary=False)¶

Opens the target path for writing, replacing whatever it already holds.

Notes

Suits a file the caller creates rather than replaces. A destination another process may read while the write runs is written through atomic_write() instead.

Costs one open() and leaves the contents in the operating system’s buffers. A path writing a file per record therefore pays neither the rename that publishing through a temporary file costs, nor the flush to disk that a durable write costs.

A reader opening the path while this context is open observes a partially written file, and a writer killed partway leaves one behind. Both are acceptable for a file whose absence or truncation the caller detects anyway, and neither is acceptable for a file that already held a complete previous version.

The written file carries 0o644 on a host using the default 0o022 umask, matching atomic_write() and the built-in open(). A destination whose parent directory does not exist yet has it created.

Parameters:

file_path (Path) – The path to the file to write. The file is created when it does not exist and truncated when it does.
binary (bool, default: False) – Determines whether the yielded file object accepts bytes instead of text.

Yields:

The file object to write the contents to.

Raises:

OSError – If the file cannot be created or written.

ataraxis_data_structures.processing.discover_marker_files(directory, marker_name)¶

Discovers every marker file with the target name stored anywhere under the target directory.

Notes

The name is compared against the entry each directory scan returned, so the traversal keeps only the matches rather than materializing every file beneath the root. That bound matters over a tree holding a large number of data files alongside the few markers describing them.

Parameters:

directory (Path) – The root directory whose tree is searched.
marker_name (str) – The exact filename every discovered marker carries.

Return type:

list[Path]

Returns:

The paths to every matching file found anywhere under the root directory, sorted by path.

Raises:

OSError – If the root directory does not exist, is not a directory, or cannot be read, if any directory beneath it cannot be read, or if the kind of an entry carrying the marker name cannot be determined.

ataraxis_data_structures.processing.discover_marker_roots(directory, marker_name, levels_up=0)¶

Discovers the directories owning every marker file stored anywhere under the target directory.

Notes

A marker describes the asset that owns it, and that asset’s directory sits a fixed number of levels above the marker rather than at the marker itself. A marker written at {root}/raw_data/session_data.yaml resolves its root with one level, while a marker written directly into the directory it describes resolves with none.

Two markers resolving to the same directory contribute one entry, since the result names owning directories rather than the markers that pointed at them.

Parameters:

directory (Path) – The root directory whose tree is searched.
marker_name (str) – The exact filename every discovered marker carries.
levels_up (int, default: 0) – The number of directory levels between a marker’s own parent and the directory that owns it.

Return type:

list[Path]

Returns:

The paths to every directory owning a matching marker, sorted by path.

Raises:

OSError – If the root directory does not exist, is not a directory, or cannot be read, if any directory beneath it cannot be read, or if the kind of an entry carrying the marker name cannot be determined.
ValueError – If the requested level count is negative, or if a discovered marker sits too close to the filesystem root to have an ancestor at that level.

ataraxis_data_structures.processing.index_marker_files(directory, marker_names, *, max_depth=None)¶

Indexes every marker file carrying one of the target names, in a single pass over the target directory.

Notes

Every requested name is present in the result, mapping to an empty tuple when the tree holds no file carrying it.

The depth bound counts the entries a directory holds as one level, so a bound of one keeps the traversal to the target directory’s own entries. Leaving the bound unset searches the whole tree.

Parameters:

directory (Path) – The root directory whose tree is searched.
marker_names (Iterable[str]) – The exact filenames to index.
max_depth (int | None, default: None) – The number of directory levels to descend, or None to descend without a bound.

Return type:

dict[str, tuple[Path, ...]]

Returns:

The paths to every matching file found under the root directory within the requested depth bound, sorted by path, keyed by the marker name each one carries.

Raises:

OSError – If the root directory does not exist, is not a directory, or cannot be read, if any directory beneath it cannot be read, or if the kind of an entry carrying a marker name cannot be determined.
ValueError – If no marker name is requested, or if the requested depth bound is less than one.

ataraxis_data_structures.processing.initialize_worker_threads(thread_count=1, maximum_thread_count=None, additional_thread_variables=None)¶

Constrains the numeric backends of the calling process to the requested thread counts.

Notes

Runs inside a worker process, as the initializer a process pool calls in each child it spawns. This covers the backends that read their variable the first time they are asked to do work, which a pool created outside limit_worker_threads() reaches no other way. A worker of a pool created inside that context already inherits the pinned environment, so calling this as well changes nothing while both name the same counts.

A pool runs this initializer before its worker unpickles the first work item, so the backends that latch their width as they load are still unloaded at this point and the width written here is the width they take. They therefore take maximum_thread_count, and a caller pairing this with limit_worker_threads() hands both of them the same arguments.

numba latches its ceiling from the environment while it is imported, which a worker does before its pool’s initializer runs, so the environment no longer reaches it. It is pinned through its own runtime setter here, narrowed to the latched ceiling, since the setter rejects a count above it.

Parameters:

thread_count (int, default: 1) – The number of threads each resizable numeric backend of the calling process may open.
maximum_thread_count (int | None, default: None) – The number of threads each import-latched numeric backend of the calling process may open, which is the widest count any job the worker runs raises itself to. Defaults to the value of thread_count.
additional_thread_variables (Mapping[str, int] | None, default: None) – The threading-layer environment variables to write alongside the ones this module already knows, each mapped to the width it takes. Naming a variable this module already writes replaces the width that variable would otherwise take. The caller pairs each entry with thread_count or with maximum_thread_count, according to whether that backend reads its variable while loading or afterward.

Raises:

ValueError – If the requested thread count is less than one. If the requested maximum thread count is less than the requested thread count. If an additional variable carries a blank name or a width less than one.

Return type:

None

ataraxis_data_structures.processing.interpolate_data(source_coordinates, source_values, target_coordinates, *, is_discrete)¶

Interpolates the data values at the requested coordinates using the source coordinate-value distribution.

Notes

Expects the source_coordinates and target_coordinates arrays to be one-dimensional and monotonically increasing.

Discrete interpolated data is returned as an array with the same datatype as the input data. Continuous interpolated data is returned as a float64 datatype array.

Continuous data is interpolated using the linear interpolation method. Discrete data is interpolated to the last known value at or to the left of each target coordinate. Target coordinates below the source range are clamped to the first source value, and those above the source range are clamped to the last source value.

Parameters:

source_coordinates (NDArray[number[Any]]) – The source coordinate values.
source_values (NDArray[number[Any]]) – The data values at each source coordinate.
target_coordinates (NDArray[number[Any]]) – The target coordinates for which to interpolate the data values.
is_discrete (bool) – Determines whether the interpolated data is discrete or continuous.

Return type:

NDArray[number[Any]]

Returns:

The interpolated data value at each target coordinate, in the order the target coordinates were supplied.

Raises:

ValueError – If the source coordinate array or the source value array holds no element.

ataraxis_data_structures.processing.limit_worker_threads(thread_count=1, maximum_thread_count=None, additional_thread_variables=None)¶

Constrains the numeric backends imported by worker processes to the requested thread counts.

Notes

The numeric backends bundled with NumPy open a thread pool sized to the host’s core count when they are imported, whatever work the importing process intends to do. A process pool that hands each worker its own backend therefore opens that pool once per worker, so a job running one worker per core holds the square of the core count in threads while using one of them.

The limit travels to the workers through the environment a spawned child inherits, so this context has to enclose the pool’s whole lifetime rather than its construction alone. A pool creates each worker when work is first submitted to it, not when the pool itself is created.

The backends that latch their width as they load take maximum_thread_count, because a job that raises its own width reaches them no other way. A pool whose jobs each run at a single thread leaves that argument unset, which pins every backend at thread_count.

Restoring the previous values on exit keeps the limit from leaking into whatever the calling process does next.

A worker of a pool created outside this context pins itself through initialize_worker_threads(), which still reaches the backends that read their variable after the worker has started.

Parameters:

thread_count (int, default: 1) – The number of threads each worker’s resizable numeric backends may open.
maximum_thread_count (int | None, default: None) – The number of threads each worker’s import-latched numeric backends may open, which is the widest count any job the pool runs raises itself to. Defaults to the value of thread_count.
additional_thread_variables (Mapping[str, int] | None, default: None) – The threading-layer environment variables to write alongside the ones this module already knows, each mapped to the width it takes. Naming a variable this module already writes replaces the width that variable would otherwise take. The caller pairs each entry with thread_count or with maximum_thread_count, according to whether that backend reads its variable while loading or afterward.

Raises:

Return type:

Generator[None, None, None]

ataraxis_data_structures.processing.resolve_unique_roots(paths)¶

Resolves the target paths to the deepest ancestor of each whose name no other path carries.

Notes

Paths sharing a structural layout differ only in the components naming the asset each one belongs to, such as a recording or a session identifier. Truncating each path at its deepest distinguishing component therefore strips the structure shared below that component without assuming a fixed depth for it.

A lone path has no sibling to differ from, so its own final component distinguishes it and it resolves to itself.

Parameters:: paths (list[Path] | tuple[Path, ...]) – The paths to resolve. Every path must carry at least one component that no other path carries.
Return type:: tuple[Path, ...]
Returns:: The resolved ancestors, one per distinct root, in the order the first path resolving to each one appears.
Raises:: ValueError – If any path shares every one of its components with the other paths.

ataraxis_data_structures.processing.transfer_directory(source, destination, num_threads=1, *, verify_integrity=False, remove_source=False, progress=False, reset_dirty_destination=False)¶

Copies the contents of the input source directory to the destination directory while preserving the underlying directory hierarchy.

Notes

Recreates the source hierarchy on the destination before copying any file, and copies with a thread pool when num_threads is greater than 1.

When integrity verification is enabled, reuses the xxHash3-128 checksum stored in the source directory’s ax_checksum.txt file when that file exists. Otherwise, generates the checksum before the transfer and writes it to the source directory as the ax_checksum.txt file. After the transfer, recomputes the checksum for the destination directory and compares it against the source checksum to detect data corruption. A reused checksum that predates the source’s current contents fails that comparison even though every transferred byte is correct, so the failure report names it alongside corruption as a possible cause.

A symlink is rejected, whether it stands for the source root itself or sits anywhere inside the source tree. A link is meaningful only relative to the filesystem that holds it, so moving one is either a silent omission or a dangling entry at the destination. A link standing for the root is refused for a second reason. Removing the source afterwards would remove the link rather than the tree it names, which leaves the caller told the source is gone while every byte of it is still on disk. Every link has to resolve into real data before the tree is transferred.

A destination holding files the source does not account for is also rejected, since the integrity check covers the whole destination and those files would fail it while every transferred byte is correct. The checksum file never counts as unaccounted, because the transfer overwrites it.

Parameters:

source (Path) – The path to the directory to be transferred.
destination (Path) – The path to the destination directory where to move the contents of the source directory.
num_threads (int, default: 1) – The number of threads to use for the parallel file transfer. Setting this value below 1 instructs the function to use all available CPU cores minus a small number reserved for the host system.
verify_integrity (bool, default: False) – Determines whether to perform integrity verification for the transferred files.
remove_source (bool, default: False) – Determines whether to remove the source directory after the transfer is complete and (optionally) verified.
progress (bool, default: False) – Determines whether to track the transfer progress using a progress bar.
reset_dirty_destination (bool, default: False) – Determines whether to delete the destination files the source does not account for, rather than rejecting the transfer when any are found.

Raises:

FileNotFoundError – If the source directory does not exist.
OSError – If any directory inside the source or the destination tree cannot be read, or if the destination path already exists as a file rather than as a directory. Also raised if a copied file cannot be read or written, or if the source cannot be removed once remove_source is enabled. A source discovery failure leaves the destination untouched, since the source tree is discovered before anything is written.
RuntimeError – If the source path is itself a symlink or the source directory contains one, or if verify_integrity is enabled while the source holds no file the checksum can cover. Also raised if the destination holds unaccounted files while reset_dirty_destination is disabled, or if the transferred files do not pass the xxHash3-128 checksum integrity verification.

Return type:

None