hpyx

HPyX: Pythonic bindings for the HPX C++ parallel runtime.

Modules:

• aio –

  hpyx.aio — asyncio integration for hpyx Futures.

• config –

  Configuration for the HPyX runtime.

• debug –

  Diagnostics and tracing hooks.

• execution –

  hpyx.execution — HPX-style execution policies for parallel algorithms.

• executor –

  hpyx.HPXExecutor — a concurrent.futures.Executor backed by HPX.

• futures –

  hpyx.futures — HPX-backed futures API (Pythonic wrapper).

• kernels –

  hpyx.kernels -- Pure C++ parallel kernels on ndarray data.

• multiprocessing –

  hpyx.multiprocessing — DEPRECATED.

• parallel –

  hpyx.parallel — Python-callback parallel algorithms over integer ranges

• runtime –

  HPXRuntime context manager — optional convenience wrapper.

• util –

  HPyX utilities subpackage.

Classes:

• Future –

  A future backed by the HPX runtime.

• HPXExecutor –

  A real concurrent.futures.Executor backed by HPX.

• HPXRuntime –

  Context manager that ensures the HPX runtime is running.

Functions:

• async_ –

  Submit a callable to an HPX worker; return a Future for its result.

• dataflow –

  Run fn(*resolved_values, **kwargs) once all input futures complete.

• init –

  Explicitly start the HPX runtime. Idempotent within a process.

• ready_future –

  Return an already-completed Future wrapping value.

• shared_future –

  Return a shareable view of f (multiple consumers can call .result()).

• shutdown –

  Explicit shutdown. Irreversible within the process.

• when_all –

  Return a Future that resolves to a tuple of all input results (in order).

• when_any –

  Return a Future that resolves to (index, futures_list) when any input completes.

Future

Future(hpx_fut: '_core.futures.HPXFuture')

Bases: Future

A future backed by the HPX runtime.

Implements the concurrent.futures.Future protocol (result, exception, done, running, cancelled, cancel, add_done_callback), plus HPX-native .then(fn) and asyncio await support.

Multiple consumers of the same Future are supported — internally every Future wraps an hpx::shared_future.

HPXExecutor

HPXExecutor(max_workers: int | None = None)

Bases: Executor

A real concurrent.futures.Executor backed by HPX.

Parameters:

• max_workers

  (int | None, default: None ) –

  Advisory. HPX's worker pool is process-global; max_workers seeds os_threads on the implicit init if the runtime isn't started yet. If the runtime is already started with a different os_threads, a UserWarning is emitted and the existing pool is used unchanged.

Notes

Multiple HPXExecutor instances share the process-global HPX runtime. Calling .shutdown() on one handle does not affect other handles or stop the runtime — atexit owns process-level teardown because HPX cannot restart in-process.

HPXRuntime

HPXRuntime(
    *,
    os_threads: int | None = None,
    cfg: list[str] | None = None,
)

Context manager that ensures the HPX runtime is running.

Parameters:

• os_threads

  (int | None, default: None ) –

  Number of HPX worker OS threads. Defaults to HPYX_OS_THREADS env var or os.cpu_count().

• cfg

  (list[str] | None, default: None ) –

  Extra HPX config strings (e.g., ["hpx.stacks.small_size=0x20000"]).

async_

async_(fn: Callable, *args: Any, **kwargs: Any) -> Future

Submit a callable to an HPX worker; return a Future for its result.

dataflow

dataflow(
    fn: Callable, *futures: Future, **kwargs: Any
) -> Future

Run fn(*resolved_values, **kwargs) once all input futures complete.

init

init(
    *,
    os_threads: int | None = None,
    cfg: list[str] | None = None,
) -> None

Explicitly start the HPX runtime. Idempotent within a process.

ready_future

ready_future(value: Any) -> Future

Return an already-completed Future wrapping value.

shared_future

shared_future(f: Future) -> Future

Return a shareable view of f (multiple consumers can call .result()).

shutdown

shutdown() -> None

Explicit shutdown. Irreversible within the process.

when_all

when_all(*futures: Future) -> Future

Return a Future that resolves to a tuple of all input results (in order).

when_any

when_any(*futures: Future) -> Future

Return a Future that resolves to (index, futures_list) when any input completes.

aio

hpyx.aio — asyncio integration for hpyx Futures.

Provides the internal _future_await used by Future.__await__, plus the await_all and await_any combinators.

Functions:

• await_all –

  Await all input futures; return a tuple of their results in order.

• await_any –

  Await any input future; return (index, futures_list).

await_all async

await_all(*futures: Future) -> tuple

Await all input futures; return a tuple of their results in order.

Unlike :func:asyncio.gather, exceptions are NOT consumed — the first exception raised aborts the operation (matches when_all).

await_any async

await_any(*futures: Future) -> tuple[int, list[Future]]

Await any input future; return (index, futures_list).

The element at futures_list[index] is the one that completed; others may still be pending. futures_list is a Python list of :class:hpyx.Future wrappers.

config

Configuration for the HPyX runtime.

Precedence: explicit hpyx.init() kwargs > environment variables > DEFAULTS.

Functions:

• from_env –

  Build a config dict from HPYX_* environment variables.

from_env

from_env() -> dict[str, Any]

Build a config dict from HPYX_* environment variables.

Returns a fresh copy of DEFAULTS with any present env vars layered on top. Unset env vars leave the default value unchanged.

debug

Diagnostics and tracing hooks.

Functions:

• disable_tracing –

  Stop capturing. Flushes buffered events to the output file.

• enable_tracing –

  Start capturing per-task trace events to a JSONL file.

• get_num_worker_threads –

  Return the number of HPX worker OS threads in the default pool.

• get_worker_thread_id –

  Return the caller's HPX worker thread id, or -1 if not on an HPX thread.

disable_tracing

disable_tracing() -> None

Stop capturing. Flushes buffered events to the output file.

enable_tracing

enable_tracing(path: Optional[str] = None) -> None

Start capturing per-task trace events to a JSONL file.

Parameters:

• path

  (str | None, default: None ) –

  File path to write events to. Falls back to HPYX_TRACE_PATH env var if not provided.

get_num_worker_threads

get_num_worker_threads() -> int

Return the number of HPX worker OS threads in the default pool.

get_worker_thread_id

get_worker_thread_id() -> int

Return the caller's HPX worker thread id, or -1 if not on an HPX thread.

execution

hpyx.execution — HPX-style execution policies for parallel algorithms.

Usage

import hpyx
from hpyx.execution import par, seq, task, static_chunk_size

hpyx.parallel.for_loop(par, 0, 1_000_000, fn)
hpyx.parallel.for_loop(par.with_(static_chunk_size(10_000)), ...)

Note: task-tagged policies (e.g. par(task)) are supported by hpyx.parallel.sort, hpyx.parallel.stable_sort, and hpyx.parallel.transform_reduce, which return a Future. Other hpyx.parallel functions raise NotImplementedError for task-tagged policies.

Classes:

• ChunkSize –

  Opaque holder for a chunk-size strategy.

Functions:

• auto_chunk_size –

  Let HPX pick chunk size automatically.

• dynamic_chunk_size –

  Dynamic (load-balanced) chunks of n elements.

• guided_chunk_size –

  Guided (shrinking) chunks.

• static_chunk_size –

  Fixed n elements per task.

ChunkSize dataclass

ChunkSize(kind: int, size: int = 0)

Opaque holder for a chunk-size strategy.

auto_chunk_size

auto_chunk_size() -> ChunkSize

Let HPX pick chunk size automatically.

dynamic_chunk_size

dynamic_chunk_size(n: int) -> ChunkSize

Dynamic (load-balanced) chunks of n elements.

guided_chunk_size

guided_chunk_size() -> ChunkSize

Guided (shrinking) chunks.

static_chunk_size

static_chunk_size(n: int) -> ChunkSize

Fixed n elements per task.

executor

hpyx.HPXExecutor — a concurrent.futures.Executor backed by HPX.

All instances share one process-wide HPX runtime (HPX cannot host multiple runtimes in a single process). shutdown() marks this executor handle unusable but does not stop the runtime — atexit owns process-level teardown because HPX cannot restart in-process.

Examples:

>>> import hpyx
>>> with hpyx.HPXExecutor() as ex:
...     fut = ex.submit(pow, 2, 10)
...     print(fut.result())  # 1024

For dask integration:

>>> import dask.array as da
>>> with hpyx.HPXExecutor() as ex:
...     result = da.arange(1e6).sum().compute(scheduler=ex)

Classes:

• HPXExecutor –

  A real concurrent.futures.Executor backed by HPX.

HPXExecutor

HPXExecutor(max_workers: int | None = None)

Bases: Executor

A real concurrent.futures.Executor backed by HPX.

Parameters:

• max_workers

  (int | None, default: None ) –

  Advisory. HPX's worker pool is process-global; max_workers seeds os_threads on the implicit init if the runtime isn't started yet. If the runtime is already started with a different os_threads, a UserWarning is emitted and the existing pool is used unchanged.

Notes

Multiple HPXExecutor instances share the process-global HPX runtime. Calling .shutdown() on one handle does not affect other handles or stop the runtime — atexit owns process-level teardown because HPX cannot restart in-process.

futures

hpyx.futures — HPX-backed futures API (Pythonic wrapper).

Public names: Future, async_, when_all, when_any, dataflow, shared_future, ready_future

Classes:

• Future –

  A future backed by the HPX runtime.

Functions:

• async_ –

  Submit a callable to an HPX worker; return a Future for its result.

• dataflow –

  Run fn(*resolved_values, **kwargs) once all input futures complete.

• ready_future –

  Return an already-completed Future wrapping value.

• shared_future –

  Return a shareable view of f (multiple consumers can call .result()).

• when_all –

  Return a Future that resolves to a tuple of all input results (in order).

• when_any –

  Return a Future that resolves to (index, futures_list) when any input completes.

Future

Future(hpx_fut: '_core.futures.HPXFuture')

Bases: Future

A future backed by the HPX runtime.

Implements the concurrent.futures.Future protocol (result, exception, done, running, cancelled, cancel, add_done_callback), plus HPX-native .then(fn) and asyncio await support.

Multiple consumers of the same Future are supported — internally every Future wraps an hpx::shared_future.

async_

async_(fn: Callable, *args: Any, **kwargs: Any) -> Future

Submit a callable to an HPX worker; return a Future for its result.

dataflow

dataflow(
    fn: Callable, *futures: Future, **kwargs: Any
) -> Future

Run fn(*resolved_values, **kwargs) once all input futures complete.

ready_future

ready_future(value: Any) -> Future

Return an already-completed Future wrapping value.

shared_future

shared_future(f: Future) -> Future

Return a shareable view of f (multiple consumers can call .result()).

when_all

when_all(*futures: Future) -> Future

Return a Future that resolves to a tuple of all input results (in order).

when_any

when_any(*futures: Future) -> Future

Return a Future that resolves to (index, futures_list) when any input completes.

kernels

hpyx.kernels -- Pure C++ parallel kernels on ndarray data.

Note: sum, max, and min shadow Python builtins. Import them explicitly (e.g. from hpyx.kernels import sum as hsum) or access via the module (hpyx.kernels.sum) to avoid masking the built-in names in calling code.

Functions:

• dot –

  Parallel dot product of two 1-D arrays. Always returns float64.

• matmul –

  Matrix multiplication. Both inputs must be 2-D.

• max –

  Parallel maximum of array elements. Raises on empty array.

• min –

  Parallel minimum of array elements. Raises on empty array.

• sum –

  Parallel sum of array elements.

dot

dot(a: ndarray, b: ndarray) -> float

Parallel dot product of two 1-D arrays. Always returns float64.

matmul

matmul(A: ndarray, B: ndarray) -> ndarray

Matrix multiplication. Both inputs must be 2-D.

Currently delegates to NumPy's @ operator rather than an HPX C++ kernel. A native HPX implementation is a candidate for a future release.

max

max(a: ndarray)

Parallel maximum of array elements. Raises on empty array.

min

min(a: ndarray)

Parallel minimum of array elements. Raises on empty array.

sum

sum(a: ndarray)

Parallel sum of array elements.

multiprocessing

hpyx.multiprocessing — DEPRECATED.

This subpackage is deprecated and will be removed in a future release. Use hpyx.parallel instead, which provides a richer set of parallel algorithms (for_each, for_loop, transform, reduce, etc.) with explicit execution policies from hpyx.execution.

Migration: Old: hpyx.multiprocessing.for_loop(fn, iterable, policy="par") New: hpyx.parallel.for_each(hpyx.execution.par, iterable, fn)

Functions:

• for_loop –

  Execute a function over an iterable using HPX's parallel for_loop.

for_loop

for_loop(
    function: Callable,
    iterable: Iterable,
    policy: Literal["seq", "par"] = "seq",
) -> None

Execute a function over an iterable using HPX's parallel for_loop.

This function applies a given function to each element in an iterable using HPX's optimized parallel for_loop algorithm. The execution can be configured to run sequentially or in parallel.

Parameters:

• function

  (callable) –

  The callable to apply to each element in the iterable. The function should accept a single argument (the iterable element).

• iterable

  (iterable) –

  The iterable to process. Elements will be passed to the function one by one.

• policy

  (('seq', 'par'), default: 'seq' ) –

  Execution policy for the loop. - 'seq' : Sequential execution - 'par' : Parallel execution using available cores

Notes

This method will modify the iterable in place if the provided function modifies its arguments. The original iterable may be changed after this function completes.

This function requires an active HPX runtime. Ensure that you call this function within an HPXRuntime context manager.

Examples:

>>> from hpyx import HPXRuntime
>>> data = [1, 2, 3, 4, 5]
>>> def square_inplace(x):
...     x[0] = x[0] ** 2  # Modify in place
>>> with HPXRuntime() as runtime:
...     for_loop(square_inplace, enumerate(data), policy="seq")
...     print(data)  # data is now modified

parallel

hpyx.parallel — Python-callback parallel algorithms over integer ranges and iterables.

Every function takes a policy (from hpyx.execution) as the first argument. Task-tagged policies (e.g. par(task)) are supported by :func:sort, :func:stable_sort, and :func:transform_reduce, which return a :class:~hpyx.futures.Future; other algorithms raise NotImplementedError for task-tagged policies.

For par and par_unseq policies with Python callbacks, each iteration is submitted as an independent hpyx.async_ task on an HPX worker thread. The seq and unseq policies call the C++ layer directly for zero overhead. Pure C++ kernels (no Python callback) can use HPX parallel policies natively — see hpyx.kernels.

Functions:

• all_of –

  Return True if pred is true for all elements.

• any_of –

  Return True if pred is true for any element.

• copy –

  Return a new list copy of iterable.

• copy_if –

  Return a list of elements satisfying pred.

• count –

  Count elements equal to value.

• count_if –

  Count elements satisfying pred.

• exclusive_scan –

  Return a list of running totals starting from init (exclusive).

• fill –

  Return a list of n copies of value.

• fill_n –

  Alias for :func:fill.

• find –

  Return index of first element equal to value, or -1.

• find_if –

  Return index of first element satisfying pred, or -1.

• for_each –

  Apply fn(x) to every element in iterable under policy.

• for_loop –

  Invoke body(i) for i in [first, last) under policy.

• inclusive_scan –

  Return a list of running totals (inclusive).

• iota –

  Return [start, start+1, ..., start+n-1].

• none_of –

  Return True if pred is false for all elements.

• reduce –

  Reduce iterable with op, starting from init.

• sort –

  Return a new sorted list, dispatching to hpx::sort.

• stable_sort –

  Return a new sorted list preserving relative order of equal elements.

• transform –

  Apply fn to each element, return a new list of results.

• transform_reduce –

  Transform each element with transform_op then reduce with reduce_op.

all_of

all_of(
    policy: Policy,
    iterable: Iterable[all_of[T]],
    pred: Callable[[all_of[T]], bool],
) -> bool

Return True if pred is true for all elements.

Note: under par/par_unseq, all predicates are launched before any result is checked — no short-circuit occurs.

any_of

any_of(
    policy: Policy,
    iterable: Iterable[any_of[T]],
    pred: Callable[[any_of[T]], bool],
) -> bool

Return True if pred is true for any element.

Note: under par/par_unseq, all predicates are launched before any result is checked — no short-circuit occurs.

copy

copy(
    policy: Policy, iterable: Iterable[copy[T]]
) -> list[copy[T]]

Return a new list copy of iterable.

copy_if

copy_if(
    policy: Policy,
    iterable: Iterable[copy_if[T]],
    pred: Callable[[copy_if[T]], bool],
) -> list[copy_if[T]]

Return a list of elements satisfying pred.

count

count(
    policy: Policy,
    iterable: Iterable[count[T]],
    value: count[T],
) -> int

Count elements equal to value.

count_if

count_if(
    policy: Policy,
    iterable: Iterable[count_if[T]],
    pred: Callable[[count_if[T]], bool],
) -> int

Count elements satisfying pred.

exclusive_scan

exclusive_scan(
    policy: Policy,
    iterable: Iterable[exclusive_scan[T]],
    *,
    init: exclusive_scan[T],
    op: Callable[
        [exclusive_scan[T], exclusive_scan[T]],
        exclusive_scan[T],
    ],
) -> list[exclusive_scan[T]]

Return a list of running totals starting from init (exclusive).

fill

fill(
    policy: Policy, n: int, value: fill[T]
) -> list[fill[T]]

Return a list of n copies of value.

fill_n

fill_n(
    policy: Policy, n: int, value: fill_n[T]
) -> list[fill_n[T]]

Alias for :func:fill.

find

find(
    policy: Policy,
    iterable: Iterable[find[T]],
    value: find[T],
) -> int

Return index of first element equal to value, or -1.

Note: under par/par_unseq, all elements are evaluated before results are checked — no short-circuit occurs.

find_if

find_if(
    policy: Policy,
    iterable: Iterable[find_if[T]],
    pred: Callable[[find_if[T]], bool],
) -> int

Return index of first element satisfying pred, or -1.

Note: under par/par_unseq, all elements are evaluated before results are checked — no short-circuit occurs.

for_each

for_each(
    policy: Policy, iterable: Any, fn: Callable[[Any], None]
) -> Union[None, Future]

Apply fn(x) to every element in iterable under policy.

for_loop

for_loop(
    policy: Policy,
    first: int,
    last: int,
    body: Callable[[int], None],
) -> Union[None, Future]

Invoke body(i) for i in [first, last) under policy.

inclusive_scan

inclusive_scan(
    policy: Policy,
    iterable: Iterable[inclusive_scan[T]],
    *,
    op: Callable[
        [inclusive_scan[T], inclusive_scan[T]],
        inclusive_scan[T],
    ],
) -> list[inclusive_scan[T]]

Return a list of running totals (inclusive).

iota

iota(policy: Policy, n: int, start: int = 0) -> list[int]

Return [start, start+1, ..., start+n-1].

none_of

none_of(
    policy: Policy,
    iterable: Iterable[none_of[T]],
    pred: Callable[[none_of[T]], bool],
) -> bool

Return True if pred is false for all elements.

Note: under par/par_unseq, all predicates are launched before any result is checked — no short-circuit occurs.

reduce

reduce(
    policy: Policy,
    iterable: Iterable[reduce[T]],
    *,
    init: reduce[T],
    op: Callable[[reduce[T], reduce[T]], reduce[T]],
) -> reduce[T]

Reduce iterable with op, starting from init.

sort

sort(
    policy: Policy,
    data: Iterable[sort[T]],
    *,
    key: Callable[[sort[T]], Any] | None = None,
    reverse: bool = False,
) -> Union[list[sort[T]], Future]

Return a new sorted list, dispatching to hpx::sort.

For par / par_unseq policies, hpx::sort with a parallel execution policy is used. With Python-object comparisons the GIL still serializes individual comparisons, so throughput gains over seq are visible only when the collection contains types whose C++ < operator can run GIL-free (see hpyx.kernels for pure-C++ numeric kernels).

With a task-tagged policy the sort is submitted as a single HPX task and a :class:~hpyx.futures.Future is returned.

stable_sort

stable_sort(
    policy: Policy,
    data: Iterable[stable_sort[T]],
    *,
    key: Callable[[stable_sort[T]], Any] | None = None,
    reverse: bool = False,
) -> Union[list[stable_sort[T]], Future]

Return a new sorted list preserving relative order of equal elements.

Dispatches to hpx::stable_sort. The same GIL note as :func:sort applies.

With a task-tagged policy a :class:~hpyx.futures.Future is returned.

transform

transform(
    policy: Policy,
    iterable: Iterable[transform[T]],
    fn: Callable[[transform[T]], transform[U]],
) -> list[transform[U]]

Apply fn to each element, return a new list of results.

transform_reduce

transform_reduce(
    policy: Policy,
    iterable: Iterable[transform_reduce[T]],
    *,
    init: transform_reduce[U],
    reduce_op: Callable[
        [transform_reduce[U], transform_reduce[U]],
        transform_reduce[U],
    ],
    transform_op: Callable[
        [transform_reduce[T]], transform_reduce[U]
    ],
) -> Union[transform_reduce[U], Future]

Transform each element with transform_op then reduce with reduce_op.

With a task-tagged policy the entire computation is submitted as a single HPX task and a :class:~hpyx.futures.Future is returned.

runtime

HPXRuntime context manager — optional convenience wrapper.

Using this is no longer required in v1 — HPyX auto-initializes on first use. This context manager remains for users who want explicit lifecycle scoping in scripts and tests, and for backward compatibility with v0.x code.

Exit does NOT shut down the runtime (HPX can't restart within a process). Shutdown is owned by atexit; call hpyx.shutdown() explicitly if you need to force an early stop.

Classes:

• HPXRuntime –

  Context manager that ensures the HPX runtime is running.

HPXRuntime

HPXRuntime(
    *,
    os_threads: int | None = None,
    cfg: list[str] | None = None,
)

Context manager that ensures the HPX runtime is running.

Parameters:

• os_threads

  (int | None, default: None ) –

  Number of HPX worker OS threads. Defaults to HPYX_OS_THREADS env var or os.cpu_count().

• cfg

  (list[str] | None, default: None ) –

  Extra HPX config strings (e.g., ["hpx.stacks.small_size=0x20000"]).

util

HPyX utilities subpackage.

This subpackage contains utility functions and modules that provide supporting functionality for HPyX, including version information, diagnostic tools, and helper functions.

Modules:

• print_versions –

  Utility module for printing version and system information.

print_versions

Utility module for printing version and system information.

This module provides functions to display comprehensive version information for HPyX and its dependencies, as well as system and environment details. This is useful for debugging, issue reporting, and system diagnostics.

The module can be executed directly from the command line:

python -m hpyx.util.print_versions

Originally adapted from xarray's print_versions utility. See: https://github.com/pydata/xarray/blob/main/xarray/util/print_versions.py

Functions:

• get_sys_info –

  Get system information and HPyX version.

• show_versions –

  Print the versions of HPyX and its dependencies.

get_sys_info

get_sys_info() -> list[
    tuple[str, str | Literal["unknown"]]
]

Get system information and HPyX version.

Collects comprehensive system information including HPyX version, Git commit hash (if available), Python version, operating system details, and environment variables.

Returns:

• list of tuple –

  A list of tuples containing system information key-value pairs. Each tuple contains (info_name, info_value) where info_value may be "unknown" if the information cannot be determined.

Notes

Git commit information is only available when running from a Git repository with the expected directory structure.

show_versions

show_versions(file=stdout)

Print the versions of HPyX and its dependencies.

Displays comprehensive version information for HPyX, its dependencies, system information, and HPX C++ library details. This information is useful for debugging, issue reporting, and system diagnostics.

Parameters:

• file

  (file-like object, default: sys.stdout ) –

  The file-like object to print to. Can be any object that supports write() method.

Examples:

>>> show_versions()  # Print to console
>>> with open('versions.txt', 'w') as f:
...     show_versions(file=f)  # Save to file