`snoozeml.client` — Synchronous & Asynchronous Clients#

High-level synchronous and asynchronous clients for the SnoozeML inference daemon.

class snoozeml.client.SnoozeClient(endpoint=None, *, http_endpoint=None, grpc_endpoint=None, use_grpc=None, api_key=None, token=None, timeout=30.0, tls_ca_cert=None, verify_tls=True, max_message_length=67108864, max_retries=0, retry_backoff=0.5, check_version=True)[source]#

Bases: object

Synchronous client for SnoozeML model management and tensor inference.

Supports both HTTP REST (JSON) and high-throughput native gRPC (raw FP32 byte buffers) with unified API Key / JWT Bearer authentication and optional TLS.

Example:

import numpy as np
from snoozeml import SnoozeClient

with SnoozeClient(endpoint="localhost:50051", use_grpc=True) as client:
    models = client.models.list()
    input_data = np.random.randn(1, 10).astype(np.float32)
    resp = client.infer("demo-model", inputs={"input": input_data})
    output = resp.get_numpy("output")

Parameters:

endpoint (Optional[str])
http_endpoint (Optional[str])
grpc_endpoint (Optional[str])
use_grpc (Optional[bool])
api_key (Optional[str])
token (Optional[str])
timeout (float)
tls_ca_cert (Optional[Union[str, bytes, Path]])
verify_tls (bool)
max_message_length (int)
max_retries (int)
retry_backoff (float)
check_version (bool)

close()[source]#

Closes underlying HTTP connection pools and gRPC channels.

Return type:: None

property server_version: str | None#: Returns the detected SnoozeML server version after the first connection, or None.

infer(model_id, inputs, *, use_grpc=None, timeout=None, client_timeout_ms=None, parameters=None, retries=None)[source]#

Executes unary tensor inference against a registered model.

Parameters:

model_id (str) – Target model identifier (e.g. "demo-model", "bert-base-uncased").
inputs (Mapping[str, Any | Sequence[Any] | TensorInput] | Sequence[TensorInput | Mapping[str, Any]] | TensorInput | Any) – Input tensors provided as a dictionary {name: np.ndarray}, a list of TensorInput or dicts, or a single np.ndarray / torch.Tensor.
use_grpc (bool | None) – Optional per-request transport override. If None, uses use_grpc.
timeout (float | None) – Optional request timeout in seconds (defaults to timeout).
client_timeout_ms (int | None) – Optional gRPC deadline hint in milliseconds embedded in the request.
parameters (Mapping[str, str] | None) – Optional gRPC execution parameters / routing hints.
retries (int | None) – Optional retry count for transient 503 (ModelLoadingError / ResourceExhaustedError). Defaults to max_retries.

Returns:

InferenceResponse containing output tensors and server latency.

Raises:

InvalidInputError – If model_id or inputs fail validation.
ModelNotFoundError – If model_id is not registered.
SnoozeError – On authentication, timeout, resource exhaustion, or runtime failure.

Return type:

InferenceResponse

infer_dataset(model_id, inputs, *, batch_size=32, use_grpc=None, timeout=None, client_timeout_ms=None, parameters=None, retries=None)[source]#

Slices a large dataset along Axis(0) into chunks of batch_size, executes inference, and concatenates outputs.

Parameters:

model_id (str) – Target model identifier.
inputs (Mapping[str, Any | Sequence[Any] | TensorInput] | Sequence[TensorInput | Mapping[str, Any]] | TensorInput | Any) – Dataset tensor inputs sharing the same leading Axis(0) length.
batch_size (int) – Maximum number of rows per micro-batch request (default 32).
use_grpc (bool | None) – Optional per-request transport override.
timeout (float | None) – Optional per-chunk timeout in seconds.
client_timeout_ms (int | None) – Optional gRPC deadline hint in milliseconds.
parameters (Mapping[str, str] | None) – Optional gRPC execution parameters.
retries (int | None) – Optional per-chunk retry count on transient 503 responses.

Returns:

Combined InferenceResponse with outputs concatenated along Axis(0).

Return type:

InferenceResponse

stream_infer(model_id, inputs, *, timeout=None, client_timeout_ms=None, parameters=None)[source]#

Executes bi-directional streaming inference over persistent gRPC connection.

Feeds an incoming stream of tensor inputs through SnoozeML’s dynamic batcher and yields responses as they complete.

Parameters:

model_id (str) – Target model identifier (e.g. "demo-model").
inputs (Iterable[Mapping[str, Any | Sequence[Any] | TensorInput] | Sequence[TensorInput | Mapping[str, Any]] | TensorInput | Any] | Iterator[Mapping[str, Any | Sequence[Any] | TensorInput] | Sequence[TensorInput | Mapping[str, Any]] | TensorInput | Any]) – An iterable or generator yielding inputs (dictionaries of tensors, lists of TensorInput, etc.).
timeout (float | None) – Optional overall gRPC stream timeout in seconds.
client_timeout_ms (int | None) – Optional per-request deadline hint in milliseconds.
parameters (Mapping[str, str] | None) – Optional execution parameters.

Yields:

InferenceResponse for each completed frame from the stream.

Raises:

InvalidInputError – If model_id is empty.
SnoozeError – If gRPC is not available or on transport/authentication failure.

Return type:

Iterator[InferenceResponse]

list_models()[source]#

Lists all registered models (shorthand for client.models.list()).

Return type:: List[ModelInfo]

get_model_stats(model_id)[source]#

Retrieves cumulative execution statistics for a model (shorthand for client.models.get_stats(model_id)).

Parameters:: model_id (str)
Return type:: ModelStats

list_model_stats()[source]#

Retrieves cumulative execution statistics across all models (shorthand for client.models.stats()).

Return type:: List[ModelStats]

get_model(model_id)[source]#

Retrieves metadata for a single model (shorthand for client.models.get(model_id)).

Parameters:: model_id (str)
Return type:: ModelInfo

load_model(model_id, *, pin=False)[source]#

Loads a model into memory (shorthand for client.models.load(model_id, pin=pin)).

Parameters:

model_id (str)
pin (bool)

Return type:

Dict[str, Any]

unload_model(model_id)[source]#

Unloads a model from memory (shorthand for client.models.unload(model_id)).

Parameters:: model_id (str)
Return type:: Dict[str, Any]

evict_model(model_id)[source]#

Evicts a model from memory (shorthand for client.models.evict(model_id)).

Parameters:: model_id (str)
Return type:: Dict[str, Any]

pin_model(model_id)[source]#

Pins a model in memory (shorthand for client.models.pin(model_id)).

Parameters:: model_id (str)
Return type:: Dict[str, Any]

unpin_model(model_id)[source]#

Unpins a model (shorthand for client.models.unpin(model_id)).

Parameters:: model_id (str)
Return type:: Dict[str, Any]

health_live()[source]#

Checks the HTTP liveness probe (GET /health/live).

Returns:: True if the server responds with {"status": "live"}.
Return type:: bool

health_ready()[source]#

Checks the HTTP readiness probe (GET /health/ready).

Returns:: True if the server responds with {"status": "ready"}.
Return type:: bool

wait_until_ready(*, timeout=30.0, poll_interval=0.2)[source]#

Polls /health/ready until the SnoozeML daemon responds ready or timeout expires.

Parameters:

timeout (float)
poll_interval (float)

Return type:

bool

grpc_health(service='')[source]#

Checks the gRPC standard health check service (grpc.health.v1.Health/Check).

Parameters:: service (str) – Service name to query ("" for root server or "snoozeml.v1.InferenceService").
Returns:: True if the service reports SERVING.
Return type:: bool

service_info()[source]#

Retrieves daemon metadata and route discovery info from GET /.

Return type:: Dict[str, Any]

metrics()[source]#

Retrieves Prometheus exposition metrics text from GET /metrics.

Return type:: str

get_metrics()[source]#

Retrieves and parses Prometheus metrics from GET /metrics into a DaemonMetrics object.

Return type:: DaemonMetrics

upload_model(file, *, filename=None, model_id=None, description=None, pinned=False, overwrite=False, max_batch_size=None, max_batch_delay_ms=None, idle_timeout_secs=None, min_residency_secs=None, autoscale_queue_threshold=None, max_replicas=None, routing_strategy=None, device=None, provider=None)[source]#

Uploads and dynamically registers a model via HTTP without mounting local volumes.

Convenience alias for ModelsManager.upload().

Parameters:

file (str | Path | bytes | Any)
filename (str | None)
model_id (str | None)
description (str | None)
pinned (bool)
overwrite (bool)
max_batch_size (int | None)
max_batch_delay_ms (int | None)
idle_timeout_secs (int | None)
min_residency_secs (int | None)
autoscale_queue_threshold (int | None)
max_replicas (int | None)
routing_strategy (str | None)
device (str | None)
provider (str | None)

Return type:

ModelInfo

deploy_sklearn(model, *, model_id, initial_types=None, sample_input=None, input_name='float_input', output_name='output', opset_version=17, overwrite=True, pinned=False, description=None, max_batch_size=None, max_batch_delay_ms=None, idle_timeout_secs=None, min_residency_secs=None, autoscale_queue_threshold=None, max_replicas=None, routing_strategy=None, device=None, provider=None)[source]#

Converts a fitted scikit-learn model to ONNX and deploys it to SnoozeML in one call.

Parameters:

model (Any)
model_id (str)
initial_types (Any | None)
sample_input (Any | None)
input_name (str)
output_name (str)
opset_version (int)
overwrite (bool)
pinned (bool)
description (str | None)
max_batch_size (int | None)
max_batch_delay_ms (int | None)
idle_timeout_secs (int | None)
min_residency_secs (int | None)
autoscale_queue_threshold (int | None)
max_replicas (int | None)
routing_strategy (str | None)
device (str | None)
provider (str | None)

Return type:

ModelInfo

deploy_torch(model, sample_input, *, model_id, input_names=None, output_names=None, dynamic_axes=None, opset_version=17, overwrite=True, pinned=False, description=None, max_batch_size=None, max_batch_delay_ms=None, idle_timeout_secs=None, min_residency_secs=None, autoscale_queue_threshold=None, max_replicas=None, routing_strategy=None, device=None, provider=None)[source]#

Exports a PyTorch nn.Module to ONNX and deploys it to SnoozeML in one call.

Parameters:

model (Any)
sample_input (Any)
model_id (str)
input_names (Any | None)
output_names (Any | None)
dynamic_axes (Mapping[str, Mapping[int, str]] | None)
opset_version (int)
overwrite (bool)
pinned (bool)
description (str | None)
max_batch_size (int | None)
max_batch_delay_ms (int | None)
idle_timeout_secs (int | None)
min_residency_secs (int | None)
autoscale_queue_threshold (int | None)
max_replicas (int | None)
routing_strategy (str | None)
device (str | None)
provider (str | None)

Return type:

ModelInfo

deploy_xgboost(model, *, model_id, overwrite=True, pinned=False, description=None, max_batch_size=None, max_batch_delay_ms=None, idle_timeout_secs=None, min_residency_secs=None, autoscale_queue_threshold=None, max_replicas=None, routing_strategy=None, device=None, provider=None)[source]#

Exports an XGBoost model to .xgb.json and deploys it to SnoozeML in one call.

Parameters:

model (Any)
model_id (str)
overwrite (bool)
pinned (bool)
description (str | None)
max_batch_size (int | None)
max_batch_delay_ms (int | None)
idle_timeout_secs (int | None)
min_residency_secs (int | None)
autoscale_queue_threshold (int | None)
max_replicas (int | None)
routing_strategy (str | None)
device (str | None)
provider (str | None)

Return type:

ModelInfo

refresh_models()[source]#

Triggers runtime discovery and refresh of models on the SnoozeML daemon via HTTP.

Convenience alias for ModelsManager.refresh().

Return type:: List[ModelInfo]

class snoozeml.client.AsyncSnoozeClient(endpoint=None, *, http_endpoint=None, grpc_endpoint=None, use_grpc=None, api_key=None, token=None, timeout=30.0, tls_ca_cert=None, verify_tls=True, max_message_length=67108864, max_retries=0, retry_backoff=0.5, check_version=True)[source]#

Bases: object

Asynchronous client for SnoozeML model management and tensor inference.

Uses grpc.aio for non-blocking gRPC transport and asynchronous HTTP execution.

Example:

import numpy as np
from snoozeml import AsyncSnoozeClient

async with AsyncSnoozeClient(endpoint="localhost:50051", use_grpc=True) as client:
    models = await client.models.list()
    resp = await client.infer("demo-model", {"input": np.ones((1, 10), dtype=np.float32)})

Parameters:

endpoint (Optional[str])
http_endpoint (Optional[str])
grpc_endpoint (Optional[str])
use_grpc (Optional[bool])
api_key (Optional[str])
token (Optional[str])
timeout (float)
tls_ca_cert (Optional[Union[str, bytes, Path]])
verify_tls (bool)
max_message_length (int)
max_retries (int)
retry_backoff (float)
check_version (bool)

async close()[source]#

Asynchronously closes underlying HTTP sessions and grpc.aio channels.

Return type:: None

property server_version: str | None#: Returns the detected SnoozeML server version after the first connection, or None.

async infer(model_id, inputs, *, use_grpc=None, timeout=None, client_timeout_ms=None, parameters=None, retries=None)[source]#

Asynchronously executes unary tensor inference against a registered model.

Parameters:

model_id (str)
inputs (Mapping[str, Any | Sequence[Any] | TensorInput] | Sequence[TensorInput | Mapping[str, Any]] | TensorInput | Any)
use_grpc (bool | None)
timeout (float | None)
client_timeout_ms (int | None)
parameters (Mapping[str, str] | None)
retries (int | None)

Return type:

InferenceResponse

async infer_dataset(model_id, inputs, *, batch_size=32, concurrency=4, use_grpc=None, timeout=None, client_timeout_ms=None, parameters=None, retries=None)[source]#

Asynchronously slices a dataset along Axis(0) into chunks and executes them concurrently.

Parameters:

model_id (str)
inputs (Mapping[str, Any | Sequence[Any] | TensorInput] | Sequence[TensorInput | Mapping[str, Any]] | TensorInput | Any)
batch_size (int)
concurrency (int)
use_grpc (bool | None)
timeout (float | None)
client_timeout_ms (int | None)
parameters (Mapping[str, str] | None)
retries (int | None)

Return type:

InferenceResponse

async stream_infer(model_id, inputs, *, timeout=None, client_timeout_ms=None, parameters=None)[source]#

Asynchronously executes bi-directional streaming inference over persistent gRPC connection.

Parameters:

model_id (str) – Target model identifier.
inputs (AsyncIterable[Mapping[str, Any | Sequence[Any] | TensorInput] | Sequence[TensorInput | Mapping[str, Any]] | TensorInput | Any] | Iterable[Mapping[str, Any | Sequence[Any] | TensorInput] | Sequence[TensorInput | Mapping[str, Any]] | TensorInput | Any]) – An async iterable or synchronous iterable yielding inputs.
timeout (float | None) – Optional overall gRPC stream timeout in seconds.
client_timeout_ms (int | None) – Optional per-request deadline hint in milliseconds.
parameters (Mapping[str, str] | None) – Optional execution parameters.

Yields:

InferenceResponse for each completed frame from the stream.

Raises:

InvalidInputError – If model_id is empty.
SnoozeError – If gRPC is not available or on transport/authentication failure.

Return type:

AsyncIterator[InferenceResponse]

async list_models()[source]#

Asynchronously lists all registered models.

Return type:: List[ModelInfo]

async get_model(model_id)[source]#

Asynchronously retrieves metadata for a specific model.

Parameters:: model_id (str)
Return type:: ModelInfo

async get_model_stats(model_id)[source]#

Asynchronously retrieves cumulative execution statistics for a model.

Parameters:: model_id (str)
Return type:: ModelStats

async list_model_stats()[source]#

Asynchronously retrieves cumulative execution statistics across all models.

Return type:: List[ModelStats]

async load_model(model_id, *, pin=False)[source]#

Asynchronously loads a model into memory.

Parameters:

model_id (str)
pin (bool)

Return type:

Dict[str, Any]

async unload_model(model_id)[source]#

Asynchronously unloads a model from memory.

Parameters:: model_id (str)
Return type:: Dict[str, Any]

async evict_model(model_id)[source]#

Asynchronously evicts a model from memory.

Parameters:: model_id (str)
Return type:: Dict[str, Any]

async pin_model(model_id)[source]#

Asynchronously pins a model in memory.

Parameters:: model_id (str)
Return type:: Dict[str, Any]

async unpin_model(model_id)[source]#

Asynchronously unpins a model.

Parameters:: model_id (str)
Return type:: Dict[str, Any]

async health_live()[source]#

Asynchronously checks the HTTP liveness probe (GET /health/live).

Return type:: bool

async health_ready()[source]#

Asynchronously checks the HTTP readiness probe (GET /health/ready).

Return type:: bool

async wait_until_ready(*, timeout=30.0, poll_interval=0.2)[source]#

Asynchronously polls /health/ready until the SnoozeML daemon responds ready.

Parameters:

timeout (float)
poll_interval (float)

Return type:

bool

async grpc_health(service='')[source]#

Asynchronously checks the gRPC health check service.

Parameters:: service (str)
Return type:: bool

async service_info()[source]#

Asynchronously retrieves daemon metadata from GET /.

Return type:: Dict[str, Any]

async metrics()[source]#

Asynchronously retrieves Prometheus metrics from GET /metrics.

Return type:: str

async get_metrics()[source]#

Asynchronously retrieves and parses Prometheus metrics into DaemonMetrics.

Return type:: DaemonMetrics

async upload_model(file, *, filename=None, model_id=None, description=None, pinned=False, overwrite=False, max_batch_size=None, max_batch_delay_ms=None, idle_timeout_secs=None, min_residency_secs=None, autoscale_queue_threshold=None, max_replicas=None, routing_strategy=None, device=None, provider=None)[source]#

Asynchronously uploads and dynamically registers a model via HTTP.

Convenience alias for AsyncModelsManager.upload().

Parameters:

file (str | Path | bytes | Any)
filename (str | None)
model_id (str | None)
description (str | None)
pinned (bool)
overwrite (bool)
max_batch_size (int | None)
max_batch_delay_ms (int | None)
idle_timeout_secs (int | None)
min_residency_secs (int | None)
autoscale_queue_threshold (int | None)
max_replicas (int | None)
routing_strategy (str | None)
device (str | None)
provider (str | None)

Return type:

ModelInfo

async deploy_sklearn(model, *, model_id, initial_types=None, sample_input=None, input_name='float_input', output_name='output', opset_version=17, overwrite=True, pinned=False, description=None, max_batch_size=None, max_batch_delay_ms=None, idle_timeout_secs=None, min_residency_secs=None, autoscale_queue_threshold=None, max_replicas=None, routing_strategy=None, device=None, provider=None)[source]#

Asynchronously converts a fitted scikit-learn model to ONNX and deploys it.

Parameters:

model (Any)
model_id (str)
initial_types (Any | None)
sample_input (Any | None)
input_name (str)
output_name (str)
opset_version (int)
overwrite (bool)
pinned (bool)
description (str | None)
max_batch_size (int | None)
max_batch_delay_ms (int | None)
idle_timeout_secs (int | None)
min_residency_secs (int | None)
autoscale_queue_threshold (int | None)
max_replicas (int | None)
routing_strategy (str | None)
device (str | None)
provider (str | None)

Return type:

ModelInfo

async deploy_torch(model, sample_input, *, model_id, input_names=None, output_names=None, dynamic_axes=None, opset_version=17, overwrite=True, pinned=False, description=None, max_batch_size=None, max_batch_delay_ms=None, idle_timeout_secs=None, min_residency_secs=None, autoscale_queue_threshold=None, max_replicas=None, routing_strategy=None, device=None, provider=None)[source]#

Asynchronously exports a PyTorch nn.Module to ONNX and deploys it.

Parameters:

model (Any)
sample_input (Any)
model_id (str)
input_names (Any | None)
output_names (Any | None)
dynamic_axes (Mapping[str, Mapping[int, str]] | None)
opset_version (int)
overwrite (bool)
pinned (bool)
description (str | None)
max_batch_size (int | None)
max_batch_delay_ms (int | None)
idle_timeout_secs (int | None)
min_residency_secs (int | None)
autoscale_queue_threshold (int | None)
max_replicas (int | None)
routing_strategy (str | None)
device (str | None)
provider (str | None)

Return type:

ModelInfo

async deploy_xgboost(model, *, model_id, overwrite=True, pinned=False, description=None, max_batch_size=None, max_batch_delay_ms=None, idle_timeout_secs=None, min_residency_secs=None, autoscale_queue_threshold=None, max_replicas=None, routing_strategy=None, device=None, provider=None)[source]#

Asynchronously exports an XGBoost model to .xgb.json and deploys it.

Parameters:

model (Any)
model_id (str)
overwrite (bool)
pinned (bool)
description (str | None)
max_batch_size (int | None)
max_batch_delay_ms (int | None)
idle_timeout_secs (int | None)
min_residency_secs (int | None)
autoscale_queue_threshold (int | None)
max_replicas (int | None)
routing_strategy (str | None)
device (str | None)
provider (str | None)

Return type:

ModelInfo

async refresh_models()[source]#

Asynchronously triggers runtime discovery and refresh of models on the daemon.

Convenience alias for AsyncModelsManager.refresh().

Return type:: List[ModelInfo]

snoozeml.client — Synchronous & Asynchronous Clients#

`snoozeml.client` — Synchronous & Asynchronous Clients#