gRPC API Reference

This page provides the complete technical specification for SnoozeML’s gRPC Inference and Health Checking API, including both the native snoozeml.v1.InferenceService and the NVIDIA Triton / Open Inference Protocol v2 (inference.GRPCInferenceService).

---

1. Protocol Buffer Specification

The native Protocol Buffer definition is located in the repository at proto/snoozeml/v1/inference.proto, and the official Triton / Open Inference Protocol v2 definitions are located under proto/third_party/ (grpc_service.proto, model_config.proto, health.proto):

syntax = "proto3";

package snoozeml.v1;

// Service providing low-latency tensor inference over gRPC.
service InferenceService {
  // Executes unary inference for a registered model.
  rpc Infer(ModelInferRequest) returns (ModelInferResponse);

  // Executes bi-directional streaming inference for a registered model.
  rpc StreamInfer(stream ModelInferRequest) returns (stream ModelInferResponse);
}

// Supported tensor element data types.
enum DataType {
  DATA_TYPE_UNSPECIFIED = 0;
  DATA_TYPE_FP32 = 1;
  DATA_TYPE_FP16 = 2;
  DATA_TYPE_BF16 = 3;
  DATA_TYPE_INT32 = 4;
  DATA_TYPE_INT64 = 5;
  DATA_TYPE_UINT8 = 6;
  DATA_TYPE_BOOL = 7;
}

// Multi-dimensional tensor payload with raw byte buffer.
message TensorData {
  // Binding identifier of the tensor node (e.g. "input_ids", "logits").
  string name = 1;

  // Tensor dimensions (e.g. [1, 3, 224, 224]).
  repeated int64 shape = 2;

  // Primitive element data type (defaults to FP32).
  DataType data_type = 3;

  // Contiguous little-endian byte representation (4 bytes per float for FP32).
  bytes raw_data = 4;
}

message ModelInferRequest {
  // Target model identifier.
  string model_id = 1;

  // Input tensors.
  repeated TensorData inputs = 2;

  // Optional client-specified request timeout in milliseconds.
  optional uint64 client_timeout_ms = 3;

  // Optional execution parameters or routing hints.
  map<string, string> parameters = 4;
}

message ModelInferResponse {
  // Identifier of the executed model.
  string model_id = 1;

  // Resulting output tensors.
  repeated TensorData outputs = 2;

  // Execution latency in milliseconds with microsecond precision.
  double duration_ms = 3;
}

---

2. Services & RPC Methods

snoozeml.v1.InferenceService

Provides synchronous and streaming model execution.

Infer

rpc Infer(ModelInferRequest) returns (ModelInferResponse);

Executes a unary inference pass for the requested model. The model is routed through the internal dynamic micro-batcher. If the model is idle, it is automatically rehydrated into memory.

• Required Role: Inference (or Admin).
• Input: ModelInferRequest
• Output: ModelInferResponse

StreamInfer

rpc StreamInfer(stream ModelInferRequest) returns (stream ModelInferResponse);

Executes bi-directional streaming inference over a persistent HTTP/2 stream, routing each incoming frame through the dynamic micro-batcher.

• Required Role: Inference (or Admin).
• Input: stream ModelInferRequest
• Output: stream ModelInferResponse

---

inference.GRPCInferenceService (NVIDIA Triton & Open Inference Protocol v2)

SnoozeML implements the official Triton / Open Inference Protocol v2 gRPC service (proto/third_party/grpc_service.proto), enabling drop-in compatibility with tritonclient.grpc.

RPC Method	Required Role	Status / Description
ServerLive	Public	Returns live: true.
ServerReady	Public	Returns ready: true.
ModelReady	ReadOnly	Returns ready: true when the model is registered and has at least one loaded replica (Ready or Busy).
ServerMetadata	ReadOnly	Returns server name ("snooze-ml"), version, and supported OIP extensions.
ModelMetadata	ReadOnly	Returns model name, versions (["1"]), backend platform, and tensor metadata.
ModelConfig	ReadOnly	Returns Triton ModelConfig mapped from SnoozeML’s dynamic batching and replica settings.
ModelStatistics	ReadOnly	Returns cumulative per-model inference counts, batch execution counts, and nanosecond durations.
RepositoryIndex	ReadOnly	Lists all registered models (READY, LOADING, or UNAVAILABLE when unloaded via scale-to-zero).
RepositoryModelLoad	Admin	Preloads or rehydrates model weights into memory.
RepositoryModelUnload	Admin	Unloads all replicas of a model to scale to zero.
ModelInfer	Inference	Unary inference supporting both raw_input_contents and typed InferTensorContents.
ModelStreamInfer	Inference	Bi-directional streaming inference (stream ModelInferRequest -> stream ModelStreamInferResponse).
SystemSharedMemory*, CudaSharedMemory*, TraceSetting, LogSettings	Admin	Out of scope; returns canonical UNIMPLEMENTED (12) status code.

---

grpc.health.v1.Health

SnoozeML implements the official gRPC Health Checking Protocol.

Check

rpc Check(HealthCheckRequest) returns (HealthCheckResponse);

• Required Role: Public (no authentication required).
• Supported Service Names:
  • "" (empty string): Overall daemon server status.
  • "snoozeml.v1.InferenceService": SnoozeML native inference service status.
  • "inference.GRPCInferenceService": Triton / OIP v2 inference service status.
• Serving Status: Returns SERVING (1) when the daemon is operational.

---

3. Message Types & Fields

ModelInferRequest

Field	Type	Required	Description
model_id	string	Yes	Identifier of the registered model to execute. Cannot be empty or whitespace.
inputs	repeated TensorData	Yes	Array of input tensors matching model graph bindings.
client_timeout_ms	optional uint64	No	Optional deadline hint from client in milliseconds.
parameters	map<string, string>	No	Optional execution hints or routing flags.

---

ModelInferResponse

Field	Type	Description
model_id	string	Identifier of the model that executed the request.
outputs	repeated TensorData	Output tensors generated by the model forward pass.
duration_ms	double	Execution wall-clock latency in milliseconds (microsecond precision).

---

TensorData

Field	Type	Required	Description
name	string	Yes	Input or output binding name in the model graph.
shape	repeated int64	Yes	Array of positive dimension sizes (e.g. [1, 3, 224, 224]). Dimensions must be $> 0$.
data_type	DataType	No	Primitive data type. Must be DATA_TYPE_FP32 or DATA_TYPE_UNSPECIFIED.
raw_data	bytes	Yes	Contiguous little-endian byte array containing binary float values.

Buffer Size Invariant

For FP32 tensors, the length of raw_data must strictly equal:

$\text{raw\_data.len} = \left(\prod_{i=0}^{N-1} \text{shape}[i]\right) \times 4$

If raw_data.len does not match this invariant, the RPC returns Status::invalid_argument.

---

DataType (Enum)

Enum Value	Number	Supported?	Description
DATA_TYPE_UNSPECIFIED	0	Yes	Defaults to FP32.
DATA_TYPE_FP32	1	Yes	Standard 32-bit single-precision IEEE-754 float (4 bytes).
DATA_TYPE_FP16	2	Future	16-bit half-precision float (2 bytes).
DATA_TYPE_BF16	3	Future	16-bit Brain floating-point (2 bytes).
DATA_TYPE_INT32	4	Future	32-bit signed integer (4 bytes).
DATA_TYPE_INT64	5	Future	64-bit signed integer (8 bytes).
DATA_TYPE_UINT8	6	Future	8-bit unsigned integer (1 byte).
DATA_TYPE_BOOL	7	Future	Boolean (1 byte).

---

4. Metadata Headers

Clients can attach metadata headers to gRPC requests:

Request Metadata

Header Key	Format / Example	Description
x-api-key	sk-infer-client-654	Static API key matching configured Inference or Admin key.
authorization	Bearer <token>	Static API key or cryptographically signed HMAC JWT token.

Response Metadata

Header Key	Format / Example	Description
retry-after	5	Present on RESOURCE_EXHAUSTED errors; indicates recommended seconds before client retries.

---

5. Status Codes & Error Mapping

SnoozeML maps internal engine errors directly to canonical gRPC status codes:

Canonical gRPC Code	Code Number	Internal Domain Error	Equivalent HTTP Code	Root Cause & Resolution
OK	0	Success	200 OK	Request succeeded.
INVALID_ARGUMENT	3	InvalidInput, ShapeMismatch	400 Bad Request	Empty model ID, non-positive shape dimension, unsupported data type, or truncated raw_data buffer.
DEADLINE_EXCEEDED	4	Timeout	504 Gateway Timeout	Processing queue deadline or batch wait budget exceeded.
NOT_FOUND	5	NotFound	404 Not Found	Requested model_id is not registered in the daemon.
RESOURCE_EXHAUSTED	8	ResourceExhausted, OutOfMemory	503 Service Unavailable	GPU/Host memory exhausted and all active models are busy. Response includes retry-after: 5 metadata.
UNIMPLEMENTED	12	UnsupportedSafetensorsArchitecture	501 Not Implemented	Model format or architecture not supported by the execution backend, or out-of-scope Triton shared-memory/trace RPC.
INTERNAL	13	Runtime, QueueError	500 Internal Server Error	Unexpected backend or engine calculation failure.
UNAVAILABLE	14	Loading	503 Service Unavailable	Model is currently loading weights into memory.
UNAUTHENTICATED	16	Missing/Invalid Credentials	401 Unauthorized	Missing x-api-key or authorization: Bearer header, invalid key, or expired JWT.
PERMISSION_DENIED	7	Insufficient Role	403 Forbidden	Caller has ReadOnly role, but Inference or Admin role is required.