The provider-neutral LLM field manual.

Usage Guide

Start with the canonical path.

The default OmniLLM path is intentionally conservative: one typed generation request, one gateway, multiple provider backends. Application code stays centered on LlmRequest and LlmResponse while the runtime handles operational concerns.

DefaultUse Gateway::call for non-streaming generation and Gateway::stream for canonical streaming. This is the right entry point when your product wants provider-neutral generation behavior without hand-writing provider adapters.

let gateway = GatewayBuilder::new(ProviderEndpoint::openai_responses())
    .add_key(KeyConfig::new("sk-key-1", "prod-1").rpm_limit(500))
    .budget_limit_usd(50.0)
    .request_timeout(Duration::from_secs(45))
    .build()?;

let response = gateway.call(request, CancellationToken::new()).await?;

Architecture

Two modes, one budget ledger.

OmniLLM exposes a normalized canonical mode and an explicit primitive mode. The first keeps generation portable. The second preserves provider-native request and response bodies for APIs that do not map cleanly onto a single generation schema.

Normalize the application contract. Preserve the provider contract when the provider API is the product.OmniLLM protocol posture

Runtime Profiles

Endpoint names describe wire shape.

Runtime configuration uses EndpointProtocol. Low-level parsing, emission, and transcoding use ProviderProtocol. Names such as ClaudeMessages and GeminiGenerateContent are wire-shape identifiers, not marketing preferences.

Native APIs

Primitive mode keeps raw provider APIs intact.

Primitive calls are additive and explicit. They are the escape hatch for provider APIs where the raw provider contract is part of your product behavior. PrimitiveRequest, primitive_call, primitive_stream, and primitive_realtime keep primitive paths additive instead of forcing raw provider APIs through canonical generation conversion.

let request = PrimitiveRequest::json(
    ProviderPrimitiveKind::ImagesGenerate,
    serde_json::json!({
        "model": "gpt-image-1",
        "prompt": "A blueprint-style gateway diagram"
    }),
);

let raw = gateway.primitive_call(request, CancellationToken::new()).await?;

Conversion

Loss is reported, not hidden.

Transcoding returns explicit bridge metadata through ConversionReport<T>. Callers can inspect bridged, lossy, and loss_reasons instead of guessing which provider-specific fields were dropped.

let report = transcode_api_request(
    WireFormat::OpenAiChatCompletions,
    WireFormat::OpenAiResponses,
    raw_chat,
)?;

assert!(report.bridged);
assert!(!report.lossy);
for reason in report.loss_reasons {
    tracing::warn!(%reason, "provider bridge dropped data");
}

API Reference

API surface is layered by intent.

Treat the API as layers. Start with the stable application contract, choose the endpoint family that matches the job, drop to primitive mode only when the raw provider contract matters, then keep transport, replay, and operational signals attached to every call.

Registry

Provider coverage is capability-scoped.

Provider support is expressed as endpoint and primitive capabilities rather than SDK parity claims. This keeps the registry honest about transport shape, request path, response preservation, and settlement behavior.

Testing

Fixtures should be useful and safe.

Record/replay tests need reviewable artifacts that do not leak secrets. OmniLLM ships ReplayFixture, sanitize_transport_request, sanitize_transport_response, and sanitize_json_value for safe fixture workflows.

1. Record real transport requests and responses only in controlled integration runs.
2. Sanitize auth headers, query tokens, JSON secrets, and large binary or base64 payload fields.
3. Review fixture diffs as API contracts instead of opaque snapshots.
4. Replay against deterministic fixtures before modifying protocol bridges or provider profiles.

let sanitized = sanitize_transport_request(recorded_request);
let fixture = ReplayFixture::from_transport(sanitized, response);
fixture.write_json("tests/fixtures/openai_responses.basic.json")?;

Operations

Runtime status belongs in the interface.

Production services can inspect gateway.pool_status() and gateway.budget_remaining_usd(). These surfaces expose key availability, inflight token pressure, RPM pressure, circuit state, and remaining budget after pre-reserve plus settlement accounting.

SignalsWhen a request fails, OmniLLM-specific errors tell operators whether the failure belongs to pool availability, spend policy, or protocol conversion.

Production

Before shipping, make the operational contract explicit.

The runtime gives you the primitives, but production readiness comes from making budget, cancellation, replay, and provider behavior visible in your service boundary.

1. Choose the canonical protocol for portable generation before reaching for provider primitives.
2. Configure every production key with explicit labels, RPM limits, cooldown behavior, and budget assumptions.
3. Use CancellationToken and request timeouts for every gateway path, including primitive and stream calls.
4. Treat ConversionReport as part of acceptance criteria when transcoding between provider formats.
5. Record replay fixtures only after sanitization has removed auth headers, query secrets, and large binary fields.
6. Expose pool_status, budget_remaining_usd, and OmniLLM error classes to operators before launch.

AI-native Project

The Skill teaches agents the real library.

The bundled OmniLLM Skill gives coding agents repository-native signals instead of generic Rust SDK guesses. It is tuned to GatewayBuilder, ProviderEndpoint, EndpointProtocol, WireFormat, ReplayFixture, primitive calls, and OmniLLM runtime errors.

1. Install the Skill into Claude Code, Codex, OpenCode, or Claude-compatible skill runners.
2. Ask for gateway setup, endpoint selection, protocol transcoding, replay fixture generation, or OmniLLM-specific error debugging.
3. Verify answers against real examples, tests, endpoint profiles, conversion reports, and runtime surfaces.

# Codex / Claude Code / OpenCode
npx @vercel-labs/skills install github:aiomni/omnillm/skill

# Rust runtime
cargo add omnillm