Architecture — Responsibility Boundaries

This document defines exactly which layer is responsible for what across Rastir’s three tiers: Adapter, Client, and Server. Every adapter implementation must conform to these boundaries.

1. Adapters (per-provider / per-framework)

Adapters are stateless extraction modules. They inspect request arguments and response objects to produce structured metadata. They must never create spans, mutate global state, or interact with the server.

1.1 Adapter Kinds

Kind Priority Range Purpose
framework 200–300 Unwrap framework wrappers (LangChain, LangGraph, LlamaIndex, CrewAI) and detect embedded model objects
provider 100–199 Extract model, provider, tokens, finish_reason from native SDK responses (OpenAI, Anthropic, Bedrock, etc.)
fallback 0 Catch-all for unrecognised responses

1.2 Adapter Capabilities

Every adapter must declare its capabilities via flags on the class:

Flag Meaning
supports_tokens Adapter extracts tokens_input / tokens_output from responses
supports_streaming Adapter handles streaming chunks via extract_stream_delta()
supports_request_metadata Adapter inspects request args to extract model/provider/config pre-invocation
supports_guardrail_metadata Adapter extracts guardrail IDs, actions, and violation categories

1.3 Adapter Interface — Required Methods

Method Phase Returns Responsibility
can_handle(result) Response bool Detect if this adapter owns the response object (use class name + module, never import the SDK)
transform(result) Response AdapterResult Extract model, provider, tokens_input, tokens_output, finish_reason, extra_attributes
can_handle_request(args, kwargs) Request bool Detect if request args contain objects this adapter understands
extract_request_metadata(args, kwargs) Request RequestMetadata Extract model, provider, and provider-specific config (e.g. guardrail_id) from request args
can_handle_stream(chunk) Streaming bool Detect if a streaming chunk belongs to this provider
extract_stream_delta(chunk) Streaming TokenDelta Extract incremental token counts and model/provider from a chunk

1.4 What Adapters Extract

Data Point Request Phase Response Phase Streaming Phase
model From model objects in args (e.g. ChatOpenAI.model_name) From response object (e.g. response.model) From first chunk
provider From module path of model objects From response object module From first chunk
tokens_input — From usage/meta dict Accumulated from deltas
tokens_output — From usage/meta dict Accumulated from deltas
finish_reason — From response choices/stop_reason —
guardrail_id From request kwargs (Bedrock) — —
guardrail_version From request kwargs (Bedrock) — —
guardrail_action — From response trace (Bedrock) —
guardrail_category — From response trace (Bedrock) —

1.5 What Adapters Must NOT Do

  • Create or manage spans / trace context
  • Import provider SDKs at module level (use class-name sniffing)
  • Write metrics or counters
  • Interact with the server or transport
  • Hold mutable state across calls

2. Client (decorators, spans, transport)

The client is the user-facing instrumentation layer. It manages span lifecycle, context propagation, and batch transport.

2.1 Decorators

Decorator Span Type Responsibilities
@llm llm Create span; call resolve_request() pre-invocation; call resolve() or stream-accumulate post-invocation; apply model=/provider= overrides; set agent from context
@agent agent Create span; set agent name in context for child spans to inherit
@trace system Create span; generic function tracing with no AI-specific logic
@metric metric Create span for metric emission only (calls, duration, failures)
@retrieval retrieval Create span; call retrieval adapter for metadata

2.2 Span Lifecycle (Client Owns)

Responsibility Method / Location
Create span (trace_id, span_id, parent_span_id) start_span()
Set span_type Decorator determines type
Set service, env, version rastir.configure()
Set agent label on child spans @agent pushes to context; @llm reads it
Compute duration_ms span.finish()
Set status (OK / ERROR) span.finish(SpanStatus.OK / ERROR)
Record exception details span.record_error(exc) — stores class, message, traceback
Capture prompt/response text _extract_request_metadata() for prompt if applicable
Apply model/provider override @llm(model=..., provider=...) sets before adapter runs
Call adapter request phase resolve_request(args, bound_kw) pre-invocation
Call adapter response phase resolve(result) post-invocation
Accumulate stream chunks _accumulate_stream_chunk() iterates and delegates to adapters
Enqueue span for export enqueue_span(span) after end_span()
Batch HTTP transport BatchTransport posts to push_url / /v1/spans

2.3 What Client Must NOT Do

  • Parse provider-specific response objects (that’s the adapter’s job)
  • Compute metrics or counters
  • Apply redaction
  • Make decisions about sampling

3. Server (processing, metrics, export)

The server is the central processing pipeline that receives spans from clients and derives all observability outputs.

3.1 Ingestion Pipeline

Step Responsibility
Receive spans FastAPI /v1/spans endpoint
Queue management Bounded async queue with backpressure and drop-oldest eviction
Sampling Probabilistic per-trace sampling (retain or drop for storage; metrics always recorded)
Redaction Regex-based PII masking on prompt/response attributes
Metrics derivation Update Prometheus counters/histograms/gauges
OTLP forwarding Convert spans to OTLP protobuf and export to Tempo
Evaluation Async eval queue with registered evaluators (toxicity, etc.)
Trace store In-memory TTL-based span storage for query API

3.2 Metrics Labels (Server Derives)

The server reads raw span attributes and derives these Prometheus labels:

Label Source Applied To
service Span dict service field All metrics
env Span dict env field All metrics
span_type Normalised from raw type → canonical set spans_ingested, duration, errors
status Span status field (OK/ERROR) spans_ingested
model Span attribute model (from adapter) llm_calls, tokens_*, duration, errors, guardrail_*, evaluation_*
provider Span attribute provider (from adapter) Same as model
agent Span attribute agent (from client context) llm_calls, tokens_*, tool_calls, guardrail_*
tool_name Span attribute tool_name (from client) tool_calls
error_type Normalised from exception class → category errors
guardrail_id Span attribute (from Bedrock adapter) guardrail_requests, guardrail_violations
guardrail_version Span attribute (from Bedrock adapter) guardrail_requests
guardrail_action Span attribute (from Bedrock adapter) guardrail_violations
guardrail_category Span attribute, bounded enum (from Bedrock adapter) guardrail_violations
evaluation_type Span attribute (from eval worker) evaluation_*

3.3 Server-Side Guards

Guard Purpose
Cardinality caps Per-dimension limits (model=50, provider=10, tool_name=200, agent=200, error_type=50, guardrail_id=100). Overflow values replaced with __cardinality_overflow__.
Label value length Truncate labels to max_label_value_length (default 128 chars)
Span type normalisation Map unknown types to system
Error type normalisation Map raw exception classes to fixed categories: timeout, rate_limit, validation_error, provider_error, internal_error, unknown
Guardrail enum validation Server-side bounded enum for guardrail_category and guardrail_action

3.4 What Server Must NOT Do

  • Import or understand provider SDK types
  • Create spans or manage trace context
  • Know about decorator logic or function signatures

4. Data Flow Summary 

User Code
    │
    ▼
┌─────────────────────────────────────────────────────────┐
│  CLIENT (decorators)                                    │
│                                                         │
│  @agent / @llm / @trace / @retrieval                    │
│    │                                                    │
│    ├─ start_span()           → create SpanRecord        │
│    ├─ resolve_request()      → ADAPTER request phase    │
│    ├─ fn(*args, **kwargs)    → execute user function    │
│    ├─ resolve() / stream()   → ADAPTER response phase   │
│    ├─ span.finish()          → set status, duration     │
│    ├─ record_error()         → capture exception        │
│    └─ enqueue_span()         → batch transport          │
│                                                         │
│  BatchTransport ──HTTP POST──▶ push_url/v1/spans        │
└─────────────────────────────────────────────────────────┘
                        │
                        ▼
┌─────────────────────────────────────────────────────────┐
│  SERVER                                                 │
│                                                         │
│  /v1/spans → ingestion queue                            │
│    │                                                    │
│    ├─ sampling (probabilistic per-trace)                │
│    ├─ redaction (PII masking)                           │
│    ├─ metrics derivation (Prometheus)                   │
│    ├─ OTLP forwarding (Tempo)                           │
│    ├─ evaluation (async queue → eval workers)           │
│    └─ trace store (in-memory TTL)                       │
└─────────────────────────────────────────────────────────┘

5. Current Adapter Compliance Matrix

Adapter Kind Priority Request Metadata Response Metadata Streaming Guardrails Compliance Notes
OpenAI provider 150 ❌ ✅ model, tokens, finish_reason ✅ — Needs can_handle_request for ChatOpenAI objects
Anthropic provider 150 ❌ ✅ model, tokens, finish_reason ✅ — Needs can_handle_request for ChatAnthropic objects
Bedrock provider 140 ✅ guardrail config ✅ model, tokens, guardrails ✅ ✅ Fully compliant
Azure OpenAI provider 155 ❌ ✅ model, tokens, finish_reason ✅ — Needs can_handle_request for Azure model objects
Gemini provider 150 ❌ ✅ model, tokens ✅ — Needs can_handle_request for Gemini model objects
Groq provider 152 ❌ ✅ model, tokens ✅ — Needs can_handle_request for Groq model objects
Mistral provider 150 ❌ ✅ model, tokens ✅ — Needs can_handle_request for Mistral model objects
Cohere provider 150 ❌ ✅ model, tokens ✅ — Needs can_handle_request for Cohere model objects
LangChain framework 250 ✅ model, provider from model objects ✅ unwrap + model ✅ (flag only) — Stream extraction defers to provider adapter
LangGraph framework 260 ✅ model, provider via node/closure walk ✅ unwrap + model ✅ — Fully compliant
LlamaIndex framework 240 ❌ ✅ unwrap + model ✅ — Needs can_handle_request for LlamaIndex model objects
CrewAI framework 245 ❌ ✅ unwrap + model ❌ — Needs request metadata + streaming
Retrieval provider 50 ❌ ✅ retrieval metadata ❌ — N/A — no LLM, no tokens
Tool provider 10 ❌ ✅ tool metadata ❌ — N/A — no LLM, no tokens
Fallback fallback 0 ❌ ✅ basic model/provider ❌ — Catch-all, minimal by design

Alignment Gaps (TODO)

  1. Provider adapters need request-phase extraction: OpenAI, Anthropic, Gemini, Groq, Mistral, Cohere, Azure OpenAI should implement can_handle_request() / extract_request_metadata() to detect their native SDK model objects in function arguments and return model + provider. This ensures model metadata survives even when the API call fails.

  2. LlamaIndex needs request metadata: Should scan args for LlamaIndex model objects.

  3. CrewAI needs request metadata + streaming: Currently the least compliant framework adapter.

  4. LangChain streaming: Declares supports_streaming = True but does not implement can_handle_stream() / extract_stream_delta() — relies on response unwrapping to provider adapter for streaming.

6. Label Inheritance — Parent → Child Propagation

6.1 Propagation Mechanism

Rastir does not copy attributes from parent spans to children. Instead, it uses dedicated ContextVar variables that decorators set and read:

ContextVar Set by Read by Value
_current_span start_span() get_current_span() Current span (for parent-child linking)
_current_agent @agent @llm, @retrieval Agent name string
_current_model @llm (in _finalize_llm_span) — Model name string
_current_provider @llm (in _finalize_llm_span) — Provider name string

6.2 Inheritance Rules (Normative)

Every span type MUST carry these labels when available. The table below specifies who sets each label and where the value comes from.

Label @agent @llm @retrieval @trace evaluation (server)
agent Sets from decorator arg Reads from _current_agent Reads from _current_agent — Copies from parent LLM span
model — Adapter extracts (request + response) or decorator override Should read from _current_model — Copies from parent LLM span
provider — Adapter extracts (request + response) or decorator override Should read from _current_provider — Copies from parent LLM span
service From configure() From configure() From configure() From configure() From parent LLM span
env From configure() From configure() From configure() From configure() From parent LLM span
evaluator_model — — — — NEW: From JudgeConfig.model
evaluator_provider — — — — NEW: From JudgeConfig.provider

6.3 Typical Call Tree & Label Flow 

@agent("travel_planner")
  │  sets: agent="travel_planner"
  │  pushes: _current_agent = "travel_planner"
  │
  └─▶ @llm(evaluate=True)
        │  reads: agent from _current_agent → "travel_planner"
        │  sets:  model="gpt-4o-mini" (from adapter request/response phase)
        │         provider="openai"   (from adapter)
        │  pushes: _current_model = "gpt-4o-mini"
        │          _current_provider = "openai"
        │
        ├─▶ @retrieval("search_docs")
        │     reads: agent    from _current_agent    → "travel_planner"
        │            model    from _current_model    → "gpt-4o-mini"   ← GAP: not implemented yet
        │            provider from _current_provider → "openai"        ← GAP: not implemented yet
        │
        └─▶ evaluate:toxicity  (server-side, child of LLM span)
              copies: model="gpt-4o-mini", provider="openai", agent="travel_planner"
              NEW:    evaluator_model="gpt-4o-mini", evaluator_provider="openai"

6.4 Server-Side Label Usage

The server reads labels from span attributes to derive metrics. These labels MUST be present on the span for the corresponding metric to be meaningful:

Metric Required Labels Label Source
rastir_spans_ingested_total service, env, span_type, status Universal
rastir_llm_calls_total service, env, model, provider, agent Adapter + context
rastir_tokens_input_total service, env, model, provider, agent Adapter
rastir_tokens_output_total service, env, model, provider, agent Adapter
rastir_tool_calls_total service, env, tool_name, agent, model, provider Context inheritance
rastir_retrieval_calls_total service, env, agent Context
rastir_duration_seconds service, env, span_type, model, provider Context inheritance for non-LLM spans
rastir_errors_total service, env, span_type, error_type, model, provider Context inheritance for non-LLM spans
rastir_tokens_per_call service, env, model, provider Adapter
rastir_guardrail_requests_total service, env, provider, model, agent, guardrail_id, guardrail_version Adapter (Bedrock)
rastir_guardrail_violations_total service, env, provider, model, agent, guardrail_id, guardrail_action, guardrail_category Adapter (Bedrock)
rastir_evaluation_runs_total service, env, model, provider, evaluation_type, evaluator_model, evaluator_provider Parent LLM span + JudgeConfig
rastir_evaluation_failures_total service, env, model, provider, evaluation_type, evaluator_model, evaluator_provider Parent LLM span + JudgeConfig
rastir_evaluation_latency_seconds service, env, model, provider, evaluation_type, evaluator_model, evaluator_provider Parent LLM span + JudgeConfig
rastir_evaluation_score service, env, model, provider, evaluation_type, evaluator_model, evaluator_provider Parent LLM span + JudgeConfig

7. Identified Gaps & Alignment Plan

7.1 Client-Side Gaps

# Gap Location Fix
1 @retrieval does not inherit model/provider from context decorators.py @retrieval wrapper Add get_current_model() / get_current_provider() reads
2 duration histogram gets empty model/provider for non-LLM spans metrics.py record_span() model extraction gated on span_type == "llm" Extract model/provider from attrs for ALL span types that carry them (tool, evaluation, retrieval)
3 errors metric gets empty model/provider for non-LLM spans Same gating as #2 Same fix as #2

7.2 Server-Side Gaps

# Gap Location Fix
4 No evaluator_model / evaluator_provider labels on evaluation metrics evaluation_worker.py, metrics.py Add evaluator_model, evaluator_provider to _eval_labels; populate from JudgeConfig passed through EvaluationTask
5 Evaluation spans re-ingested with empty model/provider in duration/errors metrics.py record_span() Fix #2 covers this — extract model/provider from attrs regardless of span_type
6 retrieval_calls_total missing model/provider labels metrics.py retrieval section Add model, provider labels to rastir_retrieval_calls_total

7.3 Adapter Gaps (from Section 5)

# Gap Fix
7 Provider adapters (OpenAI, Anthropic, Gemini, Groq, Mistral, Cohere, Azure) lack request-phase extraction Implement can_handle_request() / extract_request_metadata() on each
8 LlamaIndex, CrewAI lack request metadata Implement same
9 LangChain declares streaming but doesn’t implement stream methods Either implement or remove the flag

8. Rules for New Adapters

  1. Never import the provider SDK at module level. Use type(obj).__name__ and type(obj).__module__ for detection.
  2. Always implement both request and response phases. A well-behaved adapter provides can_handle_request() + extract_request_metadata() AND can_handle() + transform().
  3. Use BaseAdapter._find_in_args() to scan positional and keyword arguments.
  4. Use BaseAdapter._extract_model_attr() to read the first available model attribute from an object.
  5. Use detect_provider_from_module() to map module names to canonical provider strings.
  6. Declare capability flags accurately — the registry uses them to skip unnecessary calls.
  7. Set priority in the correct range for your adapter kind.
  8. Register via __init__.py — adapters are auto-registered on import.
  9. Labels must flow downward. Any label set by a parent decorator (agent, model, provider) must be readable by child spans via ContextVars. If adding a new inheritable label, add a ContextVar + getter/setter in context.py.

Rastir — LLM & Agent Observability Library