Telemetry

Roster provides vendor-neutral telemetry for understanding application health, request performance, Resolve activity, model usage, and worker processing. It exports traces and metrics to standards-compliant OTLP platforms and keeps structured application logs on stdout.

What Roster telemetry provides

• Metrics measure request traffic and latency, Resolve results, model usage, worker queues, and operational failures.
• Traces follow requests and their child operations across Roster.
• Structured Pino logs remain on stdout. While a trace is active, Roster adds trace_id and span_id so operators can correlate logs with traces.

Roster does not store time-series telemetry or bundle a monitoring backend. You select and operate the OTLP-compatible platform that receives the data.

The advanced telemetry layer is available in Enterprise. Free and Pro retain structured logs, health probes, and trace correlation, but do not send external telemetry or expose the advanced System health page and observability API/MCP tool. See Licensing for edition details.

System health and delivery status

Active administrators can open System health to see database health, active instances, worker backlog, dropped batches, and OTLP delivery state. When ROSTER_OBSERVABILITY_DASHBOARD_URL is configured, the page also provides a link to the external dashboard.

Delivery state has four values:

• disabled: export was not requested or is not available for the edition.
• pending: export is enabled but both signals have not completed delivery.
• healthy: trace and metric batches were accepted by the OTLP endpoint.
• degraded: configuration or OTLP delivery failed.

In Enterprise, the same sanitized status is available to active administrators through:

• GET /api/v1/observability with api:observability:read
• MCP get_observability_status with mcp:observability:read

Privacy and reliability

Roster telemetry uses bounded, low-cardinality attributes. It never exports prompts, model output, tool payloads, HTTP bodies, authorization data, cookies, SQL text, names, emails, IP addresses, user agents, or user-authored content. Request and job IDs may be present only in traces and structured logs, never in metric labels.

Exporter queues, batches, timeouts, and metric cardinality are bounded. Export errors affect only observability status; they do not change requests, workers, liveness, or readiness.

Use telemetry in production

Send Roster telemetry to your existing OpenTelemetry Collector or directly to a standards-compliant OTLP backend. Grafana is the reference interface used by the supplied examples, not a requirement; any compliant OTLP platform is supported.

Before you begin

Prepare the following for each production environment:

• an Enterprise-enabled Roster deployment;
• an OTLP endpoint reachable from every Roster instance;
• TLS and authentication appropriate for that endpoint;
• a secret manager for OTLP credentials; and
• an existing log pipeline for Roster’s structured stdout logs.

Configure Roster

Export is disabled by default. The default protocol is OTLP HTTP/protobuf, the default trace sampler is parent-based 10%, and metrics export every 60 seconds. The following example shows a complete production configuration:

ROSTER_OBSERVABILITY_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.example:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20<secret>
OTEL_SERVICE_NAME=roster-platform
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1
OTEL_METRIC_EXPORT_INTERVAL=60000
ROSTER_OBSERVABILITY_DASHBOARD_URL=https://grafana.example/d/roster-overview

For gRPC, use OTEL_EXPORTER_OTLP_PROTOCOL=grpc. Signal-specific endpoint, protocol, and header variables override the generic values. Keep credentials in your secret manager. Roster status returns only endpoint origins, never credentials, headers, or complete endpoint paths.

When export is enabled without an endpoint, OpenTelemetry’s local defaults are used: http://localhost:4318 for HTTP/protobuf or port 4317 for gRPC. Set an explicit endpoint in production so every Roster instance uses the intended Collector or backend.

See Environment Variables for the complete runtime variable reference.

Verify production delivery

After applying the configuration and restarting or rolling out Roster:

1. Generate normal Roster activity.
2. Open System health as an active administrator.
3. Confirm that OTLP delivery changes from pending to healthy.
4. Confirm independently that the destination platform receives metrics and traces.

A healthy state confirms that Roster delivered trace and metric batches to the configured OTLP endpoint. It does not prove that a downstream platform indexed or retained them. Monitor the Collector and backend separately.

Production security and operations

Authenticate and encrypt OTLP traffic, restrict dashboard access, store telemetry in durable backends, define retention and backup policies, and monitor the Collector and storage services separately. Do not expose OTLP receiver ports directly to the public Internet.

Roster does not export application logs through OpenTelemetry. Collect container stdout with your deployment platform or log agent. If logs are sent to Loki or another Grafana-compatible data source, preserve trace_id and span_id and configure a derived link to the trace data source.

Evaluate telemetry locally with Docker Compose

The downloadable Compose example runs OpenTelemetry Collector, Prometheus, Tempo, and Grafana on one machine. It is intended for local evaluation or a protected internal test environment, not as a production topology. Roster is configured separately and is not included as a service in this stack.

No Grafana Cloud, Prometheus, Tempo, or OpenTelemetry account is required.

Before you begin

You need:

• Docker with Docker Compose;
• an Enterprise-enabled Roster instance; and
• local ports 3001, 3200, 4317, 4318, and 9090 available.

The example binds every published service port to 127.0.0.1, which prevents access from other machines unless you deliberately change the configuration.

Understand the local services

Service	Role
OpenTelemetry Collector	Receives OTLP traces and metrics, applies limits and filtering, and routes each signal.
Prometheus	Scrapes and stores Roster metrics exported by the Collector.
Tempo	Stores Roster traces and makes them queryable from Grafana.
Grafana	Provides dashboards and Explore views backed by Prometheus and Tempo.

The local data flow is:

Roster → OpenTelemetry Collector → Prometheus or Tempo → Grafana

Download the example

Download all eight files and preserve the displayed directory structure:

File	Purpose
docker-compose.yml	Starts the Collector, Prometheus, Tempo, and Grafana on loopback ports.
otel-collector.local.yaml	Receives OTLP and routes metrics to Prometheus and traces to Tempo.
prometheus.local.yaml	Configures Prometheus to scrape the Collector.
prometheus-recording-rules.yaml	Defines the supplied dashboard aggregations and local alerts.
tempo.local.yaml	Configures local trace storage.
grafana-dashboard.json	Defines the Roster Observability dashboard.
grafana/provisioning/dashboards/roster.yaml	Loads the dashboard into Grafana.
grafana/provisioning/datasources/roster.yaml	Provisions the Prometheus and Tempo data sources.

These Advantys-provided example configurations are intended for use with Roster deployments and remain subject to the Roster license terms applicable to your deployment. The included third-party components remain subject to their respective licenses.

View complete docker-compose.yml

docker-compose.yml

name: roster-observability

services:
  otel-collector:
    image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.156.0
    command: ["--config=/etc/otelcol/config.yaml"]
    read_only: true
    cap_drop: [ALL]
    security_opt: [no-new-privileges:true]
    tmpfs: [/tmp]
    volumes:
      - ./otel-collector.local.yaml:/etc/otelcol/config.yaml:ro
    ports:
      - "127.0.0.1:4317:4317"
      - "127.0.0.1:4318:4318"
    depends_on:
      - tempo
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:v3.12.0-distroless
    command:
      - --config.file=/etc/prometheus/prometheus.yaml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=7d
      - --web.enable-lifecycle
    read_only: true
    cap_drop: [ALL]
    security_opt: [no-new-privileges:true]
    volumes:
      - ./prometheus.local.yaml:/etc/prometheus/prometheus.yaml:ro
      - ./prometheus-recording-rules.yaml:/etc/prometheus/rules/roster-recording-rules.yaml:ro
      - prometheus-data:/prometheus
    ports:
      - "127.0.0.1:9090:9090"
    depends_on:
      - otel-collector
    restart: unless-stopped

  tempo:
    image: grafana/tempo:2.10.5
    command: ["-config.file=/etc/tempo/config.yaml"]
    read_only: true
    cap_drop: [ALL]
    security_opt: [no-new-privileges:true]
    volumes:
      - ./tempo.local.yaml:/etc/tempo/config.yaml:ro
      - tempo-data:/var/tempo
    ports:
      - "127.0.0.1:3200:3200"
    restart: unless-stopped

  grafana:
    image: grafana/grafana:13.1.0
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: "false"
      GF_SECURITY_ADMIN_PASSWORD: ${ROSTER_GRAFANA_ADMIN_PASSWORD:-roster-local-only}
      GF_SECURITY_ADMIN_USER: admin
      GF_USERS_ALLOW_SIGN_UP: "false"
    cap_drop: [ALL]
    security_opt: [no-new-privileges:true]
    volumes:
      - ./grafana-dashboard.json:/var/lib/grafana/dashboards/roster.json:ro
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - grafana-data:/var/lib/grafana
    ports:
      - "127.0.0.1:3001:3000"
    depends_on:
      - prometheus
      - tempo
    restart: unless-stopped

volumes:
  grafana-data:
  prometheus-data:
  tempo-data:

On Linux or macOS:

base_url=https://docs.rosterresolve.ai/examples/observability
mkdir -p roster-observability/grafana/provisioning/{dashboards,datasources}
cd roster-observability

for file in \
  docker-compose.yml \
  otel-collector.local.yaml \
  prometheus.local.yaml \
  prometheus-recording-rules.yaml \
  tempo.local.yaml \
  grafana-dashboard.json
do
  curl -fsSLO "$base_url/$file"
done

curl -fsSLo grafana/provisioning/dashboards/roster.yaml \
  "$base_url/grafana/provisioning/dashboards/roster.yaml"
curl -fsSLo grafana/provisioning/datasources/roster.yaml \
  "$base_url/grafana/provisioning/datasources/roster.yaml"

On Windows PowerShell:

$baseUrl = "https://docs.rosterresolve.ai/examples/observability"
$root = Join-Path (Get-Location) "roster-observability"
$files = @(
  "docker-compose.yml",
  "otel-collector.local.yaml",
  "prometheus.local.yaml",
  "prometheus-recording-rules.yaml",
  "tempo.local.yaml",
  "grafana-dashboard.json",
  "grafana/provisioning/dashboards/roster.yaml",
  "grafana/provisioning/datasources/roster.yaml"
)

foreach ($file in $files) {
  $destination = Join-Path $root $file
  New-Item -ItemType Directory -Force -Path (Split-Path $destination) | Out-Null
  Invoke-WebRequest -Uri "$baseUrl/$file" -OutFile $destination
}

Set-Location $root

Start the local stack

Set a unique Grafana administrator password and start all four services.

On Linux or macOS:

ROSTER_GRAFANA_ADMIN_PASSWORD='<strong-local-password>' docker compose up -d
docker compose ps

On Windows PowerShell:

$env:ROSTER_GRAFANA_ADMIN_PASSWORD = "<strong-local-password>"
docker compose up -d
Remove-Item Env:ROSTER_GRAFANA_ADMIN_PASSWORD
docker compose ps

Confirm that Collector, Prometheus, Tempo, and Grafana are running before configuring Roster.

Connect Roster

Add the following values to the Roster environment, replace <otel-endpoint>, then restart or recreate Roster:

ROSTER_OBSERVABILITY_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=<otel-endpoint>
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=1
OTEL_METRIC_EXPORT_INTERVAL=15000
ROSTER_OBSERVABILITY_DASHBOARD_URL=http://localhost:3001/d/roster-overview

Use the appropriate local endpoint:

• If Roster runs directly on the same host, use http://localhost:4318.

• If Roster runs in Docker, use http://otel-collector:4318 and add the observability network to the existing docker run command:

  --network roster-observability_default

  Keep the existing Roster environment, port, and durable /data volume options unchanged when recreating the container. The Compose project name makes roster-observability_default the network created by this example.

Use the normal Roster deployment procedure to apply these variables. See Deployment Overview if you need to create or recreate the Roster container.

Verify the local evaluation

1. Sign in to Roster as an active administrator.
2. Open System health and confirm that OTLP delivery is enabled.
3. Generate activity by opening Roster pages and running a representative Playground or Resolve request.
4. Allow at least two metric export intervals for rate and histogram data.
5. Confirm that delivery changes from pending to healthy.
6. Open http://localhost:3001 and sign in to Grafana with user admin and the password supplied through ROSTER_GRAFANA_ADMIN_PASSWORD.
7. Open the provisioned Roster Observability dashboard and select a time range containing the recent activity.

Prometheus is available at http://localhost:9090 and Tempo at http://localhost:3200 for optional direct inspection.

Stop or reset the local stack

Stop the example while preserving its local Prometheus, Tempo, and Grafana data:

docker compose down

To delete the disposable local observability data as well, run the following only when that data is no longer needed:

docker compose down -v

Explore Roster telemetry in Grafana

The supplied Roster Observability dashboard shows request rate, server error ratio, HTTP p95 latency, normalized route outcomes, Resolve error ratio and resolution yield, p50/p95/p99 Resolve latency by mode and surface, model latency, token throughput, worker queue state, and operational failures.

Resolve error ratio is the share of Resolve requests with roster_outcome="error". Resolution yield is the share with roster_outcome="success"; not_found and out_of_scope are valid non-error outcomes and remain visible separately.

Query metrics with Prometheus

Prometheus stores numeric measurements over time. Grafana uses PromQL, the Prometheus query language, to retrieve and calculate those measurements.

To run an ad hoc query in Grafana:

1. Open Explore.
2. Select Roster Prometheus as the data source.
3. Select Code in the query editor.
4. Paste a PromQL query and select Run query.

Start with the cumulative HTTP request counters:

roster_http_server_request_count_total

A metric name by itself is a valid PromQL query. This query returns a separate series for each available combination of labels, such as request method, normalized route, and outcome. Counter values increase as requests occur.

To calculate the overall request rate per second over the last five minutes, run:

sum(rate(roster_http_server_request_count_total[5m]))

To inspect the current worker queue depth, run:

roster_worker_queue_depth_job

This query returns the queued, running, and failed queue states. A value of 0 means no jobs are currently in that state.

Other useful Roster metrics include:

Metric	What it shows	Appears after
roster_http_server_request_duration_seconds_bucket	HTTP request latency distribution	HTTP traffic
roster_resolve_request_active	Resolve requests currently in progress	A Resolve request
roster_resolve_request_duration_seconds_bucket	Resolve latency distribution	A Resolve request
roster_resolve_request_result_participant_count_bucket	Participant result-size distribution	A Resolve result
roster_resolve_request_result_user_count_bucket	User result-size distribution	A Resolve result with users
roster_llm_token_usage_total	Cumulative model token consumption	A model invocation
roster_telemetry_export_error_count_total	Cumulative telemetry export failures	An export failure

Resolve request metrics use bounded test/live mode, surface, and outcome dimensions. Result-size histograms are emitted only when Resolve produces a result.

If a query returns No data, generate the corresponding activity in Roster and confirm that the selected time range includes it. Resolve and model metrics remain empty until those features are used. Rate and histogram queries need more than one sample, so allow at least two metric export intervals after enabling telemetry.

Inspect traces with Tempo

To inspect Roster traces in Grafana:

1. Open Explore.
2. Select Roster Tempo as the data source.
3. Select TraceQL in the query editor.
4. Paste the following query and select Run query:

{ resource.service.name = "roster-platform" }

The braces select traces containing a matching span, and resource.service.name is the OpenTelemetry service name assigned by Roster. Select a returned trace to inspect its request and child operations. If OTEL_SERVICE_NAME is set to another value, replace roster-platform in the query. To inspect a known trace directly, paste a trace_id from a structured Roster log into Tempo’s trace search.

The local Compose stack does not include a log store. Use the container or hosting-platform log viewer and copy trace_id into Grafana Explore for correlation. To search Roster logs in Grafana, send the structured stdout JSON records to Loki or another Grafana-compatible log data source, preserve trace_id and span_id, and configure a derived trace link to Roster Tempo.