Sonda

Your agent's eyes into production.

Sonda is a self-hosted, read-only gateway that gives AI agents one credential and one service-centric interface to production telemetry. It stores no telemetry: it queries the sources on demand, restricts every query to the services an agent may see, and returns compact responses with links back to the source.

Source	Capabilities
Sentry	errors, releases, traces
Loki	logs
Prometheus (and Mimir, Thanos, VictoriaMetrics, Grafana Cloud)	metrics, alerts
Cloudflare Workers Observability and GraphQL Analytics	logs, metrics, traces
AWS CloudWatch Logs Insights and metrics	logs, metrics
Grafana Tempo	traces
Kubernetes pods, ReplicaSets and events	workloads
Grafana managed alerting	alerts
Vercel runtime logs	logs

agent ──sonda CLI──► sonda-server ──► Sentry / Loki / Prometheus / Cloudflare / CloudWatch / Tempo / Kubernetes / Grafana / Vercel
          one token     authz, query scoping, grouping, limits, audit

• One token per agent. Provider credentials stay on the server.
• Service-centric. Agents ask for engine-api in production; Sonda knows the Sentry project, the Loki selector, the Worker script, the log groups and the Tempo service name.
• Enforced scoping. Native queries are allowed in each backend's own dialect (LogQL, PromQL, TraceQL, Logs Insights, Cloudflare query JSON, Vercel KQL). Sonda adds the service scope to every query, so a query cannot reach another service. describe tells the agent which dialect a capability takes.
• Compact answers. Log search returns an exact count (an estimate for Cloudflare, capped at 10,000 for Vercel) and a sample grouped into patterns; metrics return summaries; a trace is a span tree. Large results stay on the server behind result handles.
• Audited and limited. Every request is logged; time range, lines, response size, timeouts, rate and concurrency are limited per identity.

Sonda does not reason about incidents. The agent does; Sonda supplies the evidence.

Quick start

go build -o . ./cmd/...            # builds sonda-server and sonda

cp sonda.example.yaml sonda.yaml   # configure sources and services
export SONDA_ADMIN_TOKEN=$(openssl rand -hex 32)
./sonda-server check -c sonda.yaml
./sonda-server serve -c sonda.yaml

# Open http://localhost:8080/admin, sign in with the admin token, create an identity
# and copy its token.
./sonda auth login --url http://localhost:8080   # paste the token; saved to ~/.config/sonda/config.yaml (0600)
./sonda services
./sonda overview engine-api -e production

Requires Go 1.26.5 or newer. A container image builds from the Dockerfile.

For agents

skills/sonda/SKILL.md teaches an agent the investigation flow and the commands. Install it as a skill for Claude Code, or paste it into the agent's instructions.

Command	Purpose
sonda services, sonda describe <svc>	What the token can access; backend and dialect per capability
sonda overview <svc>	Releases, top errors, signal summaries, pod health, active alerts
sonda errors <svc>, sonda error <svc> <id>, sonda releases <svc>	Sentry
sonda logs search/query/fields <svc>, sonda result <id>	Logs
sonda metrics list/query <svc>, sonda signal <svc> <name>	Metrics
sonda traces search/query <svc>, sonda trace <svc> <id>	Traces
sonda workloads <svc>	Kubernetes pods, rollouts and events
sonda alerts <svc>	Active alerts of Prometheus or Grafana alerting rules

The CLI prints JSON when its output is piped and compact text in a terminal.

Configuration

Sources, projects with their services, and server limits are in sonda.yaml (see sonda.example.yaml). A project groups the services of one product, across platforms. The server reads it at startup. Agent identities are in the database and are managed in the admin UI. The parts that matter for security:

• Bindings are the scoping boundary. Every query of the service is restricted to them:
  • Loki and Prometheus selector: must contain an = or =~ matcher that rejects the empty value, and the labels must really identify the service in your data.
  • Cloudflare script: the Worker script name. Metrics bindings can also map the service's Durable Object namespaces, D1 databases, queues and R2 buckets by name to their IDs (durable_object_namespaces, d1_databases, queues, r2_buckets).
  • CloudWatch log_groups: 1 to 50 exact log group names, no patterns or ARNs.
  • CloudWatch metrics namespace and dimensions: 1 to 10 dimensions.
  • Tempo service: the resource.service.name of the service's spans.
  • Sentry errors project and environment, and optionally query: Sentry search terms ANDed with every issue and event search, for services that share a project, e.g. server_name:engine-e0-*. Issue counts then cover only matching events in the range; the totals of a single issue and the releases stay project-wide.
  • Sentry traces project and environment, as for errors. Sentry has no project-scoped span search, so Sonda checks the project and environment of every span it returns.
  • Kubernetes namespace and labels: exact labels the service's pods and ReplicaSets carry.
  • Vercel project and environment: the project ID (prj_…, not the name) and the Vercel environment, e.g. production or preview.
  • Alerts selector: a Prometheus selector over the alert labels, with the same rule as above. The alerts API returns the alerts of every service, so this filter in Sonda is the scope.
• Grants give an identity capabilities (errors, logs, metrics, traces, workloads, alerts) in project environments, e.g. logs in production of project engine. A grant covers every service of the project that configures the capability in that environment, including services added to the project later. Anything else is not found. A traces grant also decides which spans of a trace an identity sees in full; spans of other services are reduced to service, timing and status.
• Provider credentials come from environment variables or, for AWS, the default credential chain or Sonda's own identity (below). Use read-only tokens and roles; the Security section of docs/design.md lists the Cloudflare permission and the AWS IAM actions.
• Admin token (admin.token) can create tokens for every project. Keep it secret and expose /admin only on a trusted network.

Sources and bindings for Cloudflare, AWS, Tempo, Kubernetes, Grafana alerting and Vercel:

sources:
  cloudflare:
    type: cloudflare
    account_id: "${CF_ACCOUNT_ID}"
    auth: { type: bearer, token: "${CF_API_TOKEN}" }  # url defaults to https://api.cloudflare.com/client/v4
  aws-prod:
    type: aws
    region: eu-west-1
    role_arn: "${SONDA_AWS_ROLE_ARN}"  # optional; the default credential chain otherwise
    max_scan_bytes: 10GB               # Logs Insights is billed by scanned bytes
  tempo-prod:
    type: tempo
    url: http://tempo.monitoring.svc:3200
    headers: { X-Scope-OrgID: prod }   # multi-tenant Tempo only
    grafana_datasource_uid: tempo-prod
  k8s-prod:
    type: kubernetes                   # without url: the pod's service account (in-cluster)
    url: https://ABC123.gr7.eu-west-1.eks.amazonaws.com
    auth: { type: bearer, token: "${K8S_TOKEN}" }  # a read-only service account token
    ca_file: /etc/sonda/k8s-prod-ca.crt
  grafana:
    type: grafana                      # Grafana-managed alerts; a prometheus source serves its own rules' alerts
    url: https://grafana.example.com
    auth: { type: bearer, token: "${GRAFANA_TOKEN}" }  # a Viewer service account token
  vercel:
    type: vercel
    team_id: team_abc123               # the team ID, not the slug
    auth: { type: bearer, token: "${VERCEL_TOKEN}" }  # url defaults to https://api.vercel.com

projects:
  web:
    services:
      web-worker:
        environments:
          production:
            logs:    { source: cloudflare, script: web-worker }
            metrics: { source: cloudflare, script: web-worker, d1_databases: { main: 6f1c2b9e-3d4a-4e5f-8a7b-0c1d2e3f4a5b } }
            traces:  { source: cloudflare, script: web-worker }
        signals:
          errors: { metric: errors, by: [status] }  # Cloudflare signals are mappings
          slow_queries: { metric: d1_query_time_p99 }
  engine:
    description: Content engine
    services:
      engine-db:
        environments:
          production:
            logs:    { source: aws-prod, log_groups: [/aws/rds/cluster/engine/postgresql] }
            metrics: { source: aws-prod, namespace: AWS/RDS, dimensions: { DBClusterIdentifier: engine } }
        signals:
          cpu: { metric: CPUUtilization, stat: Average }
      engine-api:
        environments:
          production:
            traces: { source: tempo-prod, service: engine-api }  # resource.service.name
            workloads: { source: k8s-prod, namespace: engine, labels: { app.kubernetes.io/name: engine-api } }
            alerts: { source: grafana, selector: '{service="engine-api"}' }
  site:
    services:
      site-frontend:
        environments:
          production:
            logs: { source: vercel, project: prj_abc123, environment: production }
          preview:
            logs: { source: vercel, project: prj_abc123, environment: preview }

A signal is PromQL for Prometheus and a mapping for Cloudflare or CloudWatch. All metrics environments of a service with signals must use the same source type.

AWS without access keys, and backends in a private EKS cluster

Sonda can be an OIDC issuer of its own identity, so AWS trusts it wherever it runs, without a long-lived access key and without the hosting platform's workload identity. The same AWS credentials sign EKS tokens, which reach a Loki, Prometheus or Tempo inside a cluster through the API server's service proxy; nothing new runs in the private network.

identity:
  issuer: https://sonda.example.com     # Sonda's public URL; serves /.well-known/openid-configuration

sources:
  aws-prod:
    type: aws
    region: eu-west-1
    credentials: identity               # AssumeRoleWithWebIdentity with Sonda's own token
    role_arn: arn:aws:iam::123456789012:role/sonda
  loki-prod:
    type: loki
    url: https://ABC123.gr7.eu-west-1.eks.amazonaws.com/api/v1/namespaces/logging/services/http:loki:3100/proxy
    ca_file: /etc/sonda/eks-prod-ca.crt # certificateAuthority.data of the cluster, decoded
    auth: { type: eks, cluster: prod, aws_source: aws-prod }

• In AWS: an IAM OIDC identity provider for the issuer URL with audience sts.amazonaws.com, and a role that trusts it with conditions on aud (sts.amazonaws.com) and sub (sonda).
• In the cluster: an access entry for the role in a Kubernetes group, and a Role in the backend's namespace with get on services/proxy for resourceNames: ["http:loki:3100"] (get and create for Prometheus).
• The signing key lives in Sonda's database and the issuer URL is the trust anchor; see Security in docs/design.md.

Admin UI

Set admin.token to enable the UI at /admin. Without it, the UI is not served.

• Identities: create an identity, select its grants in a project environment × capability matrix, override limits, rotate or revoke its token. Changes apply to the next request; no restart. A token is shown once; Sonda stores only its SHA-256.
• Audit log: every agent request and every admin action, filterable by identity.

The UI is server-rendered with htmx and embedded in sonda-server.

HTTP API

The CLI is a thin client over the API in api/. All endpoints need Authorization: Bearer <token>.

Endpoint	Capability
GET /v1/services, GET /v1/services/{s}	any grant
GET /v1/services/{s}/overview	any granted
GET /v1/services/{s}/errors, /errors/{id}, /releases	errors
POST /v1/services/{s}/logs/search, /logs/query; GET /logs/fields	logs
GET /v1/services/{s}/metrics, /signals/{name}; POST /metrics/query	metrics
POST /v1/services/{s}/traces/search, /traces/query; GET /traces/{id}	traces
GET /v1/results/{id}	the identity that created it

Time ranges: env, since (30m, 6h, 2d) or from/to (RFC 3339) as query parameters, or the same fields in POST bodies.

Development

go test ./...
docker compose -f dev/docker-compose.yml up -d   # Loki :13100, Prometheus :19090, Tempo :13200 (OTLP :14318), Postgres :15432
SONDA_TEST_LOKI_URL=http://localhost:13100 go test ./internal/logql   # LogQL scoping against a real Loki
dev/seed-traces.sh                               # traces of two services in the local Tempo
SONDA_TEST_TEMPO_URL=http://localhost:13200 go test ./internal/traceql   # TraceQL scoping against a real Tempo
SONDA_TEST_POSTGRES_URL=postgres://sonda:sonda@localhost:15432/sonda go test ./internal/store   # store against Postgres

The design and its trade-offs are in docs/design.md.

License

MIT