Log Insights
Log Insights captures Veda's own logs, correlates them with traces, categorizes them semantically, alerts on log content through the Ops Advisor, and can store external logs as a searchable collection.
It is deliberately not a full logs platform (no ingest agents, ECS normalization, or ILM tiers — that's outside a vector database's lane). Instead it builds the pieces that reuse Veda's strengths — the embedder, BM25, the filter DSL, TTL — and adds the one thing a vector DB can do that template-based log tools can't: semantic log categorization that groups differently-worded messages about the same underlying failure.
Log capture & trace correlation
A tracing layer captures every event at or above VEDA_LOG_CAPTURE_LEVEL (default WARN) into a bounded in-memory ring buffer. Each record is:
{
"ts_ms": 1718409600000,
"level": "WARN",
"target": "veda_gateway::notify",
"message": "send failed",
"fields": { "channel": "webhook" },
"trace_id": "…",
"span_id": "…"
}trace_id / span_id are read best-effort from the active OpenTelemetry span — populated when an OTLP endpoint is configured (see Observability), null otherwise. The per-query log carries a trace_id too, so logs and traces line up. Capture is synchronous, non-blocking, and panic-free; the same buffer feeds the REST API, the categorizer, and the Ops Advisor.
# recent captured logs, with level + substring filters
curl "http://localhost:6340/v1/admin/logs?level=WARN&q=send&limit=100"Semantic categorization
GET /v1/admin/logs/categories pulls recent WARN/ERROR messages, embeds each distinct message (cache-aware — only new messages embed), and greedily clusters them by cosine similarity:
{
"id": "…",
"representative": "send failed: connection refused",
"count": 7,
"max_severity": "ERROR",
"examples": ["send failed: connection refused", "delivery failed — host unreachable"],
"first_seen_ms": 1718409600000,
"last_seen_ms": 1718409930000
}This groups semantically similar errors, not just identical templates. When no text embedder is configured, it falls back to grouping by a normalized template (numbers, ids, and quoted strings collapsed to placeholders).
Log-content alerting
The Ops Advisor snapshot gained error_log_count / warn_log_count (over a 5-minute window), powering three rules that open events and flow through the existing alert channels (webhook, Slack, PagerDuty, email):
| Rule | Severity | Fires when |
|---|---|---|
log_error_rate | High | ERROR-log count over the window exceeds a threshold |
anomaly_error_logs | Med/High | error-log rate spikes above its EWMA/σ baseline |
new_error_category | Medium | a newly-appeared WARN/ERROR semantic category is detected |
All three are tunable, mutable, and auto-remediable like every other rule via /v1/admin/insights/rules — see Ops Advisor.
Slow queries
GET /v1/admin/slow-queries?collection=&limit= returns the slowest recent queries, server-backed from each collection's query log (tenant-scoped):
curl "http://localhost:6340/v1/admin/slow-queries?collection=products&limit=20"Logs as a searchable collection
Ship external logs into a collection and get a searchable archive — BM25 full-text and semantic similarity, plus the filter DSL, TTL retention, and analytics aggregations:
curl -X POST http://localhost:6340/v1/collections/applogs/ingest-logs \
-H "Content-Type: application/json" \
-d '{
"lines": [
"{\"ts\":\"2026-06-15T10:00:00Z\",\"level\":\"error\",\"message\":\"disk full on node 3\"}",
"2026-06-15T10:00:01Z INFO request completed in 12ms"
],
"default_level": "INFO",
"ttl_secs": 604800
}'Each line is parsed and inserted via the standard insert / embed-insert path:
- JSON line —
message(frommessage/msg/body),level(level/severity/lvl, upper-cased),ts(RFC3339 or epoch); remaining scalar keys become fields. - Plain line — a leading RFC3339 timestamp + level token are parsed off, the rest is the message; otherwise the whole line is the message with
default_leveland the current time. ttl_secssets_expires_atfor retention (see Record TTL).
When the collection has a vector field with auto_embed over message, ingested messages are embedded for semantic log search; otherwise it's a plain insert. Same auth and tenant scoping as /insert.
Recommended schema for a logs collection:
{
"name": "applogs",
"fields": [
{"name": "message", "type": "string", "bm25": true},
{"name": "level", "type": "string", "filterable": true},
{"name": "ts", "type": "int64", "filterable": true},
{"name": "embedding", "type": "vector", "dimension": 1024, "metric": "cosine",
"model": "BAAI/bge-m3", "auto_embed": {"source_field": "message"}}
]
}In Niriksha
The dashboard's Logs page gives a live tail (with level filter and text search), a trace_id column, the semantic-categories panel, the slow-query table, and a box to search a logs collection — plus nav and command-palette entries.
Configuration
| Variable | Purpose |
|---|---|
VEDA_LOG_CAPTURE_LEVEL | Capture threshold — ERROR / WARN / INFO / DEBUG / TRACE (default WARN) |
VEDA_ADVISOR_INTERVAL_SECS | Advisor tick — drives the log rules and categorization cadence (default 30s) |
OTEL_EXPORTER_OTLP_ENDPOINT | When set, log records carry trace_id / span_id |
EMBEDDING_* | Enables semantic categorization and semantic log search; without it, categorization falls back to normalized templates |
Current limitations
trace_idisnullunless an OTLP endpoint is configured (best-effort by design).- The capture buffer is in-memory and bounded (a recent window only) — for long-term history, ingest into a logs collection or use the external Loki stack.
- This is not a full log platform: no ingest agents, ECS normalization, ILM tiers, or a Logs Explorer — by design.