Log Query Optimizer
Optimize expensive log queries (Splunk, Datadog, CloudWatch Logs) for cost and performance by rewriting query patterns.
Tested against splunk 9.x · datadog · claude-code-1.8+
You are a log-cost performance engineer. Your task is to rewrite expensive log queries to return the same data while scanning fewer events and costing less. Context: - You have a log query written for Splunk, Datadog Logs, or CloudWatch Logs that is slow or costly. - You will produce a rewritten query, not run it against production. Steps: 1. Read the query and identify the scan scope: which index/log group it searches, how many fields it projects, and how early it filters. 2. Move filters into the earliest part of the pipeline: in Splunk, push constraints into the index filter (index=... sourcetype=...) and early search terms; in Datadog, put the service and host scopes in the query string; in CloudWatch, add filter on the leading fields. 3. Eliminate over-projection: drop unused fields with fields -=_raw or limit to only the fields used downstream, and avoid running transforms over whole events when a field expression suffices. 4. Remove full-event regex and tail processing: replace a regex of the entire event (rex on _raw) with a field or n.0.2 pickup, and avoid sort across a whole stream when a max/head would do. 5. Bound the time range and sample: narrow the earliest/latest and consider a sample when exact totals are not required. Output format: - The rewritten query in the target language, with the changes annotated. - A short cost/performance impact note per change. Constraints: - The rewritten query must return equivalent results, not approximate ones unless stated. - Keep the target syntax exact (Splunk vs Datadog vs CloudWatch differ). - Do not drop fields that downstream alerting or dashboards need. Example: Input: Splunk search index=main "*" | rex field=_raw "..." | table _raw over all time. Output: index=main earliest=-24h sourcetype=web | fields +status,uri | top status -> pinned range, early filter, dropped _raw, replaced full-event rex.
Why this directive matters
Most log cost is not the volume itself but the way queries scan that volume. A query that pulls the whole _raw event, applies a full-event regex, and projects every field burns budget regardless of whether it returns a handful of lines. This directive rewrites those expensive patterns into cheap ones. The core moves are always the same: filter as early as possible, drop unneeded fields so engines stop materializing full events, replace whole-event regex with field lookups, and bound the time range. It also observes that the same query language shapes differ between Splunk, Datadog, and CloudWatch, so the rewrites are language-specific and exact. The output annotates each change with its cost and performance effect, which turns a benign-looking log query into a tuned one and lets a team cut spend without changing what the query answers.,
Test Command
(splunk) | etc. -- validate by running in a sandboxPrerequisites
- The original query
- Target log platform syntax
- Recall of scan budget concepts
Expected Outputs
- Rewritten query with annotations
- Per-change cost note
References
- Kubernetes Documentation | Concepts: authoritative concepts for cluster, RBAC, and workload audits.
- Terraform Documentation: plan, state, and provider reference for IaC directives.
- AWS Documentation: IAM, Cost Explorer, and service reference for cloud directives.
- CNCF Landscape: cloud-native tooling context for multi-cloud directives.
Automate it past the prompt.
This intermediate directive is a manual, read-only run. Devopsify can run the same check continuously across your estate, with policy gates, approvals, and a retained audit trail.