Operate agents with defined controls for
evaluation, access, incident response, and change.
Production agents can access data, call tools, and affect downstream workflows. We establish release criteria, permissions, traceability requirements, incident procedures, and lifecycle controls. When ongoing support is required, we define the managed-operations scope separately, including the covered assets, service levels, and responsibilities.
Proof-of-concept accuracy alone does not establish operational readiness
Once an agent is part of a workflow, answer quality is only one aspect of operational readiness. Its access to data and tools, the actions it may take, the evidence it leaves behind, and the response to errors and changes all require defined controls.
Quality & Reliability
Evaluate correctness, completeness, consistency, and repeatability across representative workflows and known exceptions.
Access & Execution Control
Define who may use the agent, which data and tools it may access, which actions are blocked, and where review or approval is required.
Traceability & Audit
Capture inputs, source references, model responses, tool calls, execution outcomes, and user actions so executions within the agreed scope can be reviewed.
Operational Continuity
Define monitoring criteria for latency, usage, cost, errors, and incidents, as well as change, workaround, restoration, and return-to-service procedures.
Distinguish the control foundation from the work performed after launch
AgentOps defines and implements the controls needed to operate agents. Managed Operations is a separate ongoing service provided within the agreed coverage, service levels, and responsibilities.
AgentOps Foundation
Establish evaluation sets and release gates, configure trace capture and dashboards, define permission and guardrail policies, and create incident and change procedures and operating runbooks.
Managed Operations
Provide monitoring, scheduled quality reviews, incident and change support, cost and performance reviews, and scheduled service reports for the agreed scope.
Manage evaluation, runtime controls, observability, incidents, and change in one framework
Evaluation & Release
Define evaluation sets, metrics, and release criteria, and run regression tests before and after changes.
Runtime Observability
Capture execution traces, response-quality indicators, tool calls, latency, errors, and usage patterns according to the agreed monitoring scope.
Access & Guardrails
Set permissions by data, model, and tool; define prohibited actions, escalation conditions, and workflows that require human review or approval.
Cost & Performance
Review usage, processing time, and cost by model and tool, then evaluate tradeoffs among quality, latency, and cost.
Incident & Continuity
Define incident-severity criteria and procedures for notification, impact assessment, workarounds, restoration, and return to service.
Change & Lifecycle
Manage versions and approvals for models, prompts, skills, knowledge, and tools, with release and rollback criteria where applicable.
Find the failure before choosing the fix
An incorrect number may originate in source data, a metric definition, retrieval, or calculation. Scoring the final answer rarely reveals which part needs to change. Operational evaluation follows the evidence through execution and tests the proposed fix.
Retest work that already succeeded
Fixing a failed question can change a previously correct result. Include routine workflows and exceptions in regression tests, alongside held-out cases not used during development. Fix the comparison versions, data cut-off, and expected results so the effect of a change is visible.
Assess calculations and explanations differently
Code and rules check totals, formulas, input formats, and permissions. Expert review provides the reference for assessing explanations; LLM scoring can assist. Choose each evaluator for the type of error it needs to detect.
Look upstream of the final response
Execution traces show retrieved documents, rule and model versions, and tool inputs and results. They help distinguish missing knowledge from retrieval, calculation, and execution failures. Record only what is needed, with sensitive-data masking and access controls.
Review gains and regressions together
Compare versions under the same conditions, considering correctness, serious failures, response time, and cost. Introduce approved versions to a limited scope, with stopping and recovery criteria for problems encountered in operation.
Set retry rules for the actual task
A failed search may be safe to repeat. Repeating an order submission can create a duplicate transaction. Define retries, time and cost limits by task, and prevent duplicate external effects. Escalate when limits are reached or evidence is insufficient.
Update the knowledge or procedure that failed
A missing business rule may call for a knowledge update; an incorrect analytical sequence may require a skill or tool change. Review evidence and impact, then test and approve before release. Changes to model weights are a separate decision.
Operating records provide evidence for improvement. Approved knowledge and skill changes are distinct from model training, and changes to evaluation criteria also need a history. Development methodology →
Use operating evidence to evaluate and approve each release
Select the operating services required for the agreed assets and responsibilities
Monitoring & Alerts
Monitor availability, execution state, errors, latency, and agreed operating indicators, and route alerts through the defined escalation process.
Evaluation & Quality Review
Run scheduled evaluations and sample reviews to identify changes in quality and track agreed follow-up actions.
Incident & Recovery Support
Receive and classify incidents, assess impact, coordinate workaround and restoration, and track follow-up actions intended to reduce recurrence.
Release & Change Management
Review change requests and assess their impact, coordinate testing and approval, and maintain version, release, and rollback records.
Cost & Performance Review
Analyze model and tool usage, processing performance, and cost, then recommend changes within the service scope.
Knowledge & Skill Updates
Validate approved changes to business rules, exceptions, knowledge, and agent skills before publishing them to the operating environment.
Define coverage, response commitments, change authority, and reporting before service begins
Where service-level commitments are included, we first review business criticality and support responsibilities. We then agree on the assets in scope, service hours, incident-severity definitions, response and recovery targets, change procedures, and reporting cadence.
Service Coverage
Specify the agents, applications, data, models, tools, and operating environments in scope, together with service hours and exclusions.
Severity & Response
Define incident severity according to business impact, together with notification paths, initial-response targets, workaround criteria, and recovery targets.
Change & Release Windows
Define approval paths and windows for standard and emergency changes, with validation and rollback conditions.
Reporting & Review
Agree on the quality, usage, cost, incident, and change information included in scheduled service reports and reviews.
Start with the agents being prepared for release or already in operation
Tell us about the workflows and users, connected data and tools, permitted actions, and current evaluation, monitoring, incident, and change procedures. We will help define the initial scope for an AgentOps foundation or a managed-operations service.
Discuss AgentOps & Managed Operations →