Beyond Clicks: The New Analytics for AI Agents

Summary

Ever wonder how companies track the real impact of AI agents? It's not just about clicks anymore. A recent incident where an AI agent deleted a production database in seconds highlights a critical shift: the need for new product analytics. Traditional metrics like active users or session length are insufficient when the user is an AI. Instead, we need to analyze the 'agent run' – what work was attempted, what tools were used, if permissions failed, and if the user accepted the outcome. Salesforce is already moving in this direction, measuring 'agent work units' delivered. Key signals to track include user interruptions, denied approvals, and retries, as these indicate where the AI misunderstood or failed. High completion rates without user acceptance, for example, suggest an agent is finishing tasks users don't trust. To effectively shape AI agent behavior at their rapid pace, product teams must focus on events like agent run starts, task completions, and user interventions within a run, all tied to a unique agent run ID. This new layer of analytics, distinct from developer observability, is crucial for ensuring AI agents are useful and safe, preventing costly mistakes before they happen.