Tool Usage Analytics from Logs

Tinker

Home Epics Tool Usage Analytics from LogsEdit

Cancel

Title *

Project *

Description

## Problem

We have no visibility into what tools agents actually use. Are they using memory tools? Which MCP tools are never called? Which tools cause errors?

**The data already exists** in agent-bridge logs, but it's not queryable.

## Solution

Create a scheduled job that:
1. Parses agent-bridge logs for tool invocations
2. Extracts: agent_id, tool_name, timestamp, success/failure, error if any
3. Populates `tool_usage_stats` table
4. Dashboard queries the aggregated stats

### Database Schema

```ruby
create_table :tool_usage_stats do |t|
  t.references :agent, null: false
  t.string :tool_name, null: false
  t.string :mcp_server, null: false  # e.g., "tinker-worker", "tinker-reviewer"
  t.integer :call_count, default: 0, null: false
  t.integer :success_count, default: 0, null: false
  t.integer :error_count, default: 0, null: false
  t.timestamp :last_called_at
  t.timestamp :last_error_at
  t.text :last_error_message
  t.date :tracked_date, null: false  # Daily granularity
  
  t.timestamps
  
  t.index [:agent_id, :tool_name, :tracked_date], unique: true
  t.index :tool_name
  t.index :mcp_server
end
```

### Log Parsing Job

**ToolUsageAggregationJob** (runs hourly):
```ruby
# Parse agent-bridge logs since last run
# Extract MCP tool calls from log lines
# Upsert into tool_usage_stats
# Track daily stats (tracked_date)
```

### Log Format to Parse

Agent-bridge logs show:
```
[2025-12-25 03:45:12] [tinker-worker] Calling MCP tool: search_memory
[2025-12-25 03:45:13] [tinker-worker] Tool result: success
[2025-12-25 03:45:14] [tinker-worker] Calling MCP tool: store_memory
[2025-12-25 03:45:15] [tinker-worker] Tool error: validation_failed
```

### Dashboard Endpoints

```
GET /admin/tool_usage
- Table: tool_name | calls_today | calls_total | error_rate | last_called
- Filter by agent, MCP server, date range
- Sort by call count, error rate

GET /admin/tool_usage/:tool_name
- Detail view: timeline of usage, error messages
- Which agents use it most
```

## Acceptance Criteria

1. `tool_usage_stats` table created with indexes
2. `ToolUsageAggregationJob` parses logs and upserts stats
3. Job runs hourly via Solid Queue
4. Admin dashboard at `/admin/tool_usage`
5. Can see: most-used tools, unused tools, high-error tools
6. Data helps identify agents not using memory tools