Monitor API¶

Monitor Momentry's job queue, worker status, and processing pipeline in real-time.

Overview¶

Monitor API provides endpoints for: - Queue Statistics: Job counts by status - Running Jobs: Detailed view of active jobs with processor progress - Worker Status: System resources and worker process health

Endpoints¶

`GET /api/v1/monitor/queue_stats`¶

Auth: Required

Scope: system-level

Get job queue statistics grouped by status.

Request Parameters¶

None

Example¶

# Get queue statistics
curl -s "$API/api/v1/monitor/queue_stats" \
  -H "X-API-Key: $KEY"

Response (200)¶

{
  "pending": 5,
  "running": 2,
  "completed": 150,
  "failed": 3
}

Field	Type	Description
`pending`	number	Jobs waiting to be processed
`running`	number	Jobs currently being processed
`completed`	number	Jobs successfully completed
`failed`	number	Jobs that failed

Implementation Details¶

File: /Users/accusys/momentry_core/src/api/monitor.rs

Database Query:

SELECT status, COUNT(*) as count
FROM monitor_jobs
GROUP BY status

Status Values: - "pending" - Queued for processing - "running" - Currently being processed - "completed" - Successfully finished - "failed" - Processing failed

`GET /api/v1/monitor/running_jobs`¶

Auth: Required

Scope: system-level

Get detailed information about currently running jobs, including processor progress.

Request Parameters¶

None

Example¶

# Get running jobs
curl -s "$API/api/v1/monitor/running_jobs" \
  -H "X-API-Key: $KEY"

Response (200)¶

{
  "jobs": [
    {
      "job_id": 123,
      "file_uuid": "384b0ff44aaaa1f1",
      "file_name": "video.mp4",
      "status": "running",
      "started_at": "2026-10-24T15:30:00Z",
      "duration_secs": 120,
      "processors": [
        {
          "name": "asr",
          "phase": 2,
          "status": "running",
          "progress": 75,
          "duration": 45.3
        },
        {
          "name": "face",
          "phase": 2,
          "status": "pending",
          "progress": 0
        }
      ]
    }
  ]
}

Field	Type	Description
`jobs`	array	List of running jobs
`jobs[].job_id`	number	Job ID
`jobs[].file_uuid`	string	File UUID
`jobs[].file_name`	string	File name
`jobs[].status`	string	Job status (`"running"`)
`jobs[].started_at`	string	ISO 8601 timestamp
`jobs[].duration_secs`	number	Duration in seconds
`jobs[].processors`	array	List of processors for this job
`jobs[].processors[].name`	string	Processor type
`jobs[].processors[].phase`	number	Processing phase (1-4)
`jobs[].processors[].status`	string	Processor status
`jobs[].processors[].progress`	number	Progress percentage (0-100)
`jobs[].processors[].duration`	number	Duration in seconds (optional)

Implementation Details¶

File: /Users/accusys/momentry_core/src/api/monitor.rs

Database Query:

SELECT j.id, j.uuid, j.status, j.started_at, v.file_name
FROM monitor_jobs j
LEFT JOIN videos v ON v.file_uuid = j.uuid
WHERE j.status = 'running'
ORDER BY j.started_at DESC

Processor Phases: | Phase | Processors | Description | |-------|-----------|-------------| | 1 | cut | Video segmentation | | 2 | asr, asr_asrx, face, ocr | Core processing | | 3 | scene, face_trace, face_cluster | Analysis & clustering | | 4 | tkg | Knowledge graph |

`GET /api/v1/monitor/worker_status`¶

Auth: Required

Scope: system-level

Get worker process status and system resource usage.

Request Parameters¶

None

Example¶

# Get worker status
curl -s "$API/api/v1/monitor/worker_status" \
  -H "X-API-Key: $KEY"

Response (200)¶

{
  "cpu_percent": 45.5,
  "gpu_percent": 60.0,
  "memory_mb": 8192,
  "memory_percent": 52.3,
  "active_processors": 3,
  "max_processors": 6,
  "processors": [
    {
      "name": "asr",
      "job_id": 123,
      "status": "running"
    },
    {
      "name": "face",
      "job_id": 124,
      "status": "running"
    }
  ],
  "worker_pid": 12345,
  "uptime_secs": 3600,
  "last_poll_at": "2026-10-24T15:30:00Z",
  "poll_interval_secs": 5,
  "is_alive": true
}

Field	Type	Description
`cpu_percent`	number	CPU usage percentage
`gpu_percent`	number	GPU utilization percentage
`memory_mb`	number	Memory used in MB
`memory_percent`	number	Memory usage percentage
`active_processors`	number	Number of active processors
`max_processors`	number	Maximum concurrent processors
`processors`	array	List of active processors
`processors[].name`	string	Processor type
`processors[].job_id`	number	Job ID being processed
`processors[].status`	string	Processor status
`worker_pid`	number	Worker process ID
`uptime_secs`	number	Worker uptime in seconds
`last_poll_at`	string	Last activity timestamp (optional)
`poll_interval_secs`	number	Expected poll interval
`is_alive`	boolean	Worker process is running

Implementation Details¶

File: /Users/accusys/momentry_core/src/api/monitor.rs

Data Sources: - System resources: SystemResources::check() - Worker PID: pgrep -f "momentry worker" - Active processors: processor_results table - Last activity: monitor_jobs table

Resource Monitoring: - CPU: System-wide usage - GPU: Metal/ANE utilization (macOS) - Memory: Used vs total memory

Usage Patterns¶

Dashboard Monitoring¶

# Get all monitoring data
curl -s "$API/api/v1/monitor/queue_stats" -H "X-API-Key: $KEY" | jq
curl -s "$API/api/v1/monitor/running_jobs" -H "X-API-Key: $KEY" | jq
curl -s "$API/api/v1/monitor/worker_status" -H "X-API-Key: $KEY" | jq

Real-time Updates¶

# Poll every 5 seconds
watch -n 5 'curl -s "$API/api/v1/monitor/queue_stats" -H "X-API-Key: $KEY" | jq'

Health Check¶

# Check if worker is alive
curl -s "$API/api/v1/monitor/worker_status" -H "X-API-Key: $KEY" | jq '.is_alive'

Error Codes¶

Code	HTTP	When
E401	401	Missing or invalid API key
E500	500	Database query failed

Database Tables¶

`monitor_jobs`¶

Primary job queue table.

Column	Type	Description
`id`	integer	Job ID
`uuid`	string	File UUID
`status`	string	Job status
`started_at`	timestamp	Start time
`updated_at`	timestamp	Last update

`processor_results`¶

Processor execution results.

Column	Type	Description
`job_id`	integer	Job ID
`processor_type`	string	Processor name
`status`	string	Processor status
`progress_current`	integer	Current progress
`progress_total`	integer	Total steps
`duration_secs`	float	Duration in seconds

Common Issues¶

Worker Shows as Not Alive¶

Symptoms: is_alive: false

Possible Causes: 1. Worker process not running 2. Worker crashed 3. pgrep command failed

Solution:

# Check worker process
ps aux | grep momentry

# Start worker
momentry worker

# Check logs
tail -f ~/momentry_core/logs/worker_3002.log

Queue Has Stuck Jobs¶

Symptoms: Jobs stuck in "running" status

Possible Causes: 1. Worker crashed 2. Processor timeout 3. Database lock

Solution:

# Check running jobs
curl -s "$API/api/v1/monitor/running_jobs" -H "X-API-Key: $KEY"

# Reset stuck jobs (requires database access)
# UPDATE monitor_jobs SET status = 'pending' WHERE status = 'running' AND updated_at < NOW() - INTERVAL '1 hour';

High Resource Usage¶

Symptoms: cpu_percent or memory_percent very high

Possible Causes: 1. Too many concurrent jobs 2. Large video files 3. Memory leak

Solution:

# Check active processors
curl -s "$API/api/v1/monitor/worker_status" -H "X-API-Key: $KEY" | jq '.processors'

# Reduce max_processors if needed
# Restart worker

Version History¶

Version	Date	Changes
1.0	2026-10-24	Initial documentation

Monitor API¶

Overview¶

Endpoints¶

GET /api/v1/monitor/queue_stats¶

Request Parameters¶

Example¶

Response (200)¶

Implementation Details¶

GET /api/v1/monitor/running_jobs¶

Request Parameters¶

Example¶

Response (200)¶

Implementation Details¶

GET /api/v1/monitor/worker_status¶

Request Parameters¶

Example¶

Response (200)¶

Implementation Details¶

Usage Patterns¶

Dashboard Monitoring¶

Real-time Updates¶

Health Check¶

Error Codes¶

Database Tables¶

monitor_jobs¶

processor_results¶

Related Documentation¶

Common Issues¶

Worker Shows as Not Alive¶

Queue Has Stuck Jobs¶

High Resource Usage¶

Version History¶

`GET /api/v1/monitor/queue_stats`¶

`GET /api/v1/monitor/running_jobs`¶

`GET /api/v1/monitor/worker_status`¶

`monitor_jobs`¶

`processor_results`¶