VLM Search API¶

Search video scenes and traces using Vision Language Model (VLM) descriptions and tags.

Overview¶

VLM Search API provides endpoints for: - Scene Search: Search scenes by VLM-generated descriptions and objects - Trace Search: Search face traces by VLM descriptions - Debug Search: Debug VLM search functionality

Endpoints¶

`POST /api/v1/search/vlm/scenes`¶

Auth: Required

Scope: file-level or global

Search scenes using VLM-generated content and metadata.

Request Parameters¶

Field	Type	Required	Default	Description
`query`	string	Yes	—	Search query text
`file_uuid`	string	No	—	Limit to specific file
`filters`	object	No	—	Search filters
`page`	number	No	1	Page number
`page_size`	number	No	20	Results per page
`min_score`	number	No	0.0	Minimum relevance score

Filters Object¶

Field	Type	Description
`vlm_objects`	array	Filter by detected objects
`vlm_tags`	array	Filter by VLM tags
`time_range`	object	Filter by time range

Example¶

# Search scenes for "dog playing"
curl -s -X POST "$API/api/v1/search/vlm/scenes" \
  -H "X-API-Key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "dog playing in the park",
    "page": 1,
    "page_size": 20,
    "min_score": 0.5
  }'

# Search scenes with filters
curl -s -X POST "$API/api/v1/search/vlm/scenes" \
  -H "X-API-Key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "outdoor scene",
    "file_uuid": "384b0ff44aaaa1f1",
    "filters": {
      "vlm_objects": ["dog", "tree"],
      "time_range": {
        "start_sec": 60.0,
        "end_sec": 120.0
      }
    }
  }'

Response (200)¶

{
  "results": [
    {
      "file_uuid": "384b0ff44aaaa1f1",
      "file_name": "video.mp4",
      "scene_index": 5,
      "timestamp_sec": 125.5,
      "frame_number": 3750,
      "thumbnail_url": "/api/v1/media/384b0ff44aaaa1f1/scene/5/thumbnail",
      "vlm_description": "A dog playing in a sunny park",
      "vlm_description_localized": "陽光下的公園裡有隻狗在玩耍",
      "vlm_objects": ["dog", "grass", "tree", "park"],
      "vlm_tags": [
        {"en": "dog", "local": "犬"},
        {"en": "outdoor", "local": "戶外"},
        {"en": "sunny", "local": "晴天"}
      ],
      "score": 0.92,
      "source": "vlm"
    }
  ],
  "total": 15,
  "page": 1,
  "page_size": 20
}

Field	Type	Description
`results`	array	Search results
`results[].file_uuid`	string	File UUID
`results[].file_name`	string	File name
`results[].scene_index`	number	Scene index
`results[].timestamp_sec`	number	Scene timestamp
`results[].frame_number`	number	Frame number
`results[].thumbnail_url`	string	Thumbnail URL
`results[].vlm_description`	string	VLM-generated description
`results[].vlm_description_localized`	string	Localized description (optional)
`results[].vlm_objects`	array	Detected objects
`results[].vlm_tags`	array	VLM tags (bilingual)
`results[].score`	number	Relevance score (0-1)
`results[].source`	string	Search source (`"vlm"`)
`total`	number	Total results
`page`	number	Current page
`page_size`	number	Page size

`POST /api/v1/search/vlm/traces`¶

Auth: Required

Scope: file-level or global

Search face traces using VLM descriptions.

Request Parameters¶

Field	Type	Required	Default	Description
`query`	string	Yes	—	Search query text
`file_uuid`	string	No	—	Limit to specific file
`filters`	object	No	—	Search filters
`page`	number	No	1	Page number
`page_size`	number	No	20	Results per page
`min_score`	number	No	0.0	Minimum relevance score

Example¶

# Search traces for "person wearing red shirt"
curl -s -X POST "$API/api/v1/search/vlm/traces" \
  -H "X-API-Key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "person wearing red shirt",
    "page": 1,
    "page_size": 20
  }'

Response (200)¶

{
  "results": [
    {
      "file_uuid": "384b0ff44aaaa1f1",
      "trace_id": 2,
      "identity_uuid": "a9a90105-...",
      "vlm_description": "A person wearing a red t-shirt and blue jeans",
      "vlm_tags": [
        {"en": "red shirt", "local": "紅色上衣"},
        {"en": "blue jeans", "local": "藍色牛仔褲"}
      ],
      "score": 0.88
    }
  ],
  "total": 3,
  "page": 1,
  "page_size": 20
}

`POST /api/v1/search/vlm/debug`¶

Auth: Required

Scope: system-level

Debug VLM search functionality (for development).

Example¶

# Debug VLM search
curl -s -X POST "$API/api/v1/search/vlm/debug" \
  -H "X-API-Key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "test query"
  }'

Implementation Details¶

File: /Users/accusys/momentry_core/src/api/vlm_search.rs (773 lines)

Data Structures: - SceneSearchRequest - Scene search parameters - TraceSearchRequest - Trace search parameters - VlmTagValue - Bilingual tag structure

VLM Tags Structure:

{
  "en": "dog",
  "local": "犬"
}

Search Process: 1. Parse query and filters 2. Generate embedding for query text 3. Search Qdrant for matching scenes/traces 4. Apply filters (objects, tags, time range) 5. Sort by relevance score 6. Return paginated results

VLM Integration¶

Description Generation¶

VLM descriptions are generated during file processing: - Scene VLM: Analyzes scene frames - Trace VLM: Analyzes face trace appearances

Tag Structure¶

Bilingual tags enable multilingual search: - English tag: Primary search key - Local tag: Native language display

Objects vs Tags¶

Objects: Detected physical objects (dog, car, tree)
Tags: Semantic concepts (outdoor, sunny, daytime)

Search Tips¶

Effective Queries¶

✅ Good: - "dog playing in the park" - "person wearing red shirt" - "outdoor sunny scene"

❌ Less Effective: - "video" (too generic) - "frame 100" (technical reference)

Filter Usage¶

Use filters to narrow results: - vlm_objects: Specific objects - vlm_tags: Specific concepts - time_range: Specific time period

Score Thresholds¶

> 0.8: Highly relevant
0.5-0.8: Moderately relevant
< 0.5: Low relevance

Error Codes¶

Code	HTTP	When
E401	401	Missing or invalid API key
E400	400	Invalid query or parameters
E500	500	Search failed

Common Issues¶

No Results Found¶

Symptoms: Empty results array

Possible Causes: 1. Query too specific 2. Low min_score threshold 3. No matching scenes/traces

Solution: - Simplify query - Lower min_score - Check file processing status

Low Relevance Scores¶

Symptoms: All results have low scores (< 0.5)

Possible Causes: 1. Query doesn't match content 2. VLM descriptions not generated 3. Embedding quality issue

Solution: - Rephrase query - Verify VLM processing completed - Check VLM model configuration

Version History¶

Version	Date	Changes
1.0	2026-10-24	Initial documentation

VLM Search API¶

Overview¶

Endpoints¶

POST /api/v1/search/vlm/scenes¶

Request Parameters¶

Filters Object¶

Example¶

Response (200)¶

POST /api/v1/search/vlm/traces¶

Request Parameters¶

Example¶

Response (200)¶

POST /api/v1/search/vlm/debug¶

Example¶

Implementation Details¶

VLM Integration¶

Description Generation¶

Tag Structure¶

Objects vs Tags¶

Search Tips¶

Effective Queries¶

Filter Usage¶

Score Thresholds¶

Error Codes¶

Related Documentation¶

Common Issues¶

No Results Found¶

Low Relevance Scores¶

Version History¶

`POST /api/v1/search/vlm/scenes`¶

`POST /api/v1/search/vlm/traces`¶

`POST /api/v1/search/vlm/debug`¶