Skip to main content

Semantic Search

Semantic media search is an optional advanced feature. It connects Drone Media Library to an embedding provider so you can search media by meaning instead of only filename, title, tag, or note text.

Drone Media Library can use the Drone Media Library Embedding Provider as its embedding backend. The provider can run locally on your computer or on Modal.

Provider options​

Use local mode when you want everything to run on your own computer. This is the simplest option for testing and offline/local workflows, but the first model load can take a few minutes and performance depends on your machine.

Use Modal mode when you want the embedding provider to run remotely with GPU-backed compute. This is useful when your local computer is slow, when you want a more consistent indexing setup, or when multiple machines need to use the same provider endpoint.

Run the provider locally​

Clone or download the embedding provider project:

git clone https://github.com/reddotdrone/drone-media-library-embedding-provider.git
cd drone-media-library-embedding-provider

Set an API key. Use your own value instead of change-me.

export DRONE_MEDIA_LIBRARY_EMBEDDING_API_KEY="change-me"

Start the local provider:

uv run drone-media-library-embedding-provider serve --host 127.0.0.1 --port 8765

The local endpoint is:

http://127.0.0.1:8765

The health endpoint is public:

http://127.0.0.1:8765/v1/health

The model list and embedding endpoints require the bearer token API key.

Run the provider on Modal​

Install and configure Modal first, then create a Modal secret for the provider API key:

modal secret create drone-media-library-embedding-provider \
  DRONE_MEDIA_LIBRARY_EMBEDDING_API_KEY="change-me"

Serve temporarily during development:

uv run modal serve src/drone_media_library_embedding_provider/modal_app.py

Deploy permanently:

uv run modal deploy src/drone_media_library_embedding_provider/modal_app.py

After Modal starts or deploys the app, use the Modal web endpoint as the provider endpoint in Drone Media Library. The current provider app validates the same bearer token API key as local mode.

The provider code is configured to run the Modal app with a GPU-backed function. Check the provider repository before release if you need to confirm the exact GPU type for your deployment.

Supported models​

The provider currently exposes these model IDs:

openclip/xlm-roberta-base-ViT-B-32/laion5b_s13b_b90k
openclip/ViT-B-32/laion2b_s34b_b79k
mobileclip2/MobileCLIP2-S2/dfndr2b

Use one of these values in Settings as the selected model. If you are unsure, start with:

openclip/xlm-roberta-base-ViT-B-32/laion5b_s13b_b90k
  1. Open Settings.
  2. Turn on Semantic media search.
  3. Enter the provider endpoint.
    • Local example: http://127.0.0.1:8765
    • Modal example: use the Modal web endpoint shown by modal serve or modal deploy
  4. Enter the selected model.
  5. Enter the API key that matches DRONE_MEDIA_LIBRARY_EMBEDDING_API_KEY.
  6. Click Test provider.

If the provider is reachable, the Settings page shows a success message.

Provider API reference​

Drone Media Library uses the provider HTTP API:

  • GET /v1/health: public health check.
  • GET /v1/models: model list, bearer auth required.
  • POST /v1/embed/image: image embedding, bearer auth required.
  • POST /v1/embed/text: text embedding, bearer auth required.

If DRONE_MEDIA_LIBRARY_EMBEDDING_API_KEY is set on the provider, the app must send the same value as the API key in Settings.

Build the search index​

  1. Open Search Index.
  2. Click Index missing to index media that has not been indexed.
  3. Click Reindex all when you changed the provider, model, or want to rebuild existing results.

The first model load can take a few minutes. The Search Index page shows indexed, pending, failed, and not indexed media rows.

Search media​

After indexing, return to Library and search for the type of media you want to find. Semantic results can include both videos and photos.

Troubleshooting​

If the provider test fails, check that the provider is running and that the endpoint, model, and API key are correct.

If indexing says semantic search is disabled, turn on Semantic media search in Settings and try again.

If indexing fails for specific media, rebuild the index after confirming the original files still exist.