Create screenpipe.mdc

Author: louis030195

.cursor/rules/screenpipe.mdc

@@ -0,0 +1,210 @@
+---
+alwaysApply: false
+description: "Screenpipe SQLite: schema, search patterns, and best-practice queries for 24/7 screen + mic (OCR/STT) data"
+---
+
+### Screenpipe SQLite database: how to use it
+
+This rule explains what the Screenpipe database contains and exactly how to query it to serve the user. The DB is a 24/7 capture of screen and microphone activity with OCR and speech-to-text, optimized for full‑text search across modalities.
+
+- **Location**: `$HOME/.screenpipe/db.sqlite`
+- **Core idea**: Use FTS5 tables to search text quickly, then JOIN to base tables for context (timestamps, app/window, file paths). Filter by time first for speed.
+
+### High-level schema map
+
+- **Vision (screen video & frames)**
+  - `video_chunks(id, file_path, device_name)`
+  - `frames(id, video_chunk_id, offset_index, timestamp, name, device_name, browser_url, app_name, window_name, focused)`
+  - FTS + triggers: `frames_fts(id, name, browser_url, app_name, window_name, focused)`
+  - Tags: `vision_tags(vision_id=frames.id, tag_id)`
+
+- **OCR (text read from frames)**
+  - `ocr_text(frame_id, text, text_json, app_name, ocr_engine, window_name, focused, text_length)`
+  - FTS + triggers: `ocr_text_fts(text, app_name, window_name, frame_id)`
+
+- **Audio (microphone capture and transcripts)**
+  - `audio_chunks(id, file_path, timestamp)`
+  - `audio_transcriptions(id, audio_chunk_id, offset_index, timestamp, transcription, device, is_input_device, speaker_id, transcription_engine, start_time, end_time)`
+  - FTS + triggers: `audio_transcriptions_fts(transcription, device, audio_chunk_id, speaker_id, start_time, end_time)`
+  - Speakers & embeddings: `speakers(id, name, metadata, hallucination)`; `speaker_embeddings(id, embedding, speaker_id)`
+  - Tags: `audio_tags(audio_chunk_id, tag_id)`
+
+- **UI logs (optional)**
+  - `ui_monitoring(id, text_output, timestamp, app, window, initial_traversal_at, text_length)`
+  - FTS + triggers: `ui_monitoring_fts(text_output, app, window, ui_id)`
+  - Tags: `ui_monitoring_tags(ui_monitoring_id, tag_id)`
+
+- **Tags catalog**
+  - `tags(id, name, created_at)`
+
+- **Chunked text (optional internal index)**
+  - `chunked_text_index(text_id, text)` and related `*_fts` tables
+
+Notes:
+- Do not write directly to any `*_fts*` table; triggers keep FTS in sync with bases.
+- Timestamps are ISO‑8601 strings; prefer `julianday()` for time filtering.
+
+### Querying principles (fast + robust)
+
+1) Filter by time first, then text search, then JOIN context.
+   - Use `julianday(timestamp) >= julianday('now','-N hours')`.
+2) Use FTS5 for text; avoid `%LIKE%` on large text.
+3) JOIN back to base tables for metadata (app/window/url, file paths, speakers).
+4) Cap result size with `LIMIT`; rely on indexes (`timestamp`, `app_name`, `window_name`, `browser_url`, `text_length`).
+5) Never update FTS tables directly; insert/update the base table instead.
+
+### Recipes (drop-in queries)
+
+- Recent screen activity (frames) in last N hours
+
+```sql
+SELECT timestamp, app_name, window_name, browser_url
+FROM frames
+WHERE julianday(timestamp) >= julianday('now','-3 hours')
+ORDER BY timestamp DESC
+LIMIT 200;
+```
+
+- Search OCR text in last N hours (with screen context)
+
+```sql
+SELECT f.timestamp, f.app_name, f.window_name,
+       substr(replace(o.text, char(10), ' '),1,200) AS text_sample
+FROM ocr_text_fts ft
+JOIN ocr_text o ON o.frame_id = ft.frame_id
+JOIN frames f   ON f.id = o.frame_id
+WHERE julianday(f.timestamp) >= julianday('now','-3 hours')
+  AND ft MATCH 'invoice NEAR/3 total'
+ORDER BY f.timestamp DESC
+LIMIT 50;
+```
+
+- Search audio transcripts in last N hours
+
+```sql
+SELECT t.timestamp,
+       substr(replace(t.transcription, char(10), ' '),1,200) AS line,
+       t.speaker_id, t.device
+FROM audio_transcriptions_fts ft
+JOIN audio_transcriptions t ON t.audio_chunk_id = ft.audio_chunk_id
+WHERE julianday(t.timestamp) >= julianday('now','-3 hours')
+  AND ft MATCH 'plan OR decision OR blocker'
+ORDER BY t.timestamp DESC
+LIMIT 50;
+```
+
+- Join OCR to source video clip path
+
+```sql
+SELECT f.timestamp, v.file_path, f.offset_index,
+       substr(replace(o.text, char(10), ' '),1,200) AS text_sample
+FROM ocr_text o
+JOIN frames f      ON f.id = o.frame_id
+JOIN video_chunks v ON v.id = f.video_chunk_id
+WHERE julianday(f.timestamp) >= julianday('now','-1 day')
+ORDER BY f.timestamp DESC
+LIMIT 50;
+```
+
+- Top apps/windows/urls (last N hours)
+
+```sql
+-- Apps
+SELECT app_name, COUNT(*) AS n
+FROM frames
+WHERE app_name IS NOT NULL AND app_name <> ''
+  AND julianday(timestamp) >= julianday('now','-3 hours')
+GROUP BY app_name ORDER BY n DESC LIMIT 10;
+
+-- Windows
+SELECT window_name, COUNT(*) AS n
+FROM frames
+WHERE window_name IS NOT NULL AND window_name <> ''
+  AND julianday(timestamp) >= julianday('now','-3 hours')
+GROUP BY window_name ORDER BY n DESC LIMIT 10;
+
+-- URLs
+SELECT browser_url, COUNT(*) AS n
+FROM frames
+WHERE browser_url IS NOT NULL AND browser_url <> ''
+  AND julianday(timestamp) >= julianday('now','-3 hours')
+GROUP BY browser_url ORDER BY n DESC LIMIT 10;
+```
+
+- Vision/UI/Audio tags in last N hours
+
+```sql
+-- Vision tags (on frames)
+SELECT t.name, COUNT(*) AS n
+FROM vision_tags vt
+JOIN tags t ON t.id = vt.tag_id
+JOIN frames f ON f.id = vt.vision_id
+WHERE julianday(f.timestamp) >= julianday('now','-3 hours')
+GROUP BY t.name ORDER BY n DESC LIMIT 10;
+
+-- UI tags
+SELECT t.name, COUNT(*) AS n
+FROM ui_monitoring_tags ut
+JOIN tags t ON t.id = ut.tag_id
+JOIN ui_monitoring u ON u.id = ut.ui_monitoring_id
+WHERE julianday(u.timestamp) >= julianday('now','-3 hours')
+GROUP BY t.name ORDER BY n DESC LIMIT 10;
+
+-- Audio tags (by audio chunk)
+SELECT t.name, COUNT(*) AS n
+FROM audio_transcriptions tr
+JOIN audio_chunks ac ON ac.id = tr.audio_chunk_id
+JOIN audio_tags ag   ON ag.audio_chunk_id = ac.id
+JOIN tags t          ON t.id = ag.tag_id
+WHERE julianday(tr.timestamp) >= julianday('now','-3 hours')
+GROUP BY t.name ORDER BY n DESC LIMIT 10;
+```
+
+- Rank OCR hits and show snippets
+
+```sql
+SELECT f.timestamp,
+       snippet(ocr_text_fts, 0, '[', ']', '…', 8) AS snippet,
+       bm25(ocr_text_fts) AS rank,
+       f.app_name, f.window_name
+FROM ocr_text_fts ft
+JOIN ocr_text o ON o.frame_id = ft.frame_id
+JOIN frames f   ON f.id = o.frame_id
+WHERE ft MATCH 'invoice NEAR/3 total'
+  AND julianday(f.timestamp) >= julianday('now','-1 day')
+ORDER BY rank
+LIMIT 20;
+```
+
+### Practical patterns for agents
+
+- Recent context: pull last 1–3 hours from `frames` and `ocr_text` to infer what the user is reading/doing; correlate with `audio_transcriptions` to detect meetings/conversations.
+- Cross-modal search: start with FTS (`*_fts MATCH 'query'`), then JOIN to frames/audio for timestamps and `video_chunks`/`audio_chunks` file paths to retrieve the exact clip.
+- Summarize a session:
+  1) Get active windows/apps via `frames` in the time window
+  2) Extract `ocr_text` samples for high `text_length` rows
+  3) Extract top transcript lines and speakers
+  4) Produce a bullet summary of tasks, decisions, links (from `browser_url`)
+- Lightweight counts: use `EXISTS`/`LIMIT` for presence checks; avoid `COUNT(*)` on giant ranges unless filtered by time.
+
+### Safety & constraints
+
+- Contains raw on-screen text and mic transcripts; treat as sensitive. Do not export large dumps by default.
+- Writing should only target base tables; FTS tables are maintained by triggers.
+
+### Quick CLI snippets
+
+```bash
+sqlite3 -readonly $HOME/.screenpipe/db.sqlite \
+  "SELECT type,name FROM sqlite_master WHERE name NOT LIKE 'sqlite_%' ORDER BY type,name;"
+
+sqlite3 -readonly $HOME/.screenpipe/db.sqlite <<'SQL'
+.headers on
+.mode table
+SELECT f.timestamp, f.app_name, f.window_name, substr(replace(o.text, char(10),' '),1,160) AS text
+FROM ocr_text_fts ft
+JOIN ocr_text o ON o.frame_id=ft.frame_id
+JOIN frames f ON f.id=o.frame_id
+WHERE julianday(f.timestamp) >= julianday('now','-3 hours') AND ft MATCH 'error OR crash';
+SQL
+```