Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
fa39408
...
larseirikhansen Jul 28, 2025
2c3ae56
smoketest passed on meilisearch api, azure openai chat & embedding model
larseirikhansen Jul 28, 2025
2592832
ingestion script complete, working for now
larseirikhansen Jul 28, 2025
b04fe74
feat(ai): add semantic search, update deps, setup embedder, implement…
larseirikhansen Jul 28, 2025
580a305
update outdated packages for ai search
larseirikhansen Jul 28, 2025
6d5a358
feat(ai): add to ingestion - components, tokens, figma, themes & best…
larseirikhansen Jul 28, 2025
4385d84
fix(ai-api): use correct index name in embedder setup script
larseirikhansen Jul 28, 2025
c11c229
feat(ai-api): add diagnostic scripts for Meilisearch debugging
larseirikhansen Jul 28, 2025
0c45bcd
fix: update test-filtering.mjs to use correct type value for componen…
larseirikhansen Jul 28, 2025
d7e0bd8
fix: add AI search debug and utility scripts
larseirikhansen Jul 28, 2025
c367710
add env file til gitignore
larseirikhansen Jul 29, 2025
d017b0d
Merge remote-tracking branch 'origin/main' into ai-search
larseirikhansen Jul 29, 2025
6bcee01
lock file update
larseirikhansen Jul 29, 2025
d1cc7d5
add lines to gitignore
larseirikhansen Jul 29, 2025
5ea535c
create search dialog/modal
larseirikhansen Jul 29, 2025
cef716b
very very basic poc
larseirikhansen Jul 30, 2025
4b7a423
add a component synonym dictionary
larseirikhansen Jul 31, 2025
ba0da75
add more editors to gitignore
larseirikhansen Aug 1, 2025
1ff0780
feat: add 'quick search', deduplicate search results, fix Meilisearch…
larseirikhansen Aug 1, 2025
aee055d
comment tweak
larseirikhansen Aug 3, 2025
90ca10e
run biome on /ai-api directory, fix type warnings
larseirikhansen Aug 4, 2025
319801a
Merge branch 'main' into ai-search
oddvernes Oct 2, 2025
886d44f
fixed lockfile?
oddvernes Oct 2, 2025
8b43eaf
some biome fixes
oddvernes Oct 2, 2025
d7513ec
typescript fix
oddvernes Oct 2, 2025
40195a7
move meilisearch to self-host
larseirikhansen Aug 13, 2025
d0db286
tweak meilisearch config
larseirikhansen Oct 2, 2025
80581d8
refactor: switch from OpenAI to user-provided embeddings in Meilisear…
larseirikhansen Oct 2, 2025
892a1bc
feat: add 'Søk' text label next to magnifying glass icon in header
larseirikhansen Oct 2, 2025
d3cec31
feat: add smart answer box with expandable content and loading states…
larseirikhansen Oct 2, 2025
e7a4842
feat: add complete setup script and improve Meilisearch integration w…
larseirikhansen Oct 2, 2025
22d89cf
moved env and readme files to /ai-api
larseirikhansen Oct 2, 2025
0bc17ca
chore: rm unused AI smoke test script
larseirikhansen Oct 2, 2025
083b7ad
Add backup files to gitignore
larseirikhansen Oct 2, 2025
0bb35fc
Potential fix for code scanning alert no. 88: Clear-text logging of s…
larsekhansen Oct 2, 2025
54e9879
Potential fix for code scanning alert no. 90: Clear-text logging of s…
larsekhansen Oct 2, 2025
33b5c36
remove embedder uids in logs
larseirikhansen Oct 2, 2025
fd248c1
Merge branch 'main' into ai-search
oddvernes Oct 6, 2025
115ad67
fix searchbutton
oddvernes Oct 6, 2025
e302015
dialog width + biome formatting
oddvernes Oct 6, 2025
7243f2e
experiment with search border
oddvernes Oct 6, 2025
53f8d90
search field glare
oddvernes Oct 7, 2025
b0635db
border fix
oddvernes Oct 7, 2025
8da5c5b
top adjusted dialog
oddvernes Oct 7, 2025
f0f32c3
sticky header
oddvernes Oct 7, 2025
7eb70b8
biome: lint css as css-modules
oddvernes Oct 7, 2025
e590008
improve skeleton loader
oddvernes Oct 7, 2025
05b00ce
more styles and restructuring
oddvernes Oct 7, 2025
2e858d1
remove unneccessary tooltip
oddvernes Oct 8, 2025
22ba7ea
Merge branch 'main' into ai-search
oddvernes Oct 8, 2025
8143b03
themebuilder should use brand types
oddvernes Oct 8, 2025
c5017c9
biome formatting of several files
oddvernes Oct 8, 2025
0a96e63
improve empty state
oddvernes Oct 8, 2025
e2b6316
add icons to header
oddvernes Oct 8, 2025
be36db0
better overflow solution
oddvernes Oct 8, 2025
a18b185
fix backdrop color in light mode
oddvernes Oct 8, 2025
fde0a21
fix page layout shift when opening dialog
oddvernes Oct 8, 2025
08c47de
add sparkle animation to ai-result heading
oddvernes Oct 8, 2025
93fe56a
remove search button
oddvernes Oct 9, 2025
6dc3c30
try another input design
oddvernes Oct 9, 2025
f78a28c
Merge branch 'main' into ai-search
oddvernes Oct 9, 2025
6fa2a79
fix lockfile
oddvernes Oct 9, 2025
7e8a957
Merge branch 'main' into ai-search
oddvernes Oct 9, 2025
dd0b87d
fix debounce
oddvernes Oct 9, 2025
75e2eb7
different focus style test
oddvernes Oct 10, 2025
0eb9432
internal/components install mdx-bundler
oddvernes Oct 10, 2025
432fad1
wip search action
oddvernes Oct 10, 2025
dd0d435
wip try resource route and useFetcher instead
oddvernes Oct 10, 2025
b70635d
wip simplified mdxComponents
oddvernes Oct 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,10 @@ coverage/
*.coverage
*.coveragexml

# JetBrains IntelliJ IDEA
# Code editors
.idea
.windsurf
.cursor

# Hidden macOS files
.DS_Store
Expand All @@ -49,3 +51,4 @@ packages/cli/temp
# React Router
.react-router
apps/www/build
*env.backup.*
32 changes: 32 additions & 0 deletions apps/ai-api/.ai-env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# ===========================
# AI/Search Environment (.ai-env.example)
# Copy to .ai-env and fill values. This file is NOT loaded automatically by shells.
# Our scripts try several candidate paths to load .ai-env.
# ===========================

# ---- Meilisearch (self-hosted) ----
# Base URL to your Meilisearch instance
MEILISEARCH_API_URL=http://localhost:7700

# Admin key (created via /keys using the master key). Used by ingest/setup scripts and admin-only endpoints
MEILISEARCH_ADMIN_KEY=REPLACE_WITH_CREATED_ADMIN_KEY

# Search key (created via /keys). Used by the backend for search-only client if exposed
MEILISEARCH_SEARCH_KEY=REPLACE_WITH_CREATED_SEARCH_KEY

# Index name used across scripts and server
MEILISEARCH_PROJECT_NAME=designsystemet-search

# Embedder UID (must match what you set via setup:embedder)
MEILISEARCH_EMBEDDER_UID=azure-openai-small

# ---- Azure OpenAI (embeddings + chat) ----
AZURE_KEY=REPLACE
AZURE_ENDPOINT=REPLACE
AZURE_API_VERSION=2024-08-01-preview
AZURE_EMBEDDING_DEPLOY_SMALL=text-embedding-3-small
AZURE_GPT_DEPLOY=gpt-4o

# Notes:
# - Use the SAME embedding model for ingestion and query. Do NOT mix models.
# - After self-hosting Meilisearch, create Admin/Search keys and paste them here.
257 changes: 257 additions & 0 deletions apps/ai-api/AI_SEARCH_CHECKLIST.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,257 @@
# AI Search Setup Checklist

Use this checklist to ensure all steps are completed correctly.

## Prerequisites ✓
- [ ] Node.js >= 22.17.1 installed
- [ ] pnpm 10.13.1 installed
- [ ] Docker and Docker Compose installed
- [ ] Repository cloned and accessible
- [ ] Azure OpenAI credentials received (in `.ai-env` file)

---

## Setup Steps

### 1. Install Dependencies
- [ ] Run `pnpm install` from repo root

### 2. Meilisearch Setup
- [ ] Navigate to `infra/meilisearch/`
- [ ] Copy `.env.example` to `.env`
- [ ] Set `MEILI_MASTER_KEY` to a long random string (64+ chars)
- [ ] Start container: `docker compose -f infra/meilisearch/docker-compose.yml up -d`
- [ ] Verify running: `docker ps | grep meilisearch`
- [ ] Meilisearch accessible at `http://localhost:7700`

### 3. Create API Keys
- [ ] Export master key: `export MEILI_MASTER_KEY=your-master-key`
- [ ] Run: `pnpm -w -C apps/ai-api run setup:create-keys`
- [ ] Copy **Admin Key** (save it)
- [ ] Copy **Search Key** (save it)

### 4. Configure Environment
- [ ] Copy `.ai-env.example` to `.ai-env` (repo root)
- [ ] Paste Admin Key into `MEILISEARCH_ADMIN_KEY`
- [ ] Paste Search Key into `MEILISEARCH_SEARCH_KEY`
- [ ] Verify Azure credentials are present:
- [ ] `AZURE_KEY`
- [ ] `AZURE_ENDPOINT`
- [ ] `AZURE_API_VERSION`
- [ ] `AZURE_EMBEDDING_DEPLOY_SMALL`
- [ ] `AZURE_GPT_DEPLOY`

### 5. Initialize Search Index
- [ ] Navigate to `apps/ai-api/`
- [ ] Run: `pnpm run setup` (runs all setup steps - takes 5-10 min)
- [ ] Verify output shows ~768 documents indexed at the end

### 6. Start AI API
- [ ] From `apps/ai-api/`: `pnpm run dev`
- [ ] Verify server starts on port 3001
- [ ] Test health: `curl http://localhost:3001/health`
- [ ] Test search: `curl -X POST http://localhost:3001/api/search -H "Content-Type: application/json" -d '{"query": "button"}'`

### 7. Start WWW Frontend
- [ ] Open new terminal
- [ ] From repo root: `pnpm run www`
- [ ] Verify server starts on port 5173
- [ ] Open browser to `http://localhost:5173`

### 8. Test Search Feature
- [ ] Click search icon (magnifying glass) in header
- [ ] Search dialog opens
- [ ] Try single-word query: `button` (Quick mode)
- [ ] Try multi-word query: `how to use button component` (Quick + Smart mode)
- [ ] Verify results appear
- [ ] Verify AI answer appears for multi-word queries
- [ ] Click "Show more" to expand AI answer
- [ ] Verify sources are clickable
- [ ] Test synonym: search `modal` (should find Dialog)

---

## Verification Tests

### Meilisearch
```bash
# Container is running
docker ps | grep meilisearch

# Check logs (should show no errors)
docker compose -f infra/meilisearch/docker-compose.yml logs --tail=50

# Check health
curl http://localhost:7700/health
```

### AI API
```bash
# Health check
curl http://localhost:3001/health

# Quick search
curl -X POST http://localhost:3001/api/search \
-H "Content-Type: application/json" \
-d '{"query": "button"}'

# AI search
curl -X POST http://localhost:3001/api/ai-search \
-H "Content-Type: application/json" \
-d '{"query": "how do I use the button component"}'

# Check index status (or re-run full setup)
cd apps/ai-api
pnpm run setup
```

### Frontend
- [ ] Open `http://localhost:5173` in browser
- [ ] No console errors
- [ ] Search icon visible in header
- [ ] Search dialog opens on click
- [ ] Search input is focused
- [ ] Suggestions appear when empty
- [ ] Results appear when typing
- [ ] Links are clickable
- [ ] Dialog closes on Escape or X button

---

## Common Issues

### "Connection refused" to Meilisearch
- [ ] Check Docker container is running: `docker ps`
- [ ] Check port 7700 is not in use: `lsof -i :7700`
- [ ] Restart container: `docker compose -f infra/meilisearch/docker-compose.yml restart`

### "Invalid API key"
- [ ] Verify `.ai-env` has correct Admin/Search keys
- [ ] Recreate keys if needed (step 3)
- [ ] Ensure no extra spaces in `.ai-env` values

### "No documents found"
- [ ] Re-run complete setup: `pnpm run setup`
- [ ] Check for errors in setup output
- [ ] Verify `.ai-env` has correct Azure credentials

### "Azure OpenAI error"
- [ ] Verify Azure credentials in `.ai-env`
- [ ] Check Azure endpoint is accessible
- [ ] Verify deployment names match (text-embedding-3-small, gpt-4o)

### Search returns no results
- [ ] Re-run complete setup: `pnpm run setup`
- [ ] Check Meilisearch logs for errors
- [ ] Verify documents were ingested (shown at end of setup)

### Frontend can't connect to API
- [ ] Verify AI API is running on port 3001
- [ ] Check CORS configuration in `apps/ai-api/src/server.ts`
- [ ] Check browser console for CORS errors

---

## Environment Files Checklist

### `.ai-env` (repo root) - REQUIRED
```bash
MEILISEARCH_API_URL=http://localhost:7700
MEILISEARCH_ADMIN_KEY=<FROM_STEP_3>
MEILISEARCH_SEARCH_KEY=<FROM_STEP_3>
MEILISEARCH_PROJECT_NAME=designsystemet-search
MEILISEARCH_EMBEDDER_UID=azure-openai-small
AZURE_KEY=<PROVIDED>
AZURE_ENDPOINT=<PROVIDED>
AZURE_API_VERSION=2024-08-01-preview
AZURE_EMBEDDING_DEPLOY_SMALL=text-embedding-3-small
AZURE_GPT_DEPLOY=gpt-4o
```

### `infra/meilisearch/.env` - REQUIRED
```bash
MEILI_MASTER_KEY=<YOUR_LONG_RANDOM_SECRET>
MEILI_ENV=production
MEILI_NO_ANALYTICS=true
MEILI_EXPERIMENTAL_ENABLE_VECTOR_SEARCH=enabled
MEILI_EXPERIMENTAL_ENABLE_EMBEDDERS=enabled
```

---

## Running Services (3 Terminals)

### Terminal 1: Meilisearch
```bash
cd infra/meilisearch
docker compose up
# Keep running, watch logs
```

### Terminal 2: AI API
```bash
cd apps/ai-api
pnpm run dev
# Keep running, watch logs
```

### Terminal 3: WWW Frontend
```bash
# From repo root
pnpm run www
# Keep running, watch logs
```

---

## Success Criteria

✅ All three services running without errors
✅ Search dialog opens in browser
✅ Single-word queries return Quick results
✅ Multi-word queries return both Quick and Smart results
✅ AI answers are formatted with markdown
✅ Sources are expandable and clickable
✅ Synonyms work (e.g., "modal" finds Dialog)
✅ No CORS errors in browser console
✅ Health endpoint returns `{"status": "ok"}`

---

## Cleanup (Optional)

To stop all services:

```bash
# Stop frontend (Ctrl+C in Terminal 3)
# Stop AI API (Ctrl+C in Terminal 2)

# Stop Meilisearch container
docker compose -f infra/meilisearch/docker-compose.yml down

# To also remove data volume (WARNING: deletes all indexed documents)
docker compose -f infra/meilisearch/docker-compose.yml down -v
```

---

## Quick Start (After Initial Setup)

Once everything is configured, you only need to run:

```bash
# Terminal 1
docker compose -f infra/meilisearch/docker-compose.yml up -d

# Terminal 2
cd apps/ai-api && pnpm run dev

# Terminal 3
pnpm run www
```

Then open `http://localhost:5173` and start searching!

---

**Status**: [ ] Setup Complete | [ ] Tested and Working | [ ] Ready for Development
Loading