Skip to content

feat - tests + evals #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat - tests + evals #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,15 @@ This MCP service integrates with EdgeOne Pages Functions to deploy static HTML c

For more information, see the [EdgeOne Pages Functions documentation](https://edgeone.ai/document/162227908259442688) and [EdgeOne Pages KV Storage Guide](https://edgeone.ai/document/162227803822321664).



## Running evals

The evals package loads an mcp client that then runs the index.ts file, so there is no need to rebuild between tests. You can load environment variables by prefixing the npx command. Full documentation can be found [here](https://www.mcpevals.io/docs).

```bash
OPENAI_API_KEY=your-key npx mcp-eval evals.ts index.ts
```
## License

MIT
32 changes: 32 additions & 0 deletions evals.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
//evals.ts

import { EvalConfig } from 'mcp-evals';
import { openai } from "@ai-sdk/openai";
import { grade, EvalFunction } from "mcp-evals";

const deploy_htmlEval: EvalFunction = {
name: 'deploy_html Tool Evaluation',
description: 'Tests the deployment of HTML content to EdgeOne Pages, returning the public URL',
run: async () => {
const result = await grade(openai("gpt-4"), "Deploy the following HTML to EdgeOne: <html><body><h1>Hello EdgeOne!</h1></body></html> Return the public URL.");
return JSON.parse(result);
}
};

const deploy_folder_or_zipEval: EvalFunction = {
name: 'deploy_folder_or_zip Tool Evaluation',
description: 'Evaluates the deployment of a folder or zip to EdgeOne Pages and retrieving the public URL',
run: async () => {
const result = await grade(openai("gpt-4"), "Deploy the folder located at './my-website-folder' using the deploy_folder_or_zip tool and share the resulting public URL.");
return JSON.parse(result);
}
};

const config: EvalConfig = {
model: openai("gpt-4"),
evals: [deploy_htmlEval, deploy_folder_or_zipEval]
};

export default config;

export const evals = [deploy_htmlEval, deploy_folder_or_zipEval];
5 changes: 3 additions & 2 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,11 @@
"@modelcontextprotocol/sdk": "^1.7.0",
"cos-nodejs-sdk-v5": "^2.14.7",
"dotenv": "^16.4.7",
"zod": "^3.24.2"
"zod": "^3.24.2",
"mcp-evals": "^1.0.18"
},
"devDependencies": {
"@types/node": "^20.11.28",
"typescript": "^5.4.2"
}
}
}