feat: add serve mode, and manual run button #146
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🦋 Changeset detectedLatest commit: 83a81b3 The changes in this PR will be included in the next version bump. Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
this would be super useful! hopefully it can merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi @mattpocock, I hope this is handy!
This PR solves #144, #100, and another couple of similar questions/issues in the ai-hero cohort 1.
evalite watchwill work in the same way as it has done previously, detecting changedeval.tsfiles and performing a run.evalite servewill serve the UI without performing a run.A button has been added to the UI that allows runs to be triggered manually. A manual run will respect any new or changed
eval.tsfiles.The main benefit of this is that it enables a workflow where you can fiddle about with your evals, code under test, etc, and not worry about triggering many runs, eating up lots of tokens. You can wait until you've got things the way you want them and then perform the run.