A Python SDK for optimizing prompts for Nova.
- Installation
- Pre-Requisites
- Quick Start: Facility Support Analyzer Dataset
- Core Concepts
- Optimization Recommendations
- Preview Status
- Interaction with AWS Bedrock
- Acknowledgements
Install the library using
pip3 install nova-prompt-optimizerTo execute the SDK, you will need AWS credentials configured. Take a look at the AWS CLI configuration documentation for details on the various ways to configure credentials. An easy way to try out the SDK is to populate the following environment variables with your AWS API credentials. Take a look at this guide for Authenticating with short-term credentials for the AWS CLI
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
- Go to the Amazon Bedrock Model Access page
- Click "Manage model access"
- Choose Amazon as provider and Nova models
- Click "Request access"
- Wait for approval (instant in most cases)
The Facility Support Analyzer dataset consists of emails that are to be classified based on category, urgency and sentiment.
Please see the samples folder for example notebooks of how to optimize a prompt in the scenario where a user prompt template is to be optimized and the scenario where a user and system prompt is to be optimized together
We break each component of Prompt Optimization into Adapters providing a modular approach to Prompt Optimization.
Responsibility: Ability to load prompts from different formats and store them in the standardized format (JSON)
Sample Prompt Adapter Initialization
from amzn_nova_prompt_optimizer.core.input_adapters.prompt_adapter import TextPromptAdapter
prompt_adapter = TextPromptAdapter()
prompt_adapter.set_system_prompt(file_path="prompt/sys_prompt.txt", variables={"foo"})
prompt_adapter.set_user_prompt(content="You are a .....", variables={"bar"})
prompt_adapter.adapt()Supported Prompt Adapters: TextPromptAdapter
Learn More about the Prompt Adapter here
Responsibility: Ability to call an inference backend for the models e.g. Bedrock, etc.
Sample use of Inference Adapter
from amzn_nova_prompt_optimizer.core.inference.adapter import BedrockInferenceAdapter
inference_adapter = BedrockInferenceAdapter(region_name="us-east-1")You can pass rate_limit into constructor of InferenceAdapter to limit the max TPS of bedrock call to avoid throttle. Default to 2 if not set.
from amzn_nova_prompt_optimizer.core.inference.adapter import BedrockInferenceAdapter
inference_adapter = BedrockInferenceAdapter(region_name="us-east-1", rate_limit=10) # Max 10 TPSSupported Inference Adapters: BedrockInferenceAdapter
Core Functions
Call the model using the parameters
# Call the model with the passed parametrs
inference_output = inference_adapter.call_model(model_id: str, system_prompt: str, messages: List[Dict[str, str]], inf_config: Dict[str, Any])The Inference Adapter accepts the system_prompt as a string.
The input to the model as a list of User/Assistant turns (messages). e.g. [{"user": "foo"}, {"assistant": "bar"}, {"user": "What comes next?"}]
Responsibility: Ability to read/write datasets from different formats. Uses an intermediary standardized format when communicating with other adapters. It can also read list of JSON object. It can also create Train/Test splits (with stratify capability if set).
Requirements: Currently, you can only provide a singleton set as output column.
Sample Dataset Adapter Initialization
# Example Usage
from amzn_nova_prompt_optimizer.core.input_adapters.dataset_adapter import JSONDatasetAdapter
input_columns = {"input"}
output_columns = {"answer"}
dataset_adapter = JSONDatasetAdapter(input_columns, output_columns)
# Adapt
dataset_adapter.adapt(data_source="sample_data.jsonl")
# Split
train, test = dataset_adapter.split(0.5)Supported Dataset Adapters: JSONDatasetAdapter, CSVDatasetAdapter
Learn More about the Dataset Adapter here
Responsibility: Ability to load custom metrics and apply them on inference output and ground truth
Metric Adapter Class
class MetricAdapter():
def apply(self, y_pred: Any, y_true: Any) -> float:
pass
def batch_apply(self, y_preds: List[Any], y_trues: List[Any]) -> float:
passSample Custom Metric Adapter Initialization
Let's create a Custom metric adapter that parses the inference output for the answer between <answer> </answer> tags and then performs an exact match metric.
from amzn_nova_prompt_optimizer.core.input_adapters.metric_adapter import MetricAdapter
from typing import List, Any, Dict
import re
import json
class CustomMetric(MetricAdapter):
def _parse_answer(self, model_output):
# Parse Answer between tags
match = re.search(r"<answer>(.*?)</answer>", model_output)
if not match:
return "Choice not found"
return match.group(1).lower().strip()
def _calculate_metrics(self, y_pred: Any, y_true: Any) -> Dict:
# Peform Exact Match
pred = self._parse_answer(y_pred)
ground_truth = self._parse_answer(y_true)
return int(pred == ground_truth)
def apply(self, y_pred: Any, y_true: Any):
# Apply to one row of the dataset
return self._calculate_metrics(y_pred, y_true)
def batch_apply(self, y_preds: List[Any], y_trues: List[Any]):
# Apply to the whole dataset
evals = []
for y_pred, y_true in zip(y_preds, y_trues):
evals.append(self.apply(y_pred, y_true))
return sum(evals)/len(evals)
metric_adapter = CustomMetric()Core Functions Apply the metric on a prediction and ground_truth one row at a time
y_pred = "The question asks ...... <answer>3</answer>"
y_true = "<answer>3</answer>"
# Apply the metric on a prediction and ground_truth
score = metric_adapter.apply(y_pred, y_true)
# score = 1Apply the metric on a list of prediction and ground_truth i.e. for the dataset
y_preds = ["The question asks ...... <answer>3</answer>", "The question asks ...... <answer>5</answer>"]
y_trues = ["<answer>3</answer>", "<answer>4</answer>"]
# Apply the metric on a list of prediction and ground_truth
aggregeate_score = metric_adapter.batch_apply(y_preds, y_trues)
# aggregeate_score = 0.5Responsibility: Load Optimizer, Prompt Adapter, and Optionally Dataset Adapter, Metric Adapter, and Inference Adapter. Perform Optimization and ability to create a Prompt Adapter with the Optimized Prompt.
Sample Optimization Initialization
from amzn_nova_prompt_optimizer.core.optimizers import NovaPromptOptimizer
nova_prompt_optimizer = NovaPromptOptimizer(prompt_adapter=prompt_adapter, inference_adapter=inference_adapter, dataset_adapter=train_dataset_adapter, metric_adapter=metric_adapter)
optimized_prompt_adapter = nova_prompt_optimizer.optimize(mode="lite")We can take a look more deeply into the optimizers in the next section.
NovaPromptOptimizer is a combination of Meta Prompting using the Nova Guide on prompting and DSPy's MIPROv2 Optimizer using Nova Prompting Tips.
NovaPromptOptimizer first runs a meta prompter to identify system instructions and user template from the prompt adapter.
Then MIPROv2 is run on top of this to optimize system instructions and identify few-shot samples that need to be added.
The few shot samples are added as converse format so they are added as User/Assistant turns.
Requirements: NovaPromptOptimizer requires Prompt Adapter, Dataset Adapter, Metric Adapter and Inference Adapter.
Optimization Example
from amzn_nova_prompt_optimizer.core.optimizers import NovaPromptOptimizer
nova_prompt_optimizer = NovaPromptOptimizer(prompt_adapter=prompt_adapter, inference_adapter=inference_adapter, dataset_adapter=train_dataset_adapter, metric_adapter=metric_adapter)
optimized_prompt_adapter = nova_prompt_optimizer.optimize(mode="lite")NovaPromptOptimizer uses Premier for Meta Prompting and then uses MIPROv2 with 20 candidates and 50 trials with Premier as Prompting model and task model dependent on the mode it's set at. You can specify enable_json_fallback=False to disable the behavior that MIPROv2 will fallback to use JSONAdapter to parse LM model output. This will force MIPROv2 use structured output (pydantic model) to parse LM output.
You could also define a custom mode and pass your own parameter values to NovaPromptOptimizer
from amzn_nova_prompt_optimizer.core.optimizers import NovaPromptOptimizer
nova_prompt_optimizer = NovaPromptOptimizer(prompt_adapter=prompt_adapter, inference_adapter=inference_adapter, dataset_adapter=train_dataset_adapter, metric_adapter=metric_adapter)
optimized_prompt_adapter = nova_prompt_optimizer.optimize(mode="custom", custom_params={"task_model_id": "us.amazon.nova-pro-v1:0",
"num_candidates": 10,
"num_trials": 15,
"max_bootstrapped_demos": 5,
"max_labeled_demos": 0
})Learn More about the Optimizers here
The SDK also provides a way to baseline prompts and provide evaluation scores.
The evaluator has the aggregate_score and scores function.
Initialization Example
from amzn_nova_prompt_optimizer.core.evaluation import Evaluator
evaluator = Evaluator(nova_mp_optimized_prompt_adapter, test_dataset_adapter, metric_adapter, inference_adapter)
nova_mp_score = evaluator.aggregate_score(model_id="us.amazon.nova-lite-v1:0")Core Functions
Runs Batch evaluation on the dataset using the batch_apply function of the metric
# Uses Batch Apply
aggregate_score = evaluator.aggregate_score(model_id)Runs evaluation on the dataset a row at a time and returns the eval results as a whole.
# Uses Apply metric. Returns a list of scores.
scores = evaluator.score(model_id)Save the eval results.
# Save the eval results
evaluator.save("eval_results.jsonl")Note: You may come across the below warning. This is when prompt variables are missing from the prompt, the inference runner under the evaluator appends them to the end of the prompt for continuity
WARNING amzn_nova_prompt_optimizer.core.inference: Warn: Prompt Variables not found in User Prompt, injecting them at the end of the prompt- Provide representative real-world evaluation sets and split them into training and testing sets. Ensure dataset is balanced on output label when splitting train and test sets.
- For evaluation sets, the ground truth column should be as close to the inference output as possible. e.g. If the inference output is {"answer": "POSITIVE"} ground truth should also be in the same format {"answer": "POSITIVE"}
- For NovaPromptOptimizer, choose the mode (mode= "premier" | ""pro" | "lite" | "micro") based on your Nova Model of choice. By default, we use "pro".
- The
applyfunction of the evaluation metric should return a numerical value between 0 and 1 for NovaPromptOptimizer or MIPROv2.
NovaPromptOptimizer is currently in public preview. During this period:
- SDK functionality might change as we support more use cases.
- We welcome feedback and contributions
NovaPromptOptimizer only uses AWS Bedrock for Model Calls. Bedrock by default disables model invocation logging. Data is not retained in the user's AWS account unless the user enables Bedrock Model Invocation logging. Data storage by Amazon Bedrock is independent of the use of the SDK.
Please refer to Amazon Bedrock's Data Protection Guide for additional guidance
- Special acknowledgment to DSPy – your innovations continue to inspire us.