Skip to content

Adding notebooks for Model Monitoring #344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
manu-sj wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Adding notebooks for Model Monitoring #344

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
174646f
adding monitoring notebook for fraud batch
manu-sj Nov 21, 2025
ef1ed77
adding monitoring to fraud online too
manu-sj Nov 21, 2025
71c8934
adding more comments and refactoring
manu-sj Nov 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
539 changes: 505 additions & 34 deletions batch-ai-systems/fraud_batch/1_fraud_batch_feature_pipeline.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

1,238 changes: 1,162 additions & 76 deletions batch-ai-systems/fraud_batch/2_fraud_batch_training_pipeline.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

1,602 changes: 1,555 additions & 47 deletions batch-ai-systems/fraud_batch/3_fraud_batch_inference.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

192,527 changes: 192,527 additions & 0 deletions batch-ai-systems/fraud_batch/4_fraud_batch_monitoring.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

386 changes: 359 additions & 27 deletions real-time-ai-systems/fraud_online/1_fraud_online_feature_pipeline.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

1,672 changes: 1,535 additions & 137 deletions real-time-ai-systems/fraud_online/2_fraud_online_training_pipeline.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

165 changes: 139 additions & 26 deletions real-time-ai-systems/fraud_online/3_fraud_online_inference_pipeline.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"id": "d28eba60",
"metadata": {},
"source": [
"# <span style=\"font-width:bold; font-size: 3rem; color:#1EB182;\"><img src=\"../images/icon102.png\" width=\"38px\"></img> **Hopsworks Feature Store** </span><span style=\"font-width:bold; font-size: 3rem; color:#333;\">- Part 03: Inference Pipeline</span>\n"
"# <span style=\"font-width:bold; font-size: 3rem; color:#1EB182;\"><img src=\"../../images/icon102.png\" width=\"38px\"></img> **Hopsworks Feature Store** </span><span style=\"font-width:bold; font-size: 3rem; color:#333;\">- Part 03: Inference Pipeline</span>\n"
]
},
{
Expand All @@ -18,10 +18,25 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 13,
"id": "ed952ece",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2025-11-21 17:07:19,389 INFO: Closing external client and cleaning up certificates.\n",
"The history saving thread hit an unexpected error (OperationalError('attempt to write a readonly database')).History will not be written to the database.\n",
"2025-11-21 17:07:19,413 INFO: Connection closed.\n",
"2025-11-21 17:07:19,427 INFO: Initializing external client\n",
"2025-11-21 17:07:19,427 INFO: Base URL: https://10.87.45.81:28181\n",
"2025-11-21 17:07:19,926 INFO: Python Engine initialized.\n",
"\n",
"Logged in to project, explore it here https://10.87.45.81:28181/p/120\n"
]
}
],
"source": [
"import hopsworks\n",
"\n",
Expand All @@ -41,7 +56,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 14,
"id": "d2a8475b",
"metadata": {},
"outputs": [],
Expand All @@ -55,10 +70,29 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 15,
"id": "c37c5197",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Finished: Reading data from Hopsworks, using Hopsworks Feature Query Service (3.97s) \n"
]
},
{
"data": {
"text/plain": [
"array([4896635955418431, 4837978576778318, 4003885636606306,\n",
" 4085049862790828, 4130716471597621])"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Retrieve the first 5 unique credit card numbers (cc_nums)\n",
"cc_nums = trans_fg.select('cc_num').show(5).cc_num.values\n",
Expand All @@ -77,7 +111,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 16,
"id": "be66f4c8",
"metadata": {},
"outputs": [],
Expand All @@ -96,10 +130,32 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 17,
"id": "4303ac82",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a37d7e26238c4cb480bf4a086c85e997",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/5 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Start making predictions by using `.predict()`\n"
]
}
],
"source": [
"# Access the Model Serving\n",
"ms = project.get_model_serving()\n",
Expand Down Expand Up @@ -129,10 +185,21 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 18,
"id": "42196023",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"4896635955418431"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Get the first credit card number\n",
"cc_num = cc_nums[0]\n",
Expand All @@ -141,10 +208,21 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 19,
"id": "596f3241",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"{'predictions': [0]}"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Make a prediction\n",
"deployment.predict(\n",
Expand All @@ -154,10 +232,21 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 20,
"id": "f260b7b6",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"[[0], [0], [0], [0], [0]]"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Predict for several cc_nums\n",
"predictions = [\n",
Expand All @@ -178,10 +267,18 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 21,
"id": "1f0c13d5-f92c-4077-ba36-829595cff12d",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2025-11-21 17:08:04,840 INFO: Initializing for batch retrieval of feature vectors\n"
]
}
],
"source": [
"retrieved_model = mr.get_model(\n",
" name=\"xgboost_fraud_online_model\",\n",
Expand All @@ -195,7 +292,19 @@
"execution_count": null,
"id": "2dedf684",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Launching job: transactions_fraud_online_fv_1_log_1_offline_fg_materialization\n",
"Job started successfully, you can follow the progress at \n",
"https://10.87.45.81:28181/p/120/jobs/named/transactions_fraud_online_fv_1_log_1_offline_fg_materialization/executions\n",
"2025-11-21 17:08:14,423 INFO: Waiting for execution to finish. Current state: SUBMITTED. Final status: UNDEFINED\n",
"2025-11-21 17:08:17,509 INFO: Waiting for execution to finish. Current state: RUNNING. Final status: UNDEFINED\n"
]
}
],
"source": [
"# Stop the job materialization schedule and materialize log manually\n",
"feature_view.pause_logging()\n",
Expand Down Expand Up @@ -265,19 +374,23 @@
"metadata": {},
"source": [
"---\n",
"## <span style=\"color:#ff5f27;\">⏭️ **Next:** Part 04: Model Monitoring</span>\n",
"\n",
"### <span style=\"color:#ff5f27;\">🥳 <b> Next Steps </b> </span>\n",
"Congratulations you've now completed the Fraud Online tutorial for Managed Hopsworks.\n",
"\n",
"Check out our other tutorials on ➡ https://github.com/logicalclocks/hopsworks-tutorials\n",
"\n",
"Or documentation at ➡ https://docs.hopsworks.ai"
"In the following notebook you will monitor your model using the logs that have been written."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b360f625-95fd-4d14-adb3-c9bce66eff2d",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
Expand All @@ -291,7 +404,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.11"
"version": "3.11.13"
}
},
"nbformat": 4,
Expand Down
Loading