From 2dcd1c70e1a754490765406983e809768567bbba Mon Sep 17 00:00:00 2001 From: AayushTyagi1 Date: Tue, 23 Sep 2025 09:42:56 +0530 Subject: [PATCH 1/9] DA-1075: Modified: Added FTS in folder --- pydantic_ai/{ => fts}/.env.sample | 0 pydantic_ai/{ => fts}/RAG_with_Couchbase_and_PydanticAI.ipynb | 0 pydantic_ai/{ => fts}/frontmatter.md | 0 pydantic_ai/{ => fts}/pydantic_ai_index.json | 0 4 files changed, 0 insertions(+), 0 deletions(-) rename pydantic_ai/{ => fts}/.env.sample (100%) rename pydantic_ai/{ => fts}/RAG_with_Couchbase_and_PydanticAI.ipynb (100%) rename pydantic_ai/{ => fts}/frontmatter.md (100%) rename pydantic_ai/{ => fts}/pydantic_ai_index.json (100%) diff --git a/pydantic_ai/.env.sample b/pydantic_ai/fts/.env.sample similarity index 100% rename from pydantic_ai/.env.sample rename to pydantic_ai/fts/.env.sample diff --git a/pydantic_ai/RAG_with_Couchbase_and_PydanticAI.ipynb b/pydantic_ai/fts/RAG_with_Couchbase_and_PydanticAI.ipynb similarity index 100% rename from pydantic_ai/RAG_with_Couchbase_and_PydanticAI.ipynb rename to pydantic_ai/fts/RAG_with_Couchbase_and_PydanticAI.ipynb diff --git a/pydantic_ai/frontmatter.md b/pydantic_ai/fts/frontmatter.md similarity index 100% rename from pydantic_ai/frontmatter.md rename to pydantic_ai/fts/frontmatter.md diff --git a/pydantic_ai/pydantic_ai_index.json b/pydantic_ai/fts/pydantic_ai_index.json similarity index 100% rename from pydantic_ai/pydantic_ai_index.json rename to pydantic_ai/fts/pydantic_ai_index.json From 39dcbfd18c8949b1635a2c97820298adcb121702 Mon Sep 17 00:00:00 2001 From: AayushTyagi1 Date: Tue, 23 Sep 2025 09:47:20 +0530 Subject: [PATCH 2/9] Added env and frontmatter --- pydantic_ai/fts/frontmatter.md | 5 +++-- pydantic_ai/gsi/.env.sample | 8 ++++++++ pydantic_ai/gsi/frontmatter.md | 23 +++++++++++++++++++++++ 3 files changed, 34 insertions(+), 2 deletions(-) create mode 100644 pydantic_ai/gsi/.env.sample create mode 100644 pydantic_ai/gsi/frontmatter.md diff --git a/pydantic_ai/fts/frontmatter.md b/pydantic_ai/fts/frontmatter.md index ea637126..74ad650e 100644 --- a/pydantic_ai/fts/frontmatter.md +++ b/pydantic_ai/fts/frontmatter.md @@ -1,10 +1,10 @@ --- # frontmatter path: "/tutorial-pydantic-ai-couchbase-rag" -title: Retrieval-Augmented Generation (RAG) with Couchbase and PydanticAI +title: Retrieval-Augmented Generation (RAG) with Couchbase and PydanticAI using FTS short_title: RAG with Couchbase and PydanticAI description: - - Learn how to build a semantic search engine using Couchbase and PydanticAI. + - Learn how to build a semantic search engine using Couchbase and PydanticAI using FTS. - This tutorial demonstrates how to integrate Couchbase's vector search capabilities with PydanticAI using tool calling. - You'll understand how to perform Retrieval-Augmented Generation (RAG) using PydanticAI and Couchbase. content_type: tutorial @@ -16,6 +16,7 @@ tags: - LangChain - OpenAI - PydanticAI + - FTS sdk_language: - python length: 30 Mins diff --git a/pydantic_ai/gsi/.env.sample b/pydantic_ai/gsi/.env.sample new file mode 100644 index 00000000..04ca34ce --- /dev/null +++ b/pydantic_ai/gsi/.env.sample @@ -0,0 +1,8 @@ +OPENAI_API_KEY= +CB_HOST= +CB_USERNAME= +CB_PASSWORD= +CB_BUCKET_NAME= +INDEX_NAME= +SCOPE_NAME= +COLLECTION_NAME= \ No newline at end of file diff --git a/pydantic_ai/gsi/frontmatter.md b/pydantic_ai/gsi/frontmatter.md new file mode 100644 index 00000000..307f717e --- /dev/null +++ b/pydantic_ai/gsi/frontmatter.md @@ -0,0 +1,23 @@ +--- +# frontmatter +path: "/tutorial-pydantic-ai-couchbase-rag" +title: Retrieval-Augmented Generation (RAG) with Couchbase and PydanticAI using GSI +short_title: RAG with Couchbase and PydanticAI +description: + - Learn how to build a semantic search engine using Couchbase and PydanticAI using GSI. + - This tutorial demonstrates how to integrate Couchbase's vector search capabilities with PydanticAI using tool calling. + - You'll understand how to perform Retrieval-Augmented Generation (RAG) using PydanticAI and Couchbase. +content_type: tutorial +filter: sdk +technology: + - vector search +tags: + - Artificial Intelligence + - LangChain + - OpenAI + - PydanticAI + - GSI +sdk_language: + - python +length: 30 Mins +--- \ No newline at end of file From ac36e323481d3a7af1eb06d089aaa54c02f6de62 Mon Sep 17 00:00:00 2001 From: AayushTyagi1 Date: Tue, 23 Sep 2025 11:47:57 +0530 Subject: [PATCH 3/9] DA-1075 Added Tutorial for Pydantic AI --- pydantic_ai/gsi/.env.sample | 1 - .../RAG_with_Couchbase_and_PydanticAI.ipynb | 1527 +++++++++++++++++ pydantic_ai/gsi/frontmatter.md | 8 +- 3 files changed, 1531 insertions(+), 5 deletions(-) create mode 100644 pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb diff --git a/pydantic_ai/gsi/.env.sample b/pydantic_ai/gsi/.env.sample index 04ca34ce..484cb8d7 100644 --- a/pydantic_ai/gsi/.env.sample +++ b/pydantic_ai/gsi/.env.sample @@ -3,6 +3,5 @@ CB_HOST= CB_USERNAME= CB_PASSWORD= CB_BUCKET_NAME= -INDEX_NAME= SCOPE_NAME= COLLECTION_NAME= \ No newline at end of file diff --git a/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb b/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb new file mode 100644 index 00000000..fe680020 --- /dev/null +++ b/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb @@ -0,0 +1,1527 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "31c4cb22", + "metadata": {}, + "source": [ + "# Introduction\n", + "In this guide, we will walk you through building a powerful semantic search engine using Couchbase as the backend database, [OpenAI](https://openai.com) as the embedding and LLM provider, and [PydanticAI](https://ai.pydantic.dev) as an agent orchestrator. Semantic search goes beyond simple keyword matching by understanding the context and meaning behind the words in a query, making it an essential tool for applications that require intelligent information retrieval. This tutorial is designed to be beginner-friendly, with clear, step-by-step instructions that will equip you with the knowledge to create a fully functional semantic search system using GSI (Global Secondary Index) from scratch. Alternatively if you want to perform semantic search using the FTS index, please take a look at [this.](https://developer.couchbase.com/tutorial-pydantic-ai-couchbase-rag/)\n" + ] + }, + { + "cell_type": "markdown", + "id": "adbf16dd", + "metadata": {}, + "source": [ + "# How to run this tutorial\n", + "\n", + "This tutorial is available as a Jupyter Notebook (`.ipynb` file) that you can run interactively. You can access the original notebook [here](https://github.com/couchbase-examples/vector-search-cookbook/blob/main/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb).\n", + "\n", + "You can either download the notebook file and run it on [Google Colab](https://colab.research.google.com/) or run it on your system by setting up the Python environment.\n" + ] + }, + { + "cell_type": "markdown", + "id": "3ed15541", + "metadata": {}, + "source": [ + "# Before you start\n", + "\n", + "## Get Credentials for OpenAI\n", + "\n", + "* Please follow the [instructions](https://platform.openai.com/docs/quickstart) to generate the OpenAI credentials.\n", + "\n", + "## Create and Deploy Your Free Tier Operational cluster on Capella\n", + "\n", + "To get started with Couchbase Capella, create an account and use it to deploy a forever free tier operational cluster. This account provides you with an environment where you can explore and learn about Capella with no time constraint.\n", + "\n", + "To learn more, please follow the [instructions](https://docs.couchbase.com/cloud/get-started/create-account.html).\n", + "\n", + "Note: To run this this tutorial, you will need Capella with Couchbase Server version 8.0 or above as GSI vector search is supported only from version 8.0\n", + "\n", + "### Couchbase Capella Configuration\n", + "\n", + "When running Couchbase using [Capella](https://cloud.couchbase.com/sign-in), the following prerequisites need to be met.\n", + "\n", + "* Create the [database credentials](https://docs.couchbase.com/cloud/clusters/manage-database-users.html) to access the required bucket (Read and Write) used in the application.\n", + "\n", + "* [Allow access](https://docs.couchbase.com/cloud/clusters/allow-ip-address.html) to the Cluster from the IP on which the application is running.\n" + ] + }, + { + "cell_type": "markdown", + "id": "c6197bd1", + "metadata": {}, + "source": [ + "# Setting the Stage: Installing Necessary Libraries\n", + "To build our semantic search engine, we need a robust set of tools. The libraries we install handle everything from connecting to databases to performing complex machine learning tasks. Each library has a specific role: Couchbase libraries manage database operations, LangChain handles AI model integrations, and OpenAI provides advanced AI models for generating embeddings and understanding natural language. By setting up these libraries, we ensure our environment is equipped to handle the data-intensive and computationally complex tasks required for semantic search.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b885b1a8", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install --quiet datasets==3.5.0 langchain-couchbase==0.5.0rc1 langchain-openai==0.3.32 python-dotenv==1.1.1 pydantic-ai==0.1.1" + ] + }, + { + "cell_type": "markdown", + "id": "72c292e1", + "metadata": {}, + "source": [ + "# Importing Necessary Libraries\n", + "The script starts by importing a series of libraries required for various tasks, including handling JSON, logging, time tracking, Couchbase connections, embedding generation, and dataset loading. These libraries provide essential functions for working with data, managing database connections, and processing machine learning models.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "bafc8b3c", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/aayush.tyagi/Documents/couchbaselabs-vector-search-cookbook/vector-search-cookbook/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", + " from .autonotebook import tqdm as notebook_tqdm\n" + ] + } + ], + "source": [ + "import getpass\n", + "import json\n", + "import logging\n", + "import os\n", + "import time\n", + "from datetime import timedelta\n", + "\n", + "from couchbase.auth import PasswordAuthenticator\n", + "from couchbase.cluster import Cluster\n", + "from couchbase.exceptions import (CouchbaseException,\n", + " InternalServerFailureException,\n", + " QueryIndexAlreadyExistsException)\n", + "from couchbase.management.buckets import CreateBucketSettings\n", + "from couchbase.options import ClusterOptions\n", + "from datasets import load_dataset\n", + "from dotenv import load_dotenv\n", + "\n", + "from langchain_couchbase.vectorstores import CouchbaseQueryVectorStore\n", + "from langchain_couchbase.vectorstores import DistanceStrategy\n", + "from langchain_couchbase.vectorstores import IndexType\n", + "from langchain_openai import OpenAIEmbeddings\n", + "\n", + "from dataclasses import dataclass\n", + "from pydantic_ai import Agent, RunContext\n" + ] + }, + { + "cell_type": "markdown", + "id": "7664b42d", + "metadata": {}, + "source": [ + "# Setup Logging\n", + "Logging is configured to track the progress of the script and capture any errors or warnings. This is crucial for debugging and understanding the flow of execution. The logging output includes timestamps, log levels (e.g., INFO, ERROR), and messages that describe what is happening in the script.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "fafabd73", + "metadata": {}, + "outputs": [], + "source": [ + "logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s', force=True)\n", + "\n", + "# Disable all logging except critical to prevent OpenAI API request logs\n", + "logging.getLogger(\"httpx\").setLevel(logging.CRITICAL)\n" + ] + }, + { + "cell_type": "markdown", + "id": "dde5f3aa", + "metadata": {}, + "source": [ + "# Loading Sensitive Information\n", + "In this section, we prompt the user to input essential configuration settings needed. These settings include sensitive information like API keys, database credentials, and specific configuration names. Instead of hardcoding these details into the script, we request the user to provide them at runtime, ensuring flexibility and security.\n", + "\n", + "The script also validates that all required inputs are provided, raising an error if any crucial information is missing. This approach ensures that your integration is both secure and correctly configured without hardcoding sensitive information, enhancing the overall security and maintainability of your code.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "39c14f4b", + "metadata": {}, + "outputs": [], + "source": [ + "load_dotenv()\n", + "\n", + "# Load from environment variables or prompt for input in one-liners\n", + "OPENAI_API_KEY = os.getenv('OPENAI_API_KEY') or getpass.getpass('Enter your OpenAI API key: ')\n", + "CB_HOST = os.getenv('CB_HOST', 'couchbase://localhost') or input('Enter your Couchbase host (default: couchbase://localhost): ') or 'couchbase://localhost'\n", + "CB_USERNAME = os.getenv('CB_USERNAME', 'Administrator') or input('Enter your Couchbase username (default: Administrator): ') or 'Administrator'\n", + "CB_PASSWORD = os.getenv('CB_PASSWORD', 'password') or getpass.getpass('Enter your Couchbase password (default: password): ') or 'password'\n", + "CB_BUCKET_NAME = os.getenv('CB_BUCKET_NAME', 'query-vector-search-testing') or input('Enter your Couchbase bucket name (default: query-vector-search-testing): ') or 'query-vector-search-testing'\n", + "SCOPE_NAME = os.getenv('SCOPE_NAME', 'shared') or input('Enter your scope name (default: shared): ') or 'shared'\n", + "COLLECTION_NAME = os.getenv('COLLECTION_NAME', 'pydantic_ai') or input('Enter your collection name (default: pydantic_ai): ') or 'pydantic_ai'\n", + "\n", + "# Check if the variables are correctly loaded\n", + "if not OPENAI_API_KEY:\n", + " raise ValueError(\"OPENAI_API_KEY is not set in the environment.\")\n", + " \n", + "if 'OPENAI_API_KEY' not in os.environ:\n", + " os.environ['OPENAI_API_KEY'] = OPENAI_API_KEY\n" + ] + }, + { + "cell_type": "markdown", + "id": "ecccef39", + "metadata": {}, + "source": [ + "# Connecting to the Couchbase Cluster\n", + "Connecting to a Couchbase cluster is the foundation of our project. Couchbase will serve as our primary data store, handling all the storage and retrieval operations required for our semantic search engine. By establishing this connection, we enable our application to interact with the database, allowing us to perform operations such as storing embeddings, querying data, and managing collections. This connection is the gateway through which all data will flow, so ensuring it's set up correctly is paramount.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "fb36557f", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2025-09-23 11:44:21,729 - INFO - Successfully connected to Couchbase\n" + ] + } + ], + "source": [ + "try:\n", + " auth = PasswordAuthenticator(CB_USERNAME, CB_PASSWORD)\n", + " options = ClusterOptions(auth)\n", + " cluster = Cluster(CB_HOST, options)\n", + " cluster.wait_until_ready(timedelta(seconds=5))\n", + " logging.info(\"Successfully connected to Couchbase\")\n", + "except Exception as e:\n", + " raise ConnectionError(f\"Failed to connect to Couchbase: {str(e)}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "c7a71bd4", + "metadata": {}, + "source": [ + "## Setting Up Collections in Couchbase\n", + "\n", + "The setup_collection() function handles creating and configuring the hierarchical data organization in Couchbase:\n", + "\n", + "1. Bucket Creation:\n", + " - Checks if specified bucket exists, creates it if not\n", + " - Sets bucket properties like RAM quota (1024MB) and replication (disabled)\n", + " - Note: You will not be able to create a bucket on Capella\n", + "\n", + "2. Scope Management: \n", + " - Verifies if requested scope exists within bucket\n", + " - Creates new scope if needed (unless it's the default \"_default\" scope)\n", + "\n", + "3. Collection Setup:\n", + " - Checks for collection existence within scope\n", + " - Creates collection if it doesn't exist\n", + " - Waits 2 seconds for collection to be ready\n", + "\n", + "Additional Tasks:\n", + "- Clears any existing documents for clean state\n", + "- Implements comprehensive error handling and logging\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "d9494a6f", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2025-09-23 11:44:23,574 - INFO - Bucket 'query-vector-search-testing' exists.\n", + "2025-09-23 11:44:23,581 - INFO - Collection 'pydantic_ai' already exists. Skipping creation.\n", + "2025-09-23 11:44:25,657 - INFO - All documents cleared from the collection.\n" + ] + }, + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def setup_collection(cluster, bucket_name, scope_name, collection_name):\n", + " try:\n", + " # Check if bucket exists, create if it doesn't\n", + " try:\n", + " bucket = cluster.bucket(bucket_name)\n", + " logging.info(f\"Bucket '{bucket_name}' exists.\")\n", + " except Exception as e:\n", + " logging.info(f\"Bucket '{bucket_name}' does not exist. Creating it...\")\n", + " bucket_settings = CreateBucketSettings(\n", + " name=bucket_name,\n", + " bucket_type='couchbase',\n", + " ram_quota_mb=1024,\n", + " flush_enabled=True,\n", + " num_replicas=0\n", + " )\n", + " cluster.buckets().create_bucket(bucket_settings)\n", + " time.sleep(2) # Wait for bucket creation to complete and become available\n", + " bucket = cluster.bucket(bucket_name)\n", + " logging.info(f\"Bucket '{bucket_name}' created successfully.\")\n", + "\n", + " bucket_manager = bucket.collections()\n", + "\n", + " # Check if scope exists, create if it doesn't\n", + " scopes = bucket_manager.get_all_scopes()\n", + " scope_exists = any(scope.name == scope_name for scope in scopes)\n", + " \n", + " if not scope_exists and scope_name != \"_default\":\n", + " logging.info(f\"Scope '{scope_name}' does not exist. Creating it...\")\n", + " bucket_manager.create_scope(scope_name)\n", + " logging.info(f\"Scope '{scope_name}' created successfully.\")\n", + "\n", + " # Check if collection exists, create if it doesn't\n", + " collections = bucket_manager.get_all_scopes()\n", + " collection_exists = any(\n", + " scope.name == scope_name and collection_name in [col.name for col in scope.collections]\n", + " for scope in collections\n", + " )\n", + "\n", + " if not collection_exists:\n", + " logging.info(f\"Collection '{collection_name}' does not exist. Creating it...\")\n", + " bucket_manager.create_collection(scope_name, collection_name)\n", + " logging.info(f\"Collection '{collection_name}' created successfully.\")\n", + " else:\n", + " logging.info(f\"Collection '{collection_name}' already exists. Skipping creation.\")\n", + "\n", + " # Wait for collection to be ready\n", + " collection = bucket.scope(scope_name).collection(collection_name)\n", + " time.sleep(2) # Give the collection time to be ready for queries\n", + "\n", + " # Clear all documents in the collection\n", + " try:\n", + " query = f\"DELETE FROM `{bucket_name}`.`{scope_name}`.`{collection_name}`\"\n", + " cluster.query(query).execute()\n", + " logging.info(\"All documents cleared from the collection.\")\n", + " except Exception as e:\n", + " logging.warning(f\"Error while clearing documents: {str(e)}. The collection might be empty.\")\n", + "\n", + " return collection\n", + " except Exception as e:\n", + " raise RuntimeError(f\"Error setting up collection: {str(e)}\")\n", + " \n", + "setup_collection(cluster, CB_BUCKET_NAME, SCOPE_NAME, COLLECTION_NAME)\n" + ] + }, + { + "cell_type": "markdown", + "id": "c52f6a2a", + "metadata": {}, + "source": [ + "# Creating OpenAI Embeddings\n", + "Embeddings are at the heart of semantic search. They are numerical representations of text that capture the semantic meaning of the words and phrases. Unlike traditional keyword-based search, which looks for exact matches, embeddings allow our search engine to understand the context and nuances of language, enabling it to retrieve documents that are semantically similar to the query, even if they don't contain the exact keywords. By creating embeddings using OpenAI, we equip our search engine with the ability to understand and process natural language in a way that's much closer to how humans understand language. This step transforms our raw text data into a format that the search engine can use to find and rank relevant documents.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "162e5be5", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2025-09-23 11:44:28,895 - INFO - Successfully created OpenAIEmbeddings\n" + ] + } + ], + "source": [ + "try:\n", + " embeddings = OpenAIEmbeddings(openai_api_key=OPENAI_API_KEY, model='text-embedding-3-small')\n", + " logging.info(\"Successfully created OpenAIEmbeddings\")\n", + "except Exception as e:\n", + " raise ValueError(f\"Error creating OpenAIEmbeddings: {str(e)}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "2f67cece", + "metadata": {}, + "source": [ + "# Setting Up the Couchbase Query Vector Store\n", + "A vector store is where we'll keep our embeddings. The query vector store is specifically designed to handle embeddings and perform similarity searches. When a user inputs a query, GSI converts the query into an embedding and compares it against the embeddings stored in the vector store. This allows the engine to find documents that are semantically similar to the query, even if they don't contain the exact same words. By setting up the vector store in Couchbase, we create a powerful tool that enables us to understand and retrieve information based on the meaning and context of the query, rather than just the specific words used.\n", + "\n", + "The vector store requires a distance metric to determine how similarity between vectors is calculated. This is crucial for accurate semantic search results as different distance metrics can yield different similarity rankings. Some of the supported Distance strategies are dot, l2, euclidean, cosine, l2_squared, euclidean_squared. In our implementation we will use cosine which is particularly effective for text embeddings.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "2603a98a", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2025-09-23 11:44:32,820 - INFO - Successfully created vector store\n" + ] + } + ], + "source": [ + "try:\n", + " vector_store = CouchbaseQueryVectorStore(\n", + " cluster=cluster,\n", + " bucket_name=CB_BUCKET_NAME,\n", + " scope_name=SCOPE_NAME,\n", + " collection_name=COLLECTION_NAME,\n", + " embedding = embeddings,\n", + " distance_metric=DistanceStrategy.COSINE\n", + " )\n", + " logging.info(\"Successfully created vector store\")\n", + "except Exception as e:\n", + " raise ValueError(f\"Failed to create vector store: {str(e)}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "b18592f1", + "metadata": {}, + "source": [ + "# Load the BBC News Dataset\n", + "To build a search engine, we need data to search through. We use the BBC News dataset from RealTimeData, which provides real-world news articles. This dataset contains news articles from BBC covering various topics and time periods. Loading the dataset is a crucial step because it provides the raw material that our search engine will work with. The quality and diversity of the news articles make it an excellent choice for testing and refining our search engine, ensuring it can handle real-world news content effectively.\n", + "\n", + "The BBC News dataset allows us to work with authentic news articles, enabling us to build and test a search engine that can effectively process and retrieve relevant news content. The dataset is loaded using the Hugging Face datasets library, specifically accessing the \"RealTimeData/bbc_news_alltime\" dataset with the \"2024-12\" version.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "2bfb848c", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2025-09-23 11:44:53,772 - INFO - Successfully loaded the BBC News dataset with 2687 rows.\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Loaded the BBC News dataset with 2687 rows\n" + ] + } + ], + "source": [ + "try:\n", + " news_dataset = load_dataset(\n", + " \"RealTimeData/bbc_news_alltime\", \"2024-12\", split=\"train\"\n", + " )\n", + " print(f\"Loaded the BBC News dataset with {len(news_dataset)} rows\")\n", + " logging.info(f\"Successfully loaded the BBC News dataset with {len(news_dataset)} rows.\")\n", + "except Exception as e:\n", + " raise ValueError(f\"Error loading the BBC News dataset: {str(e)}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "65b1a9bc", + "metadata": {}, + "source": [ + "## Cleaning up the Data\n", + "We will use the content of the news articles for our RAG system.\n", + "\n", + "The dataset contains a few duplicate records. We are removing them to avoid duplicate results in the retrieval stage of our RAG system.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "81457c3a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "We have 1749 unique articles in our database.\n" + ] + } + ], + "source": [ + "news_articles = news_dataset[\"content\"]\n", + "unique_articles = set()\n", + "for article in news_articles:\n", + " if article:\n", + " unique_articles.add(article)\n", + "unique_news_articles = list(unique_articles)\n", + "print(f\"We have {len(unique_news_articles)} unique articles in our database.\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "aca7e9e3", + "metadata": {}, + "source": [ + "## Saving Data to the Vector Store\n", + "To efficiently handle the large number of articles, we process them in batches of articles at a time. This batch processing approach helps manage memory usage and provides better control over the ingestion process.\n", + "\n", + "We first filter out any articles that exceed 50,000 characters to avoid potential issues with token limits. Then, using the vector store's add_texts method, we add the filtered articles to our vector database. The batch_size parameter controls how many articles are processed in each iteration.\n", + "\n", + "This approach offers several benefits:\n", + "1. Memory Efficiency: Processing in smaller batches prevents memory overload\n", + "2. Progress Tracking: Easier to monitor and track the ingestion progress\n", + "3. Resource Management: Better control over CPU and network resource utilization\n", + "\n", + "We use a conservative batch size of 100 to ensure reliable operation.\n", + "The optimal batch size depends on many factors including:\n", + "- Document sizes being inserted\n", + "- Available system resources\n", + "- Network conditions\n", + "- Concurrent workload\n", + "\n", + "Consider measuring performance with your specific workload before adjusting.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "7d85286e", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2025-09-23 11:46:08,521 - INFO - Document ingestion completed successfully.\n" + ] + } + ], + "source": [ + "batch_size = 100\n", + "\n", + "# Automatic Batch Processing\n", + "articles = [article for article in unique_news_articles if article and len(article) <= 50000]\n", + "\n", + "try:\n", + " vector_store.add_texts(\n", + " texts=articles,\n", + " batch_size=batch_size\n", + " )\n", + " logging.info(\"Document ingestion completed successfully.\")\n", + "except Exception as e:\n", + " raise ValueError(f\"Failed to save documents to vector store: {str(e)}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "0b66ca05", + "metadata": {}, + "source": [ + "# Perform Semantic Search\n", + "Semantic search in Couchbase involves converting queries and documents into vector representations using an embeddings model. These vectors capture the semantic meaning of the text and are stored directly in Couchbase. When a query is made, Couchbase performs a similarity search by comparing the query vector against the stored document vectors. The similarity metric used for this comparison is configurable, allowing flexibility in how the relevance of documents is determined. Common metrics include cosine similarity, Euclidean distance, or dot product, but other metrics can be implemented based on specific use cases. Different embedding models like BERT, Word2Vec, or GloVe can also be used depending on the application's needs, with the vectors generated by these models stored and searched within Couchbase itself.\n", + "\n", + "In the provided code, the search process begins by recording the start time, followed by executing the `similarity_search_with_score` method of the `CouchbaseQueryVectorStore`. This method searches Couchbase for the most relevant documents based on the vector similarity to the query. The search results include the document content and the distance that reflects how closely each document aligns with the query in the defined semantic space. The time taken to perform this search is then calculated and logged, and the results are displayed, showing the most relevant documents along with their similarity scores. This approach leverages Couchbase as both a storage and retrieval engine for vector data, enabling efficient and scalable semantic searches. The integration of vector storage and search capabilities within Couchbase allows for sophisticated semantic search operations without relying on external services for vector storage or comparison." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "be49ce6f", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2025-09-23 11:46:56,960 - INFO - Semantic search completed in 1.69 seconds\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Semantic Search Results (completed in 1.69 seconds):\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.2955, Text: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", + "\n", + "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affected by the worst run of results in his entire managerial career. In an interview with former Italy international Luca Toni for Amazon Prime Sport before Wednesday's Champions League defeat by Juventus, Guardiola touched on the personal impact City's sudden downturn in form has had. Guardiola said his state of mind was \"ugly\", that his sleep was \"worse\" and he was eating lighter as his digestion had suffered. City go into Sunday's derby against Manchester United at Etihad Stadium having won just one of their past 10 games. The Juventus loss means there is a chance they may not even secure a play-off spot in the Champions League. Asked to elaborate on his comments to Toni, Guardiola said: \"I'm fine. \"In our jobs we always want to do our best or the best as possible. When that doesn't happen you are more uncomfortable than when the situation is going well, always that happened. \"In good moments I am happier but when I get to the next game I am still concerned about what I have to do. There is no human being that makes an activity and it doesn't matter how they do.\" Guardiola said City have to defend better and \"avoid making mistakes at both ends\". To emphasise his point, Guardiola referred back to the third game of City's current run, against a Sporting side managed by Ruben Amorim, who will be in the United dugout at the weekend. City dominated the first half in Lisbon, led thanks to Phil Foden's early effort and looked to be cruising. Instead, they conceded three times in 11 minutes either side of half-time as Sporting eventually ran out 4-1 winners. \"I would like to play the game like we played in Lisbon on Sunday, believe me,\" said Guardiola, who is facing the prospect of only having three fit defenders for the derby as Nathan Ake and Manuel Akanji try to overcome injury concerns. If there is solace for City, it comes from the knowledge United are not exactly flying. Their comeback Europa League victory against Viktoria Plzen on Thursday was their third win of Amorim's short reign so far but only one of those successes has come in the Premier League, where United have lost their past two games against Arsenal and Nottingham Forest. Nevertheless, Guardiola can see improvements already on the red side of the city. \"It's already there,\" he said. \"You see all the patterns, the movements, the runners and the pace. He will do a good job at United, I'm pretty sure of that.\"\n", + "\n", + "Guardiola says skipper Kyle Walker has been offered support by the club after the City defender highlighted the racial abuse he had received on social media in the wake of the Juventus trip. \"It's unacceptable,\" he said. \"Not because it's Kyle - for any human being. \"Unfortunately it happens many times in the real world. It is not necessary to say he has the support of the entire club. It is completely unacceptable and we give our support to him.\"\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.3100, Text: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", + "\n", + "The former Barcelona and Bayern Munich boss has won 15 major trophies since taking charge of City in 2016.\n", + "\n", + "The 53-year-old Spaniard was approached in the summer about the possibility of becoming England manager, but last month signed a two-year contract extension with City until 2027.\n", + "\n", + "Speaking to celebrity chef Dani Garcia on YouTube, Guardiola did not indicate when he intends to step down at City but said he would not return to club football - in the Premier League or overseas.\n", + "\n", + "\"I'm not going to manage another team,\" he said.\n", + "\n", + "\"I'm not talking about the long-term future, but what I'm not going to do is leave Manchester City, go to another country, and do the same thing as now.\n", + "\n", + "\"I wouldn't have the energy. The thought of starting somewhere else, all the process of training and so on. No, no, no. Maybe a national team, but that's different.\n", + "\n", + "\"I want to leave it and go and play golf, but I can't [if he takes a club job]. I think stopping would do me good.\"\n", + "\n", + "City have won just once since Guardiola extended his contract - and once in nine games since beating Southampton on 26 October.\n", + "\n", + "That victory came at home to Nottingham Forest last Wednesday, but was followed by a 2-2 draw at Crystal Palace at the weekend.\n", + "\n", + "The Blues visit Juventus next in the Champions League on Wednesday (20:00 GMT), before hosting Manchester United in the Premier League on Sunday (16:30).\n", + "\n", + "\"Right now we are not in the position - when we have had the results of the last seven, eight games - to talk about winning games in plural,\" said Guardiola at his pre-match news conference.\n", + "\n", + "\"We have to win the game and not look at what happens in the next one yet.\"\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.3311, Text: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. 'I am not good enough' - Guardiola says he must find a 'solution' after derby loss\n", + "\n", + "Pep Guardiola says his sleep has suffered during Manchester City's deepening crisis, so he will not be helped by a nightmarish conclusion to one of the most stunning defeats of his long reign. Guardiola looked agitated, animated and on edge even after City led the Manchester derby through Josko Gvardiol's 36th-minute header, his reaction to the goal one of almost disdain that it came via a deflected cross as opposed to in his purist style. He sat alone with his eyes closed sipping from a water bottle before the resumption of the second half, then was denied even the respite of victory when Manchester United gave this largely dismal derby a dramatic conclusion it barely deserved with a remarkable late comeback. First, with 88 minutes on the clock, Matheus Nunes presented Amad Diallo with the ball before compounding his error by flattening the forward as he made an attempt to recover his mistake. Bruno Fernandes completed the formalities from the penalty spot. Worse was to come two minutes later when Lisandro Martinez's routine long ball caught City's defence inexplicably statuesque. Goalkeeper Ederson's positioning was awry, allowing the lively Diallo to pounce from an acute angle to leave Guardiola and his players stunned. It was the latest into any game, 88 minutes, that reigning Premier League champions had led then lost. It was also the first time City had lost a game they were leading so late on. And in a sign of City's previous excellence that is now being challenged, they have only lost four of 105 Premier League home games under Guardiola in which they have been ahead at half-time, winning 94 and drawing seven. Guardiola delivered a brutal self-analysis as he told Match of the Day: \"I am not good enough. I am the boss. I am the manager. I have to find solutions and so far I haven't. That's the reality. \"Not much else to say. No defence. Manchester United were incredibly persistent. We have not lost eight games in two seasons. We can't defend that.\"\n", + "\n", + "Manchester City manager Pep Guardiola in despair during the derby defeat to Manchester United\n", + "\n", + "Guardiola suggested the serious renewal will wait until the summer but the red flags have been appearing for weeks in the sudden and shocking decline of a team that has lost the aura of invincibility that left many opponents beaten before kick-off in previous years. He has had stated City must \"survive\" this season - whatever qualifies as survival for a club of such rich ambition - but the quest for a record fifth successive Premier League title is surely over as they lie nine points behind leaders Liverpool having played a game more. Their Champions League aspirations are also in jeopardy after another loss, this time against Juventus in Turin. City's squad has been allowed to grow too old together. The insatiable thirst for success seems to have gone, the scales of superiority have fallen away and opponents now sense vulnerability right until the final whistle, as United did here. The manner in which United were able, and felt able, to snatch this victory drove right to the heart of how City, and Guardiola, are allowing opponents to prey on their downfall. Guardiola has every reason to cite injuries, most significantly to Rodri and also John Stones as well as others, but this cannot be used an excuse for such a dramatic decline in standards, allied to the appearance of a soft underbelly that is so easily exploited. And City's rebuild will not be a quick fix. With every performance, every defeat, the scale of what lies in front of Guardiola becomes more obvious - and daunting. Manchester City's fans did their best to reassure Guardiola of their faith in him with a giant Barcelona-inspired banner draped from the stands before kick-off emblazoned with his image reading \"Més que un entrenador\" - \"More Than A Coach\". And Guardiola will now need to be more than a coach than at any time in his career. He will have the finances but it will be done with City's challengers also strengthening. Kevin de Bruyne, 34 in June, lasted 68 minutes here before he was substituted. Age and injuries are catching up with one of the greatest players of the Premier League era and he is unlikely to be at City next season. Mateo Kovacic, who replaced De Bruyne, is also 31 in May. Kyle Walker, 34, is being increasingly exposed. His most notable contribution here was an embarrassing collapse to the ground after the mildest head-to-head collision with Rasmus Hojlund. Ilkay Gundogan, another 34-year-old and a previous pillar of Guardiola's great successes, no longer has the legs or energy to exert influence. This looks increasingly like a season too far following his return from Barcelona. Flaws are also being exposed elsewhere, with previously reliable performers failing to hit previous standards. Phil Foden scored 27 goals and had 12 assists when he was Premier League Player of the Season last term. This year he has just three goals and two assists in 18 appearances in all competitions. He has no goals and just one assist in 11 Premier League games. Jack Grealish, who came on after 77 minutes against United, has not scored in a year for Manchester City, his last goal coming in a 2-2 draw against Crystal Palace on 16 December last year. He has, in the meantime, scored twice for England. Erling Haaland is also struggling as City lack creativity and cutting edge. He has three goals in his past 11 Premier League games after scoring 10 in his first five. And in another indication of City's impotence, and their reliance on Haaland, defender Gvardiol's goal against United was his fourth this season, making him their second highest scorer in all competitions behind the Norwegian striker, who has 18. Goalkeeper Ederson, so reliable for so long, has already been dropped once this season and did not cover himself in glory for United's winner. Guardiola, with that freshly signed two-year contract, insists he \"wants it\" as he treads on this alien territory of failure. He will be under no illusions about the size of the job in front of him as he placed his head in his hands in anguish after yet another damaging and deeply revealing defeat. City and Guardiola are in new, unforgiving territory.\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.3474, Text: 'Self-doubt, errors & big changes' - inside the crisis at Man City\n", + "\n", + "Pep Guardiola has not been through a moment like this in his managerial career. Manchester City have lost nine matches in their past 12 - as many defeats as they had suffered in their previous 106 fixtures. At the end of October, City were still unbeaten at the top of the Premier League and favourites to win a fifth successive title. Now they are seventh, 12 points behind leaders Liverpool having played a game more. It has been an incredible fall from grace and left people trying to work out what has happened - and whether Guardiola can make it right. After discussing the situation with those who know him best, I have taken a closer look at the future - both short and long term - and how the current crisis at Man City is going to be solved.\n", + "\n", + "Pep Guardiola's Man City have lost nine of their past 12 matches\n", + "\n", + "Guardiola has also been giving it a lot of thought. He has not been sleeping very well, as he has said, and has not been himself at times when talking to the media. He has been talking to a lot of people about what is going on as he tries to work out the reasons for City's demise. Some reasons he knows, others he still doesn't. What people perhaps do not realise is Guardiola hugely doubts himself and always has. He will be thinking \"I'm not going to be able to get us out of this\" and needs the support of people close to him to push away those insecurities - and he has that. He is protected by his people who are very aware, like he is, that there are a lot of people that want City to fail. It has been a turbulent time for Guardiola. Remember those marks he had on his head after the 3-3 draw with Feyenoord in the Champions League? He always scratches his head, it is a gesture of nervousness. Normally nothing happens but on that day one of his nails was far too sharp so, after talking to the players in the changing room where he scratched his head because of his usual agitated gesturing, he went to the news conference. His right-hand man Manel Estiarte sent him photos in a message saying \"what have you got on your head?\", but by the time Guardiola returned to the coaching room there was hardly anything there again. He started that day with a cover on his nose after the same thing happened at the training ground the day before. Guardiola was having a footballing debate with Kyle Walker about positional stuff and marked his nose with that same nail. There was also that remarkable news conference after the Manchester derby when he said \"I don't know what to do\". That is partly true and partly not true. Ignore the fact Guardiola suggested he was \"not good enough\". He actually meant he was not good enough to resolve the situation with the group of players he has available and with all the other current difficulties. There are obviously logical explanations for the crisis and the first one has been talked about many times - the absence of injured midfielder Rodri. You know the game Jenga? When you take the wrong piece out, the whole tower collapses. That is what has happened here. It is normal for teams to have an over-reliance on one player if he is the best in the world in his position. And you cannot calculate the consequences of an injury that rules someone like Rodri out for the season. City are a team, like many modern ones, in which the holding midfielder is a key element to the construction. So, when you take Rodri out, it is difficult to hold it together. There were Plan Bs - John Stones, Manuel Akanji, even Nathan Ake - but injuries struck. The big injury list has been out of the ordinary and the busy calendar has also played a part in compounding the issues. However, one factor even Guardiola cannot explain is the big uncharacteristic errors in almost every game from international players. Why did Matheus Nunes make that challenge to give away the penalty against Manchester United? Jack Grealish is sent on at the end to keep the ball and cannot do that. There are errors from Walker and other defenders. These are some of the best players in the world. Of course the players' mindset is important, and confidence is diminishing. Wrong decisions get taken so there is almost panic on the pitch instead of calm. There are also players badly out of form who are having to play because of injuries. Walker is now unable to hide behind his pace, I'm not sure Kevin de Bruyne is ever getting back to the level he used to be at, Bernardo Silva and Ilkay Gundogan do not have time to rest, Grealish is not playing at his best. Some of these players were only meant to be playing one game a week but, because of injuries, have played 12 games in 40 days. It all has a domino effect. One consequence is that Erling Haaland isn't getting the service to score. But the Norwegian still remains City's top-scorer with 13. Defender Josko Gvardiol is next on the list with just four. The way their form has been analysed inside the City camp is there have only been three games where they deserved to lose (Liverpool, Bournemouth and Aston Villa). But of course it is time to change the dynamic.\n", + "\n", + "Guardiola has never protected his players so much. He has not criticised them and is not going to do so. They have won everything with him. Instead of doing more with them, he has tried doing less. He has sometimes given them more days off to clear their heads, so they can reset - two days this week for instance. Perhaps the time to change a team is when you are winning, but no-one was suggesting Man City were about to collapse when they were top and unbeaten after nine league games. Some people have asked how bad it has to get before City make a decision on Guardiola. The answer is that there is no decision to be made. Maybe if this was Real Madrid, Barcelona or Juventus, the pressure from outside would be massive and the argument would be made that Guardiola has to go. At City he has won the lot, so how can anyone say he is failing? Yes, this is a crisis. But given all their problems, City's renewed target is finishing in the top four. That is what is in all their heads now. The idea is to recover their essence by improving defensive concepts that are not there and re-establishing the intensity they are known for. Guardiola is planning to use the next two years of his contract, which is expected to be his last as a club manager, to prepare a new Manchester City. When he was at the end of his four years at Barcelona, he asked two managers what to do when you feel people are not responding to your instructions. Do you go or do the players go? Sir Alex Ferguson and Rafael Benitez both told him that the players need to go. Guardiola did not listen because of his emotional attachment to his players back then and he decided to leave the Camp Nou because he felt the cycle was over. He will still protect his players now but there is not the same emotional attachment - so it is the players who are going to leave this time. It is likely City will look to replace five or six regular starters. Guardiola knows it is the end of an era and the start of a new one. Changes will not be immediate and the majority of the work will be done in the summer. But they are open to any opportunities in January - and a holding midfielder is one thing they need. In the summer City might want to get Spain's Martin Zubimendi from Real Sociedad and they know 60m euros (£50m) will get him. He said no to Liverpool last summer even though everything was agreed, but he now wants to move on and the Premier League is the target. Even if they do not get Zubimendi, that is the calibre of footballer they are after. A new Manchester City is on its way - with changes driven by Guardiola, incoming sporting director Hugo Viana and the football department.\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.3665, Text: Man City's Dias ruled out for 'three or four weeks'\n", + "\n", + "Ruben Dias has won 10 major trophies during his time at Manchester City\n", + "\n", + "Manchester City have suffered a fresh injury blow with manager Pep Guardiola confirming Portugal central defender Ruben Dias has been ruled out for \"three or four weeks\" with a muscle injury. Dias, who suffered the injury in Saturday's 2-1 defeat by Manchester United, will miss the entire festive programme and potentially the FA Cup third-round tie with Salford on 11 January. The 27-year-old also faces a battle to be fit for City's crucial Champions League trip to Paris St-Germain on 22 January. Dias has already missed seven games with a calf injury this season, adding to a defensive injury list that has seen John Stones, Nathan Ake, Manuel Akanji and Kyle Walker all ruled out at various points, while Ballon d'Or winner Rodri will miss the remainder of the domestic season after suffering a cruciate knee ligament injury. \"It's a muscular problem and he will be out for three to four weeks,\" said Guardiola. \"After 75 minutes against United he felt something. But he's so strong and wanted to stay on the pitch. Now he's injured.\" Guardiola confirmed Stones, Akanji and midfielder Mateo Kovacic have all trained this week and could feature at Aston Villa on Saturday (12:30 GMT), but said goalkeeper Ederson was \"a doubt\" with an unspecified problem. \"Ederson has been struggling with some niggles in his leg, he doesn't feel completely fine,\" said Guardiola. \"Ederson is so important for us.\" Amid City's current run of one win in 11 games, surprise has been expressed about Guardiola's use of youngsters James McAtee and Nico O'Reilly. City made a point of keeping both players despite numerous loan options. Yet McAtee has made just two substitute appearances - coming on in the last minute on both occasions - while O'Reilly is yet to make his league debut. But it seems they will stay at the club for the second half of the season, with Guardiola replying \"I don't think so\" when asked if players might leave during the January transfer window. The Spaniard said he is \"not a big fan\" of buying players in January but it is \"possible\" City will look to sign someone because \"the circumstances of this season have been special\".\n", + "\n", + "Guardiola's mood was so downbeat in the immediate aftermath of the United defeat it was easy to imagine he might conclude he was no longer capable of doing the job. He gave his players a couple of days off afterwards and was brighter when he spoke to journalists in his scheduled briefing before the Villa trip. \"We'd just finished a game that we lost in the circumstances and I was not happy,\" he said. \"I try to be honest about the feelings of my teams. We fell down six times [number of Premier League games without a win], we have to stand up seven. There is no alternative. \"I'm fine. I'm a normal person with feelings like all of us. When the situation is going well we are better but it's normal. I would not go to the press conference if we were 1-0 up and expressing something that I didn't feel.\" Former Villa forward Jack Grealish has not scored for City in over a year but Guardiola pointed out he is not the only attacking player struggling this season. He added: \"We are struggling to create a little bit up front, but always I am optimistic about my players that they are going to turn and perform well.\"\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.3818, Text: 'We have to find a way' - Guardiola vows to end relegation form\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. 'Worrying' and 'staggering' - Why do Manchester City keep conceding?\n", + "\n", + "Manchester City are currently in relegation form and there is little sign of it ending. Saturday's 2-1 defeat at Aston Villa left them joint bottom of the form table over the past eight games with just Southampton for company. Saints, at the foot of the Premier League, have the same number of points, four, as City over their past eight matches having won one, drawn one and lost six - the same record as the floundering champions. And if Southampton - who appointed Ivan Juric as their new manager on Saturday - get at least a point at Fulham on Sunday, City will be on the worst run in the division. Even Wolves, who sacked boss Gary O'Neil last Sunday and replaced him with Vitor Pereira, have earned double the number of points during the same period having played a game fewer. They are damning statistics for Pep Guardiola, even if he does have some mitigating circumstances with injuries to Ederson, Nathan Ake and Ruben Dias - who all missed the loss at Villa Park - and the long-term loss of midfield powerhouse Rodri. Guardiola was happy with Saturday's performance, despite defeat in Birmingham, but there is little solace to take at slipping further out of the title race. He may have needed to field a half-fit Manuel Akanji and John Stones at Villa Park but that does not account for City looking a shadow of their former selves. That does not justify the error Josko Gvardiol made to gift Jhon Duran a golden chance inside the first 20 seconds, or £100m man Jack Grealish again failing to have an impact on a game. There may be legitimate reasons for City's drop off, whether that be injuries, mental fatigue or just simply a team coming to the end of its lifecycle, but their form, which has plunged off a cliff edge, would have been unthinkable as they strolled to a fourth straight title last season. \"The worrying thing is the number of goals conceded,\" said ex-England captain Alan Shearer on BBC Match of the Day. \"The number of times they were opened up because of the lack of protection and legs in midfield was staggering. There are so many things that are wrong at this moment in time.\"\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. Man City 'have to find a way' to return to form - Guardiola\n", + "\n", + "Afterwards Guardiola was calm, so much so it was difficult to hear him in the news conference, a contrast to the frustrated figure he cut on the touchline. He said: \"It depends on us. The solution is bring the players back. We have just one central defender fit, that is difficult. We are going to try next game - another opportunity and we don't think much further than that. \"Of course there are more reasons. We concede the goals we don't concede in the past, we [don't] score the goals we score in the past. Football is not just one reason. There are a lot of little factors. \"Last season we won the Premier League, but we came here and lost. We have to think positive and I have incredible trust in the guys. Some of them have incredible pride and desire to do it. We have to find a way, step by step, sooner or later to find a way back.\" Villa boss Unai Emery highlighted City's frailties, saying he felt Villa could seize on the visitors' lack of belief. \"Manchester City are a little bit under the confidence they have normally,\" he said. \"The second half was different, we dominated and we scored. Through those circumstances they were feeling worse than even in the first half.\"\n", + "\n", + "Erling Haaland had one touch in the Villa box\n", + "\n", + "There are chinks in the armour never seen before at City under Guardiola and Erling Haaland conceded belief within the squad is low. He told TNT after the game: \"Of course, [confidence levels are] not the best. We know how important confidence is and you can see that it affects every human being. That is how it is, we have to continue and stay positive even though it is difficult.\" Haaland, with 76 goals in 83 Premier League appearances since joining City from Borussia Dortmund in 2022, had one shot and one touch in the Villa box. His 18 touches in the whole game were the lowest of all starting players and he has been self critical, despite scoring 13 goals in the top flight this season. Over City's last eight games he has netted just twice though, but Guardiola refused to criticise his star striker. He said: \"Without him we will be even worse but I like the players feeling that way. I don't agree with Erling. He needs to have the balls delivered in the right spots but he will fight for the next one.\"\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.4157, Text: Man City might miss out on Champions League - Guardiola\n", + "\n", + "Erling Haaland was part of the Manchester City side that won the Champions League for the first time in 2023\n", + "\n", + "Manchester City boss Pep Guardiola says the club are in danger of missing out on a place in next season's Champions League. City are currently in their 14th consecutive season in European football's most prestigious club competition. Only Arsenal between 1998 and 2017, and Manchester United between 1996 and 2014, have a longer record of qualifying among English clubs. City are seventh in the Premier League after 17 matches, four points behind Nottingham Forest in fourth and a point behind fifth-placed Bournemouth. England are currently top of Uefa's European Performance Spot table and well placed to secure a fifth place in next season's Champions League, although City would still not qualify on current standings. \"When I said before, people laughed,\" said Guardiola. \"They said, 'qualifying for the Champions League is not a big success'. \"But I know it because it happens with clubs in this country. They were dominant for many years and after they were many years not qualifying for the Champions League.\"\n", + "\n", + "Guardiola's side host Everton on Boxing Day, before a trip to Leicester on 29 December and a home match against West Ham on 4 January. Given all three opponents are in the bottom seven, it offers City a chance to improve on an appalling recent record of four points from eight games, which Guardiola acknowledges has left their lofty European ambitions in doubt. \"The one team that has been in the Champions League for the past years has been Manchester City,\" he added. \"Now we are at risk, of course we are. Definitely.\" Arsenal, Chelsea, Liverpool and Manchester United finished in the Premier League's top four from the 2005-06 season to 2008-09. At least three of them also occupied the top four spots for 15 successive campaigns until 2012. But United have spent five out of the past 11 seasons outside the Champions League. Arsenal spent six seasons out of the competition before returning last term. Liverpool missed out all but one year in seven from 2010, while Chelsea are in their second successive campaign outside Europe's elite. This term the threat to City comes from unexpected sources. As well as Forest and Bournemouth, Aston Villa are ahead of City, while Newcastle, Fulham and Brighton are also within a couple of points. \"There are a lot of contenders,\" said Guardiola, whose side have lost nine of their last 12 games in all competitions. \"For every club it is so important and if we are not winning games, we will be out. \"If we don't qualify it is because we don't deserve it, because we were not prepared and because we had a lot of problems and didn't solve them.\"\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.4705, Text: 'So happy he is back' - 'integral' De Bruyne 'one of best we've seen'\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. Match of the Day: How Kevin de Bruyne inspired Man City back to winning ways\n", + "\n", + "As Kevin de Bruyne made his way off the pitch after being replaced by Rico Lewis 16 minutes from the end of Manchester City's 3-0 win over Nottingham Forest, Pep Guardiola grabbed the Belgian and pulled him into an embrace. It was like a father offering affection to his son at a job well done. De Bruyne responded with a smile of satisfaction before continuing on his way to the top of the small terrace of City benches. A day earlier, Guardiola scoffed at the suggestions of Sky Sports duo Jamie Carragher and Gary Neville that there was some kind of rift between the boss and his star man. Why on earth, Guardiola countered, would he leave out someone who is capable of delivering moments like no-one else, even in his star-studded squad? Finally back on the pitch to start a Premier League game for the first time since August, De Bruyne proved exactly what Guardiola meant.\n", + "\n", + "Manchester City's Kevin de Bruyne has now scored two goals this season\n", + "\n", + "On a night Guardiola conceded City had to win, it was the Belgian's firm header that created an eighth-minute opener for Bernardo Silva as they went on to finally end their seven-match winless run and close the gap to leaders Liverpool to nine points. The goal from De Bruyne that followed was a thing of beauty as he backed away from Jeremy Doku as his fellow countryman ran with the ball, arriving in enough space to take the short pass and send his shot into the corner. Afforded extra space by his manager's decision to play Jack Grealish alongside him in a central position, De Bruyne schemed in the way he usually does. As chances came and went, he was playing some significant role. De Bruyne lasted 74 minutes, his longest match time since completing the full 90 minutes against Brentford on 14 September. He was on the bench when he was announced as man of the match, a decision received with enthusiasm by the City support. \"I am so happy he is back,\" said Guardiola. \"He played 75 fantastic minutes. \"He deserves the best because he's a lovely guy and has been massively important for so many years since he arrived.\"\n", + "\n", + "City are now unbeaten in their past 31 Premier League games with De Bruyne starting. He has been involved in 25 goals (nine goals, 16 assists) in those games. De Bruyne said: \"There have never been issues between me and Pep. He knows I've been struggling. It's painful and uncomfortable. \"Hopefully I can get back to my body with not much pain and then I'll be fine.\" However, a bit like the victory itself - tarnished by an injury to Manuel Akanji that may rule the Switzerland defender out of the weekend trip to Crystal Palace, and a hamstring problem for Nathan Ake who has already missed five weeks with a similar injury this season that Guardiola said \"doesn’t look good\" and makes him feel \"sad\" for the Dutchman - there was a caveat as the City boss assessed De Bruyne's contribution. \"He fought a lot and he prepared himself,\" he said. \"He is back to his physicality. The minutes he played at Anfield were really good. \"Last season he was out for many months, this season as well. We will see how he recovers after a long time injured and how he feels in three days.\" De Bruyne recorded four or more shots and created four or more chances for the third time in a Premier League game this season. Despite only starting five Premier League games, only Arsenal's Bukayo Saka has done so more often this term. Guardiola's fear must be that if he pushes De Bruyne too far too quickly, his body will let him down. Former Man City defender Micah Richards told BBC Match of the Day: \"He is a top-quality player and one of the best we have seen. He always manages to find space on the pitch.\" \"He has been integral to Man City's success over a number of years,\" added former City boss Stuart Pearce on Amazon Prime. \"He is the go-to player that sets Erling Haaland alight with his passing. He creates goals, he scores goals. \"If you were to pick one player out over the last eight or nine years De Bruyne would be at the top of almost everyone's list.\" It is a delicate balance given his team are still nine points adrift of Premier League leaders Liverpool and are also outside the Champions League top eight before next week's trip to Italy and a meeting with Juventus, after which City will only have January first-phase games remaining to ensure they secure qualification for the last-16 without needing to be bothered by February's play-off round. However, as with his team, De Bruyne's recovery had to start somewhere.\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.4831, Text: 'Life is not easy' - Haaland penalty miss sums up Man City crisis\n", + "\n", + "Manchester City striker Erling Haaland has now missed two of his 17 penalties taken in the Premier League\n", + "\n", + "Nothing seems to be going Manchester City's way at the moment - and star striker Erling Haaland is not a happy man. If there was any player currently in the Premier League you would hand the ball to for a penalty to win a match, it would be the prolific Norwegian. Not on this occasion, though. Looking to land a knockout blow, Haaland saw his second-half strike saved by spot-kick expert Jordan Pickford, as Manchester City's crisis continued with a 1-1 draw against Everton at Etihad Stadium. \"How do you react mentally?\" asked boss Pep Guardiola about the penalty miss. \"Life is not easy. Sport is not easy. When it happens, it is OK. \"There are still a lot of minutes to play and we had the chances afterwards. We created, incredible how they ran and fight. In some games it was not good but today well played.\" At the full-time whistle, there were a smattering of boos from home supporters at Etihad Stadium and a despondent Haaland ripped off his hair bobble and shirt before heading straight down the tunnel. As Haaland trudged off out of sight, Guardiola stood motionless on the pitch with the look of a man unable to find answers to their current crisis. City's all-conquering side have suffered a remarkable drop-off and now won just once in their last 13 games in all competitions. In the post-match news conference, the stunned Spaniard was barely audible in the responses to the questions fired his way. \"My body language was positive,\" he uttered. \"The team played really good. We had I don't know how many shots. The first half was brilliant.\"\n", + "\n", + "City have a lengthy injury list, but their downturn has coincided with Haaland's struggles in front of goal. The striker made a blistering start to the campaign looked to be on his way to setting more goalscoring records when netting 10 goals in his first five games, but has only managed three in 13 since. This has largely been down to underperformance on his expected goal (xG) rate, with statistics showing he has scored five fewer goals than he should have done during this recent run. There is also a stark contrast from his shot conversion rate from the first five games to his last 13, dropping from 38.5% to a lowly 6.4%. The penalty miss was Haaland's only shot on target in the contest - but Everton did not set out to pay him extra attention. Asked if there had been a special plan to handle the frontman, Dyche said: \"No. I was asked the other day when playing Chelsea about Cole Palmer and what a great player he is turning out to be. \"We know Haaland is too but I believe in the team. I don't look at one aspect of it, I look at the team, what they are doing and they have done a good job today.\"\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser.\n", + "\n", + "Pickford saved his seventh penalty in the Premier League since joining Everton in 2017\n", + "\n", + "After Toffees forward Iliman Ndiaye had cancelled out Bernardo Silva's opener, the defining moment came in the second half when Vitalii Mykolenko tripped Savinho in the box. During the VAR check, it was a battle of the minds with Everton captain Seamus Coleman receiving a yellow card for trying to put off Haaland, who had the ball in his hand. The booking ultimately appeared to be a well-earned one for the team and Pickford seemed to revel in his big moment, jumping up and down on his line, sticking his tongue out before stooping low to his right to push the ball away. Pickford has made big penalty saves for England on the international stage and this was seventh spot-kick stop since joining Everton in 2017. Such is City's luck at the moment that they managed to recycle the ball and Haaland headed in, but it was ruled out for offside, much to the joy of the travelling supporters. Haaland had scored 15 of 16 of his previous penalties in the league, with the other hitting the woodwork, while this was the first he had seen saved. \"It is good from the analysts to give him as much benefit of as many penalties as they can,\" Dyche said of Pickford's save. \"Credit to him for making the right decision in the spur of the moment.\" Former England goalkeeper Paul Robinson said on BBC Radio 5 Live: \"Pickford is good at putting pressure on strikers. Haaland must have felt the pressure there, this place went silent. \"It was a decent save but a really poor penalty. He didn't whip it around like a left-footer should. A poor penalty, but a good save.\" Ex-England defender Matt Upson added on Amazon Prime: \"It was brilliant save from Pickford. He relishes situations like this. He doesn't make life any easier for the penalty taker. \"But for Haaland to miss that penalty speaks a bit about where the confidence is in this team at the moment.\"\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. Dyche 'very pleased' with Everton defence in Man City draw\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.5382, Text: Amorim knows job in 'danger' without victories\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. 'I know that every manager is in danger'\n", + "\n", + "Manchester United head coach Ruben Amorim says the vast expense of bringing him in will not shield him from the sack if he fails to produce a winning team. While United sources stress there is total support for the new boss inside Old Trafford, recent results and performances have made some fans nervous. Away supporters booed their team at the final whistle of the 2-0 defeat by Wolves at Molineux on Boxing Day, and with many exiting quickly it left the players to acknowledge hundreds of empty yellow seats before heading to the tunnel. \"The manager of Manchester United can never, no matter what, be comfortable,\" said Amorim. \"You can argue I have been here one month and I've had four training [sessions], but we are not winning. That is the reality.\" Amorim has collected seven points from seven Premier League games since taking charge last month - only one more point than fellow Portuguese Vitor Pereira, who has won both his games since becoming Wolves boss. Five defeats in Amorim's first 10 games is the worst record of any new United manager since Walter Crickmer, who stepped up from being club secretary in the 1930s. It is not what was anticipated when chief executive Omar Berrada flew to Lisbon to offer Amorim the job in the wake of Erik ten Hag's dismissal on 28 October. United were so convinced in Amorim, they paid Sporting £10.6m in compensation to get him out of his contract. But Amorim does not believe that will save him if results do not improve. \"I know that if we don't win, regardless if they pay the buyout or not, every manager is in danger,\" he said. \"I like that because that is the job.\"\n", + "\n", + "The five-day gap between the home game with Newcastle on 30 December and an immensely difficult visit to old rivals and title favourites Liverpool on 5 January is the longest spell Amorim will have had to work with his players since his appointment. He will have another spare week after that, then three more midweek games to work with a squad United sources say is not expected to change much in personnel during the January transfer window, because of the club's tight Profit and Sustainability position. Evidently, it would have been far easier for such a dramatic transition to take place during the summer. Amorim did ask if his switch could be delayed until the end of the season but that request was rejected by Berrada. \"There's no point talking or thinking about that,\" said Amorim. \"I'm here and have to focus on the job. \"It's part of football to have these difficult moments. I already knew it was going to be tough. You expect to win more games, to have players with more confidence to sell the idea and to work and improve things. \"At this moment it's really hard. We have to survive to have time and then to improve the team.\"\n", + "\n", + "Manchester United head coach Ruben Amorim has won four and lost five of his first 10 games in charge\n", + "\n", + "Former United star Cristiano Ronaldo has backed Portuguese compatriot Amorim to turn the club's fortunes around. Ronaldo's second spell at Old Trafford ended in November 2022 following an explosive TV interview with presenter and journalist Piers Morgan. \"He [Amorim] did a fantastic job in Portugal with my [club] Sporting,\" said Ronaldo. \"But the Premier League is a different beast, the most competitive league in the world. I knew that it would be tough and they will continue the storm. \"But the storm will finish and the sun will rise. Things crossed, it will be good with him and I hope the best for Manchester United because it is a club I still love.\" Al Nassr forward Ronaldo, 39, who was speaking at the Globe Soccer Awards in Dubai where he was named the Best Middle East player of 2024, added: \"I will continue to say, the problem is not the coaches. \"It's like the aquarium and you have the fish inside and it's sick, and you take him out and fix the problem. \"If you put it back in the aquarium it will be sick again. This is the problem of Manchester United. It is the same.\"\n", + "--------------------------------------------------------------------------------\n" + ] + } + ], + "source": [ + "query = \"What was manchester city manager pep guardiola's reaction to the team's current form?\"\n", + "\n", + "try:\n", + " # Perform the semantic search\n", + " start_time = time.time()\n", + " search_results = vector_store.similarity_search_with_score(query, k=10)\n", + " search_elapsed_time = time.time() - start_time\n", + "\n", + " logging.info(f\"Semantic search completed in {search_elapsed_time:.2f} seconds\")\n", + "\n", + " # Display search results\n", + " print(f\"\\nSemantic Search Results (completed in {search_elapsed_time:.2f} seconds):\")\n", + " print(\"-\" * 80) # Add separator line\n", + " for doc, score in search_results:\n", + " print(f\"Score: {score:.4f}, Text: {doc.page_content}\")\n", + " print(\"-\" * 80) # Add separator between results\n", + "\n", + "except CouchbaseException as e:\n", + " raise RuntimeError(f\"Error performing semantic search: {str(e)}\")\n", + "except Exception as e:\n", + " raise RuntimeError(f\"Unexpected error: {str(e)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "f6b5ae0a", + "metadata": {}, + "source": [ + "\n", + "# Optimizing Vector Search with Global Secondary Index (GSI)\n", + "\n", + "While the above semantic search using similarity_search_with_score works effectively, we can significantly improve query performance by leveraging Global Secondary Index (GSI) in Couchbase.\n", + "\n", + "Couchbase offers three types of vector indexes, but for GSI-based vector search we focus on two main types:\n", + "\n", + "Hyperscale Vector Indexes (BHIVE)\n", + "- Best for pure vector searches - content discovery, recommendations, semantic search\n", + "- High performance with low memory footprint - designed to scale to billions of vectors\n", + "- Optimized for concurrent operations - supports simultaneous searches and inserts\n", + "- Use when: You primarily perform vector-only queries without complex scalar filtering\n", + "- Ideal for: Large-scale semantic search, recommendation systems, content discovery\n", + "\n", + "Composite Vector Indexes \n", + "- Best for filtered vector searches - combines vector search with scalar value filtering\n", + "- Efficient pre-filtering - scalar attributes reduce the vector comparison scope\n", + "- Use when: Your queries combine vector similarity with scalar filters that eliminate large portions of data\n", + "- Ideal for: Compliance-based filtering, user-specific searches, time-bounded queries\n", + "\n", + "Choosing the Right Index Type\n", + "- Start with Hyperscale Vector Index for pure vector searches and large datasets\n", + "- Use Composite Vector Index when scalar filters significantly reduce your search space\n", + "- Consider your dataset size: Hyperscale scales to billions, Composite works well for tens of millions to billions\n", + "\n", + "For more details, see the [Couchbase Vector Index documentation](https://preview.docs-test.couchbase.com/docs-server-DOC-12565_vector_search_concepts/server/current/vector-index/use-vector-indexes.html).\n", + "\n", + "\n", + "## Understanding Index Configuration (Couchbase 8.0 Feature)\n", + "\n", + "The index_description parameter controls how Couchbase optimizes vector storage and search performance through centroids and quantization:\n", + "\n", + "Format: `'IVF[],{PQ|SQ}'`\n", + "\n", + "Centroids (IVF - Inverted File):\n", + "- Controls how the dataset is subdivided for faster searches\n", + "- More centroids = faster search, slower training \n", + "- Fewer centroids = slower search, faster training\n", + "- If omitted (like IVF,SQ8), Couchbase auto-selects based on dataset size\n", + "\n", + "Quantization Options:\n", + "- SQ (Scalar Quantization): SQ4, SQ6, SQ8 (4, 6, or 8 bits per dimension)\n", + "- PQ (Product Quantization): PQx (e.g., PQ32x8)\n", + "- Higher values = better accuracy, larger index size\n", + "\n", + "Common Examples:\n", + "- IVF,SQ8 - Auto centroids, 8-bit scalar quantization (good default)\n", + "- IVF1000,SQ6 - 1000 centroids, 6-bit scalar quantization \n", + "- IVF,PQ32x8 - Auto centroids, 32 subquantizers with 8 bits\n", + "\n", + "For detailed configuration options, see the [Quantization & Centroid Settings](https://preview.docs-test.couchbase.com/docs-server-DOC-12565_vector_search_concepts/server/current/vector-index/hyperscale-vector-index.html#algo_settings).\n", + "\n", + "In the code below, we demonstrate creating a BHIVE index. This method takes an index type (BHIVE or COMPOSITE) and description parameter for optimization settings. Alternatively, GSI indexes can be created manually from the Couchbase UI." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "19596edc", + "metadata": {}, + "outputs": [], + "source": [ + "vector_store.create_index(index_type=IndexType.BHIVE, index_name=\"pydantic_ai_bhive_index\", index_description=\"IVF,SQ8\")\n", + "\n", + "# Note: To create a COMPOSITE index, the below code can be used.\n", + "# Choose based on your specific use case and query patterns. For this tutorial's news search scenario, either index type would work, but BHIVE might be more efficient for pure semantic search across news articles.\n", + "# vector_store.create_index(index_type=IndexType.COMPOSITE, index_name=\"pydantic_ai_composite_index\", index_description=\"IVF,SQ8\")" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "d1f4812c", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2025-09-23 11:47:17,010 - INFO - Semantic search completed in 1.28 seconds\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Semantic Search Results (completed in 1.28 seconds):\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.2955, Text: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", + "\n", + "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affected by the worst run of results in his entire managerial career. In an interview with former Italy international Luca Toni for Amazon Prime Sport before Wednesday's Champions League defeat by Juventus, Guardiola touched on the personal impact City's sudden downturn in form has had. Guardiola said his state of mind was \"ugly\", that his sleep was \"worse\" and he was eating lighter as his digestion had suffered. City go into Sunday's derby against Manchester United at Etihad Stadium having won just one of their past 10 games. The Juventus loss means there is a chance they may not even secure a play-off spot in the Champions League. Asked to elaborate on his comments to Toni, Guardiola said: \"I'm fine. \"In our jobs we always want to do our best or the best as possible. When that doesn't happen you are more uncomfortable than when the situation is going well, always that happened. \"In good moments I am happier but when I get to the next game I am still concerned about what I have to do. There is no human being that makes an activity and it doesn't matter how they do.\" Guardiola said City have to defend better and \"avoid making mistakes at both ends\". To emphasise his point, Guardiola referred back to the third game of City's current run, against a Sporting side managed by Ruben Amorim, who will be in the United dugout at the weekend. City dominated the first half in Lisbon, led thanks to Phil Foden's early effort and looked to be cruising. Instead, they conceded three times in 11 minutes either side of half-time as Sporting eventually ran out 4-1 winners. \"I would like to play the game like we played in Lisbon on Sunday, believe me,\" said Guardiola, who is facing the prospect of only having three fit defenders for the derby as Nathan Ake and Manuel Akanji try to overcome injury concerns. If there is solace for City, it comes from the knowledge United are not exactly flying. Their comeback Europa League victory against Viktoria Plzen on Thursday was their third win of Amorim's short reign so far but only one of those successes has come in the Premier League, where United have lost their past two games against Arsenal and Nottingham Forest. Nevertheless, Guardiola can see improvements already on the red side of the city. \"It's already there,\" he said. \"You see all the patterns, the movements, the runners and the pace. He will do a good job at United, I'm pretty sure of that.\"\n", + "\n", + "Guardiola says skipper Kyle Walker has been offered support by the club after the City defender highlighted the racial abuse he had received on social media in the wake of the Juventus trip. \"It's unacceptable,\" he said. \"Not because it's Kyle - for any human being. \"Unfortunately it happens many times in the real world. It is not necessary to say he has the support of the entire club. It is completely unacceptable and we give our support to him.\"\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.3100, Text: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", + "\n", + "The former Barcelona and Bayern Munich boss has won 15 major trophies since taking charge of City in 2016.\n", + "\n", + "The 53-year-old Spaniard was approached in the summer about the possibility of becoming England manager, but last month signed a two-year contract extension with City until 2027.\n", + "\n", + "Speaking to celebrity chef Dani Garcia on YouTube, Guardiola did not indicate when he intends to step down at City but said he would not return to club football - in the Premier League or overseas.\n", + "\n", + "\"I'm not going to manage another team,\" he said.\n", + "\n", + "\"I'm not talking about the long-term future, but what I'm not going to do is leave Manchester City, go to another country, and do the same thing as now.\n", + "\n", + "\"I wouldn't have the energy. The thought of starting somewhere else, all the process of training and so on. No, no, no. Maybe a national team, but that's different.\n", + "\n", + "\"I want to leave it and go and play golf, but I can't [if he takes a club job]. I think stopping would do me good.\"\n", + "\n", + "City have won just once since Guardiola extended his contract - and once in nine games since beating Southampton on 26 October.\n", + "\n", + "That victory came at home to Nottingham Forest last Wednesday, but was followed by a 2-2 draw at Crystal Palace at the weekend.\n", + "\n", + "The Blues visit Juventus next in the Champions League on Wednesday (20:00 GMT), before hosting Manchester United in the Premier League on Sunday (16:30).\n", + "\n", + "\"Right now we are not in the position - when we have had the results of the last seven, eight games - to talk about winning games in plural,\" said Guardiola at his pre-match news conference.\n", + "\n", + "\"We have to win the game and not look at what happens in the next one yet.\"\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.3311, Text: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. 'I am not good enough' - Guardiola says he must find a 'solution' after derby loss\n", + "\n", + "Pep Guardiola says his sleep has suffered during Manchester City's deepening crisis, so he will not be helped by a nightmarish conclusion to one of the most stunning defeats of his long reign. Guardiola looked agitated, animated and on edge even after City led the Manchester derby through Josko Gvardiol's 36th-minute header, his reaction to the goal one of almost disdain that it came via a deflected cross as opposed to in his purist style. He sat alone with his eyes closed sipping from a water bottle before the resumption of the second half, then was denied even the respite of victory when Manchester United gave this largely dismal derby a dramatic conclusion it barely deserved with a remarkable late comeback. First, with 88 minutes on the clock, Matheus Nunes presented Amad Diallo with the ball before compounding his error by flattening the forward as he made an attempt to recover his mistake. Bruno Fernandes completed the formalities from the penalty spot. Worse was to come two minutes later when Lisandro Martinez's routine long ball caught City's defence inexplicably statuesque. Goalkeeper Ederson's positioning was awry, allowing the lively Diallo to pounce from an acute angle to leave Guardiola and his players stunned. It was the latest into any game, 88 minutes, that reigning Premier League champions had led then lost. It was also the first time City had lost a game they were leading so late on. And in a sign of City's previous excellence that is now being challenged, they have only lost four of 105 Premier League home games under Guardiola in which they have been ahead at half-time, winning 94 and drawing seven. Guardiola delivered a brutal self-analysis as he told Match of the Day: \"I am not good enough. I am the boss. I am the manager. I have to find solutions and so far I haven't. That's the reality. \"Not much else to say. No defence. Manchester United were incredibly persistent. We have not lost eight games in two seasons. We can't defend that.\"\n", + "\n", + "Manchester City manager Pep Guardiola in despair during the derby defeat to Manchester United\n", + "\n", + "Guardiola suggested the serious renewal will wait until the summer but the red flags have been appearing for weeks in the sudden and shocking decline of a team that has lost the aura of invincibility that left many opponents beaten before kick-off in previous years. He has had stated City must \"survive\" this season - whatever qualifies as survival for a club of such rich ambition - but the quest for a record fifth successive Premier League title is surely over as they lie nine points behind leaders Liverpool having played a game more. Their Champions League aspirations are also in jeopardy after another loss, this time against Juventus in Turin. City's squad has been allowed to grow too old together. The insatiable thirst for success seems to have gone, the scales of superiority have fallen away and opponents now sense vulnerability right until the final whistle, as United did here. The manner in which United were able, and felt able, to snatch this victory drove right to the heart of how City, and Guardiola, are allowing opponents to prey on their downfall. Guardiola has every reason to cite injuries, most significantly to Rodri and also John Stones as well as others, but this cannot be used an excuse for such a dramatic decline in standards, allied to the appearance of a soft underbelly that is so easily exploited. And City's rebuild will not be a quick fix. With every performance, every defeat, the scale of what lies in front of Guardiola becomes more obvious - and daunting. Manchester City's fans did their best to reassure Guardiola of their faith in him with a giant Barcelona-inspired banner draped from the stands before kick-off emblazoned with his image reading \"Més que un entrenador\" - \"More Than A Coach\". And Guardiola will now need to be more than a coach than at any time in his career. He will have the finances but it will be done with City's challengers also strengthening. Kevin de Bruyne, 34 in June, lasted 68 minutes here before he was substituted. Age and injuries are catching up with one of the greatest players of the Premier League era and he is unlikely to be at City next season. Mateo Kovacic, who replaced De Bruyne, is also 31 in May. Kyle Walker, 34, is being increasingly exposed. His most notable contribution here was an embarrassing collapse to the ground after the mildest head-to-head collision with Rasmus Hojlund. Ilkay Gundogan, another 34-year-old and a previous pillar of Guardiola's great successes, no longer has the legs or energy to exert influence. This looks increasingly like a season too far following his return from Barcelona. Flaws are also being exposed elsewhere, with previously reliable performers failing to hit previous standards. Phil Foden scored 27 goals and had 12 assists when he was Premier League Player of the Season last term. This year he has just three goals and two assists in 18 appearances in all competitions. He has no goals and just one assist in 11 Premier League games. Jack Grealish, who came on after 77 minutes against United, has not scored in a year for Manchester City, his last goal coming in a 2-2 draw against Crystal Palace on 16 December last year. He has, in the meantime, scored twice for England. Erling Haaland is also struggling as City lack creativity and cutting edge. He has three goals in his past 11 Premier League games after scoring 10 in his first five. And in another indication of City's impotence, and their reliance on Haaland, defender Gvardiol's goal against United was his fourth this season, making him their second highest scorer in all competitions behind the Norwegian striker, who has 18. Goalkeeper Ederson, so reliable for so long, has already been dropped once this season and did not cover himself in glory for United's winner. Guardiola, with that freshly signed two-year contract, insists he \"wants it\" as he treads on this alien territory of failure. He will be under no illusions about the size of the job in front of him as he placed his head in his hands in anguish after yet another damaging and deeply revealing defeat. City and Guardiola are in new, unforgiving territory.\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.3474, Text: 'Self-doubt, errors & big changes' - inside the crisis at Man City\n", + "\n", + "Pep Guardiola has not been through a moment like this in his managerial career. Manchester City have lost nine matches in their past 12 - as many defeats as they had suffered in their previous 106 fixtures. At the end of October, City were still unbeaten at the top of the Premier League and favourites to win a fifth successive title. Now they are seventh, 12 points behind leaders Liverpool having played a game more. It has been an incredible fall from grace and left people trying to work out what has happened - and whether Guardiola can make it right. After discussing the situation with those who know him best, I have taken a closer look at the future - both short and long term - and how the current crisis at Man City is going to be solved.\n", + "\n", + "Pep Guardiola's Man City have lost nine of their past 12 matches\n", + "\n", + "Guardiola has also been giving it a lot of thought. He has not been sleeping very well, as he has said, and has not been himself at times when talking to the media. He has been talking to a lot of people about what is going on as he tries to work out the reasons for City's demise. Some reasons he knows, others he still doesn't. What people perhaps do not realise is Guardiola hugely doubts himself and always has. He will be thinking \"I'm not going to be able to get us out of this\" and needs the support of people close to him to push away those insecurities - and he has that. He is protected by his people who are very aware, like he is, that there are a lot of people that want City to fail. It has been a turbulent time for Guardiola. Remember those marks he had on his head after the 3-3 draw with Feyenoord in the Champions League? He always scratches his head, it is a gesture of nervousness. Normally nothing happens but on that day one of his nails was far too sharp so, after talking to the players in the changing room where he scratched his head because of his usual agitated gesturing, he went to the news conference. His right-hand man Manel Estiarte sent him photos in a message saying \"what have you got on your head?\", but by the time Guardiola returned to the coaching room there was hardly anything there again. He started that day with a cover on his nose after the same thing happened at the training ground the day before. Guardiola was having a footballing debate with Kyle Walker about positional stuff and marked his nose with that same nail. There was also that remarkable news conference after the Manchester derby when he said \"I don't know what to do\". That is partly true and partly not true. Ignore the fact Guardiola suggested he was \"not good enough\". He actually meant he was not good enough to resolve the situation with the group of players he has available and with all the other current difficulties. There are obviously logical explanations for the crisis and the first one has been talked about many times - the absence of injured midfielder Rodri. You know the game Jenga? When you take the wrong piece out, the whole tower collapses. That is what has happened here. It is normal for teams to have an over-reliance on one player if he is the best in the world in his position. And you cannot calculate the consequences of an injury that rules someone like Rodri out for the season. City are a team, like many modern ones, in which the holding midfielder is a key element to the construction. So, when you take Rodri out, it is difficult to hold it together. There were Plan Bs - John Stones, Manuel Akanji, even Nathan Ake - but injuries struck. The big injury list has been out of the ordinary and the busy calendar has also played a part in compounding the issues. However, one factor even Guardiola cannot explain is the big uncharacteristic errors in almost every game from international players. Why did Matheus Nunes make that challenge to give away the penalty against Manchester United? Jack Grealish is sent on at the end to keep the ball and cannot do that. There are errors from Walker and other defenders. These are some of the best players in the world. Of course the players' mindset is important, and confidence is diminishing. Wrong decisions get taken so there is almost panic on the pitch instead of calm. There are also players badly out of form who are having to play because of injuries. Walker is now unable to hide behind his pace, I'm not sure Kevin de Bruyne is ever getting back to the level he used to be at, Bernardo Silva and Ilkay Gundogan do not have time to rest, Grealish is not playing at his best. Some of these players were only meant to be playing one game a week but, because of injuries, have played 12 games in 40 days. It all has a domino effect. One consequence is that Erling Haaland isn't getting the service to score. But the Norwegian still remains City's top-scorer with 13. Defender Josko Gvardiol is next on the list with just four. The way their form has been analysed inside the City camp is there have only been three games where they deserved to lose (Liverpool, Bournemouth and Aston Villa). But of course it is time to change the dynamic.\n", + "\n", + "Guardiola has never protected his players so much. He has not criticised them and is not going to do so. They have won everything with him. Instead of doing more with them, he has tried doing less. He has sometimes given them more days off to clear their heads, so they can reset - two days this week for instance. Perhaps the time to change a team is when you are winning, but no-one was suggesting Man City were about to collapse when they were top and unbeaten after nine league games. Some people have asked how bad it has to get before City make a decision on Guardiola. The answer is that there is no decision to be made. Maybe if this was Real Madrid, Barcelona or Juventus, the pressure from outside would be massive and the argument would be made that Guardiola has to go. At City he has won the lot, so how can anyone say he is failing? Yes, this is a crisis. But given all their problems, City's renewed target is finishing in the top four. That is what is in all their heads now. The idea is to recover their essence by improving defensive concepts that are not there and re-establishing the intensity they are known for. Guardiola is planning to use the next two years of his contract, which is expected to be his last as a club manager, to prepare a new Manchester City. When he was at the end of his four years at Barcelona, he asked two managers what to do when you feel people are not responding to your instructions. Do you go or do the players go? Sir Alex Ferguson and Rafael Benitez both told him that the players need to go. Guardiola did not listen because of his emotional attachment to his players back then and he decided to leave the Camp Nou because he felt the cycle was over. He will still protect his players now but there is not the same emotional attachment - so it is the players who are going to leave this time. It is likely City will look to replace five or six regular starters. Guardiola knows it is the end of an era and the start of a new one. Changes will not be immediate and the majority of the work will be done in the summer. But they are open to any opportunities in January - and a holding midfielder is one thing they need. In the summer City might want to get Spain's Martin Zubimendi from Real Sociedad and they know 60m euros (£50m) will get him. He said no to Liverpool last summer even though everything was agreed, but he now wants to move on and the Premier League is the target. Even if they do not get Zubimendi, that is the calibre of footballer they are after. A new Manchester City is on its way - with changes driven by Guardiola, incoming sporting director Hugo Viana and the football department.\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.3665, Text: Man City's Dias ruled out for 'three or four weeks'\n", + "\n", + "Ruben Dias has won 10 major trophies during his time at Manchester City\n", + "\n", + "Manchester City have suffered a fresh injury blow with manager Pep Guardiola confirming Portugal central defender Ruben Dias has been ruled out for \"three or four weeks\" with a muscle injury. Dias, who suffered the injury in Saturday's 2-1 defeat by Manchester United, will miss the entire festive programme and potentially the FA Cup third-round tie with Salford on 11 January. The 27-year-old also faces a battle to be fit for City's crucial Champions League trip to Paris St-Germain on 22 January. Dias has already missed seven games with a calf injury this season, adding to a defensive injury list that has seen John Stones, Nathan Ake, Manuel Akanji and Kyle Walker all ruled out at various points, while Ballon d'Or winner Rodri will miss the remainder of the domestic season after suffering a cruciate knee ligament injury. \"It's a muscular problem and he will be out for three to four weeks,\" said Guardiola. \"After 75 minutes against United he felt something. But he's so strong and wanted to stay on the pitch. Now he's injured.\" Guardiola confirmed Stones, Akanji and midfielder Mateo Kovacic have all trained this week and could feature at Aston Villa on Saturday (12:30 GMT), but said goalkeeper Ederson was \"a doubt\" with an unspecified problem. \"Ederson has been struggling with some niggles in his leg, he doesn't feel completely fine,\" said Guardiola. \"Ederson is so important for us.\" Amid City's current run of one win in 11 games, surprise has been expressed about Guardiola's use of youngsters James McAtee and Nico O'Reilly. City made a point of keeping both players despite numerous loan options. Yet McAtee has made just two substitute appearances - coming on in the last minute on both occasions - while O'Reilly is yet to make his league debut. But it seems they will stay at the club for the second half of the season, with Guardiola replying \"I don't think so\" when asked if players might leave during the January transfer window. The Spaniard said he is \"not a big fan\" of buying players in January but it is \"possible\" City will look to sign someone because \"the circumstances of this season have been special\".\n", + "\n", + "Guardiola's mood was so downbeat in the immediate aftermath of the United defeat it was easy to imagine he might conclude he was no longer capable of doing the job. He gave his players a couple of days off afterwards and was brighter when he spoke to journalists in his scheduled briefing before the Villa trip. \"We'd just finished a game that we lost in the circumstances and I was not happy,\" he said. \"I try to be honest about the feelings of my teams. We fell down six times [number of Premier League games without a win], we have to stand up seven. There is no alternative. \"I'm fine. I'm a normal person with feelings like all of us. When the situation is going well we are better but it's normal. I would not go to the press conference if we were 1-0 up and expressing something that I didn't feel.\" Former Villa forward Jack Grealish has not scored for City in over a year but Guardiola pointed out he is not the only attacking player struggling this season. He added: \"We are struggling to create a little bit up front, but always I am optimistic about my players that they are going to turn and perform well.\"\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.3818, Text: 'We have to find a way' - Guardiola vows to end relegation form\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. 'Worrying' and 'staggering' - Why do Manchester City keep conceding?\n", + "\n", + "Manchester City are currently in relegation form and there is little sign of it ending. Saturday's 2-1 defeat at Aston Villa left them joint bottom of the form table over the past eight games with just Southampton for company. Saints, at the foot of the Premier League, have the same number of points, four, as City over their past eight matches having won one, drawn one and lost six - the same record as the floundering champions. And if Southampton - who appointed Ivan Juric as their new manager on Saturday - get at least a point at Fulham on Sunday, City will be on the worst run in the division. Even Wolves, who sacked boss Gary O'Neil last Sunday and replaced him with Vitor Pereira, have earned double the number of points during the same period having played a game fewer. They are damning statistics for Pep Guardiola, even if he does have some mitigating circumstances with injuries to Ederson, Nathan Ake and Ruben Dias - who all missed the loss at Villa Park - and the long-term loss of midfield powerhouse Rodri. Guardiola was happy with Saturday's performance, despite defeat in Birmingham, but there is little solace to take at slipping further out of the title race. He may have needed to field a half-fit Manuel Akanji and John Stones at Villa Park but that does not account for City looking a shadow of their former selves. That does not justify the error Josko Gvardiol made to gift Jhon Duran a golden chance inside the first 20 seconds, or £100m man Jack Grealish again failing to have an impact on a game. There may be legitimate reasons for City's drop off, whether that be injuries, mental fatigue or just simply a team coming to the end of its lifecycle, but their form, which has plunged off a cliff edge, would have been unthinkable as they strolled to a fourth straight title last season. \"The worrying thing is the number of goals conceded,\" said ex-England captain Alan Shearer on BBC Match of the Day. \"The number of times they were opened up because of the lack of protection and legs in midfield was staggering. There are so many things that are wrong at this moment in time.\"\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. Man City 'have to find a way' to return to form - Guardiola\n", + "\n", + "Afterwards Guardiola was calm, so much so it was difficult to hear him in the news conference, a contrast to the frustrated figure he cut on the touchline. He said: \"It depends on us. The solution is bring the players back. We have just one central defender fit, that is difficult. We are going to try next game - another opportunity and we don't think much further than that. \"Of course there are more reasons. We concede the goals we don't concede in the past, we [don't] score the goals we score in the past. Football is not just one reason. There are a lot of little factors. \"Last season we won the Premier League, but we came here and lost. We have to think positive and I have incredible trust in the guys. Some of them have incredible pride and desire to do it. We have to find a way, step by step, sooner or later to find a way back.\" Villa boss Unai Emery highlighted City's frailties, saying he felt Villa could seize on the visitors' lack of belief. \"Manchester City are a little bit under the confidence they have normally,\" he said. \"The second half was different, we dominated and we scored. Through those circumstances they were feeling worse than even in the first half.\"\n", + "\n", + "Erling Haaland had one touch in the Villa box\n", + "\n", + "There are chinks in the armour never seen before at City under Guardiola and Erling Haaland conceded belief within the squad is low. He told TNT after the game: \"Of course, [confidence levels are] not the best. We know how important confidence is and you can see that it affects every human being. That is how it is, we have to continue and stay positive even though it is difficult.\" Haaland, with 76 goals in 83 Premier League appearances since joining City from Borussia Dortmund in 2022, had one shot and one touch in the Villa box. His 18 touches in the whole game were the lowest of all starting players and he has been self critical, despite scoring 13 goals in the top flight this season. Over City's last eight games he has netted just twice though, but Guardiola refused to criticise his star striker. He said: \"Without him we will be even worse but I like the players feeling that way. I don't agree with Erling. He needs to have the balls delivered in the right spots but he will fight for the next one.\"\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.4157, Text: Man City might miss out on Champions League - Guardiola\n", + "\n", + "Erling Haaland was part of the Manchester City side that won the Champions League for the first time in 2023\n", + "\n", + "Manchester City boss Pep Guardiola says the club are in danger of missing out on a place in next season's Champions League. City are currently in their 14th consecutive season in European football's most prestigious club competition. Only Arsenal between 1998 and 2017, and Manchester United between 1996 and 2014, have a longer record of qualifying among English clubs. City are seventh in the Premier League after 17 matches, four points behind Nottingham Forest in fourth and a point behind fifth-placed Bournemouth. England are currently top of Uefa's European Performance Spot table and well placed to secure a fifth place in next season's Champions League, although City would still not qualify on current standings. \"When I said before, people laughed,\" said Guardiola. \"They said, 'qualifying for the Champions League is not a big success'. \"But I know it because it happens with clubs in this country. They were dominant for many years and after they were many years not qualifying for the Champions League.\"\n", + "\n", + "Guardiola's side host Everton on Boxing Day, before a trip to Leicester on 29 December and a home match against West Ham on 4 January. Given all three opponents are in the bottom seven, it offers City a chance to improve on an appalling recent record of four points from eight games, which Guardiola acknowledges has left their lofty European ambitions in doubt. \"The one team that has been in the Champions League for the past years has been Manchester City,\" he added. \"Now we are at risk, of course we are. Definitely.\" Arsenal, Chelsea, Liverpool and Manchester United finished in the Premier League's top four from the 2005-06 season to 2008-09. At least three of them also occupied the top four spots for 15 successive campaigns until 2012. But United have spent five out of the past 11 seasons outside the Champions League. Arsenal spent six seasons out of the competition before returning last term. Liverpool missed out all but one year in seven from 2010, while Chelsea are in their second successive campaign outside Europe's elite. This term the threat to City comes from unexpected sources. As well as Forest and Bournemouth, Aston Villa are ahead of City, while Newcastle, Fulham and Brighton are also within a couple of points. \"There are a lot of contenders,\" said Guardiola, whose side have lost nine of their last 12 games in all competitions. \"For every club it is so important and if we are not winning games, we will be out. \"If we don't qualify it is because we don't deserve it, because we were not prepared and because we had a lot of problems and didn't solve them.\"\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.4705, Text: 'So happy he is back' - 'integral' De Bruyne 'one of best we've seen'\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. Match of the Day: How Kevin de Bruyne inspired Man City back to winning ways\n", + "\n", + "As Kevin de Bruyne made his way off the pitch after being replaced by Rico Lewis 16 minutes from the end of Manchester City's 3-0 win over Nottingham Forest, Pep Guardiola grabbed the Belgian and pulled him into an embrace. It was like a father offering affection to his son at a job well done. De Bruyne responded with a smile of satisfaction before continuing on his way to the top of the small terrace of City benches. A day earlier, Guardiola scoffed at the suggestions of Sky Sports duo Jamie Carragher and Gary Neville that there was some kind of rift between the boss and his star man. Why on earth, Guardiola countered, would he leave out someone who is capable of delivering moments like no-one else, even in his star-studded squad? Finally back on the pitch to start a Premier League game for the first time since August, De Bruyne proved exactly what Guardiola meant.\n", + "\n", + "Manchester City's Kevin de Bruyne has now scored two goals this season\n", + "\n", + "On a night Guardiola conceded City had to win, it was the Belgian's firm header that created an eighth-minute opener for Bernardo Silva as they went on to finally end their seven-match winless run and close the gap to leaders Liverpool to nine points. The goal from De Bruyne that followed was a thing of beauty as he backed away from Jeremy Doku as his fellow countryman ran with the ball, arriving in enough space to take the short pass and send his shot into the corner. Afforded extra space by his manager's decision to play Jack Grealish alongside him in a central position, De Bruyne schemed in the way he usually does. As chances came and went, he was playing some significant role. De Bruyne lasted 74 minutes, his longest match time since completing the full 90 minutes against Brentford on 14 September. He was on the bench when he was announced as man of the match, a decision received with enthusiasm by the City support. \"I am so happy he is back,\" said Guardiola. \"He played 75 fantastic minutes. \"He deserves the best because he's a lovely guy and has been massively important for so many years since he arrived.\"\n", + "\n", + "City are now unbeaten in their past 31 Premier League games with De Bruyne starting. He has been involved in 25 goals (nine goals, 16 assists) in those games. De Bruyne said: \"There have never been issues between me and Pep. He knows I've been struggling. It's painful and uncomfortable. \"Hopefully I can get back to my body with not much pain and then I'll be fine.\" However, a bit like the victory itself - tarnished by an injury to Manuel Akanji that may rule the Switzerland defender out of the weekend trip to Crystal Palace, and a hamstring problem for Nathan Ake who has already missed five weeks with a similar injury this season that Guardiola said \"doesn’t look good\" and makes him feel \"sad\" for the Dutchman - there was a caveat as the City boss assessed De Bruyne's contribution. \"He fought a lot and he prepared himself,\" he said. \"He is back to his physicality. The minutes he played at Anfield were really good. \"Last season he was out for many months, this season as well. We will see how he recovers after a long time injured and how he feels in three days.\" De Bruyne recorded four or more shots and created four or more chances for the third time in a Premier League game this season. Despite only starting five Premier League games, only Arsenal's Bukayo Saka has done so more often this term. Guardiola's fear must be that if he pushes De Bruyne too far too quickly, his body will let him down. Former Man City defender Micah Richards told BBC Match of the Day: \"He is a top-quality player and one of the best we have seen. He always manages to find space on the pitch.\" \"He has been integral to Man City's success over a number of years,\" added former City boss Stuart Pearce on Amazon Prime. \"He is the go-to player that sets Erling Haaland alight with his passing. He creates goals, he scores goals. \"If you were to pick one player out over the last eight or nine years De Bruyne would be at the top of almost everyone's list.\" It is a delicate balance given his team are still nine points adrift of Premier League leaders Liverpool and are also outside the Champions League top eight before next week's trip to Italy and a meeting with Juventus, after which City will only have January first-phase games remaining to ensure they secure qualification for the last-16 without needing to be bothered by February's play-off round. However, as with his team, De Bruyne's recovery had to start somewhere.\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.4831, Text: 'Life is not easy' - Haaland penalty miss sums up Man City crisis\n", + "\n", + "Manchester City striker Erling Haaland has now missed two of his 17 penalties taken in the Premier League\n", + "\n", + "Nothing seems to be going Manchester City's way at the moment - and star striker Erling Haaland is not a happy man. If there was any player currently in the Premier League you would hand the ball to for a penalty to win a match, it would be the prolific Norwegian. Not on this occasion, though. Looking to land a knockout blow, Haaland saw his second-half strike saved by spot-kick expert Jordan Pickford, as Manchester City's crisis continued with a 1-1 draw against Everton at Etihad Stadium. \"How do you react mentally?\" asked boss Pep Guardiola about the penalty miss. \"Life is not easy. Sport is not easy. When it happens, it is OK. \"There are still a lot of minutes to play and we had the chances afterwards. We created, incredible how they ran and fight. In some games it was not good but today well played.\" At the full-time whistle, there were a smattering of boos from home supporters at Etihad Stadium and a despondent Haaland ripped off his hair bobble and shirt before heading straight down the tunnel. As Haaland trudged off out of sight, Guardiola stood motionless on the pitch with the look of a man unable to find answers to their current crisis. City's all-conquering side have suffered a remarkable drop-off and now won just once in their last 13 games in all competitions. In the post-match news conference, the stunned Spaniard was barely audible in the responses to the questions fired his way. \"My body language was positive,\" he uttered. \"The team played really good. We had I don't know how many shots. The first half was brilliant.\"\n", + "\n", + "City have a lengthy injury list, but their downturn has coincided with Haaland's struggles in front of goal. The striker made a blistering start to the campaign looked to be on his way to setting more goalscoring records when netting 10 goals in his first five games, but has only managed three in 13 since. This has largely been down to underperformance on his expected goal (xG) rate, with statistics showing he has scored five fewer goals than he should have done during this recent run. There is also a stark contrast from his shot conversion rate from the first five games to his last 13, dropping from 38.5% to a lowly 6.4%. The penalty miss was Haaland's only shot on target in the contest - but Everton did not set out to pay him extra attention. Asked if there had been a special plan to handle the frontman, Dyche said: \"No. I was asked the other day when playing Chelsea about Cole Palmer and what a great player he is turning out to be. \"We know Haaland is too but I believe in the team. I don't look at one aspect of it, I look at the team, what they are doing and they have done a good job today.\"\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser.\n", + "\n", + "Pickford saved his seventh penalty in the Premier League since joining Everton in 2017\n", + "\n", + "After Toffees forward Iliman Ndiaye had cancelled out Bernardo Silva's opener, the defining moment came in the second half when Vitalii Mykolenko tripped Savinho in the box. During the VAR check, it was a battle of the minds with Everton captain Seamus Coleman receiving a yellow card for trying to put off Haaland, who had the ball in his hand. The booking ultimately appeared to be a well-earned one for the team and Pickford seemed to revel in his big moment, jumping up and down on his line, sticking his tongue out before stooping low to his right to push the ball away. Pickford has made big penalty saves for England on the international stage and this was seventh spot-kick stop since joining Everton in 2017. Such is City's luck at the moment that they managed to recycle the ball and Haaland headed in, but it was ruled out for offside, much to the joy of the travelling supporters. Haaland had scored 15 of 16 of his previous penalties in the league, with the other hitting the woodwork, while this was the first he had seen saved. \"It is good from the analysts to give him as much benefit of as many penalties as they can,\" Dyche said of Pickford's save. \"Credit to him for making the right decision in the spur of the moment.\" Former England goalkeeper Paul Robinson said on BBC Radio 5 Live: \"Pickford is good at putting pressure on strikers. Haaland must have felt the pressure there, this place went silent. \"It was a decent save but a really poor penalty. He didn't whip it around like a left-footer should. A poor penalty, but a good save.\" Ex-England defender Matt Upson added on Amazon Prime: \"It was brilliant save from Pickford. He relishes situations like this. He doesn't make life any easier for the penalty taker. \"But for Haaland to miss that penalty speaks a bit about where the confidence is in this team at the moment.\"\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. Dyche 'very pleased' with Everton defence in Man City draw\n", + "--------------------------------------------------------------------------------\n", + "Score: 0.5382, Text: Amorim knows job in 'danger' without victories\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. 'I know that every manager is in danger'\n", + "\n", + "Manchester United head coach Ruben Amorim says the vast expense of bringing him in will not shield him from the sack if he fails to produce a winning team. While United sources stress there is total support for the new boss inside Old Trafford, recent results and performances have made some fans nervous. Away supporters booed their team at the final whistle of the 2-0 defeat by Wolves at Molineux on Boxing Day, and with many exiting quickly it left the players to acknowledge hundreds of empty yellow seats before heading to the tunnel. \"The manager of Manchester United can never, no matter what, be comfortable,\" said Amorim. \"You can argue I have been here one month and I've had four training [sessions], but we are not winning. That is the reality.\" Amorim has collected seven points from seven Premier League games since taking charge last month - only one more point than fellow Portuguese Vitor Pereira, who has won both his games since becoming Wolves boss. Five defeats in Amorim's first 10 games is the worst record of any new United manager since Walter Crickmer, who stepped up from being club secretary in the 1930s. It is not what was anticipated when chief executive Omar Berrada flew to Lisbon to offer Amorim the job in the wake of Erik ten Hag's dismissal on 28 October. United were so convinced in Amorim, they paid Sporting £10.6m in compensation to get him out of his contract. But Amorim does not believe that will save him if results do not improve. \"I know that if we don't win, regardless if they pay the buyout or not, every manager is in danger,\" he said. \"I like that because that is the job.\"\n", + "\n", + "The five-day gap between the home game with Newcastle on 30 December and an immensely difficult visit to old rivals and title favourites Liverpool on 5 January is the longest spell Amorim will have had to work with his players since his appointment. He will have another spare week after that, then three more midweek games to work with a squad United sources say is not expected to change much in personnel during the January transfer window, because of the club's tight Profit and Sustainability position. Evidently, it would have been far easier for such a dramatic transition to take place during the summer. Amorim did ask if his switch could be delayed until the end of the season but that request was rejected by Berrada. \"There's no point talking or thinking about that,\" said Amorim. \"I'm here and have to focus on the job. \"It's part of football to have these difficult moments. I already knew it was going to be tough. You expect to win more games, to have players with more confidence to sell the idea and to work and improve things. \"At this moment it's really hard. We have to survive to have time and then to improve the team.\"\n", + "\n", + "Manchester United head coach Ruben Amorim has won four and lost five of his first 10 games in charge\n", + "\n", + "Former United star Cristiano Ronaldo has backed Portuguese compatriot Amorim to turn the club's fortunes around. Ronaldo's second spell at Old Trafford ended in November 2022 following an explosive TV interview with presenter and journalist Piers Morgan. \"He [Amorim] did a fantastic job in Portugal with my [club] Sporting,\" said Ronaldo. \"But the Premier League is a different beast, the most competitive league in the world. I knew that it would be tough and they will continue the storm. \"But the storm will finish and the sun will rise. Things crossed, it will be good with him and I hope the best for Manchester United because it is a club I still love.\" Al Nassr forward Ronaldo, 39, who was speaking at the Globe Soccer Awards in Dubai where he was named the Best Middle East player of 2024, added: \"I will continue to say, the problem is not the coaches. \"It's like the aquarium and you have the fish inside and it's sick, and you take him out and fix the problem. \"If you put it back in the aquarium it will be sick again. This is the problem of Manchester United. It is the same.\"\n", + "--------------------------------------------------------------------------------\n" + ] + } + ], + "source": [ + "query = \"What was manchester city manager pep guardiola's reaction to the team's current form?\"\n", + "\n", + "try:\n", + " # Perform the semantic search\n", + " start_time = time.time()\n", + " search_results = vector_store.similarity_search_with_score(query, k=10)\n", + " search_elapsed_time = time.time() - start_time\n", + "\n", + " logging.info(f\"Semantic search completed in {search_elapsed_time:.2f} seconds\")\n", + "\n", + " # Display search results\n", + " print(f\"\\nSemantic Search Results (completed in {search_elapsed_time:.2f} seconds):\")\n", + " print(\"-\" * 80) # Add separator line\n", + " for doc, score in search_results:\n", + " print(f\"Score: {score:.4f}, Text: {doc.page_content}\")\n", + " print(\"-\" * 80) # Add separator between results\n", + "\n", + "except CouchbaseException as e:\n", + " raise RuntimeError(f\"Error performing semantic search: {str(e)}\")\n", + "except Exception as e:\n", + " raise RuntimeError(f\"Unexpected error: {str(e)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "bc188844", + "metadata": {}, + "source": [ + "# PydanticAI: An Introduction\n", + "From [PydanticAI](https://ai.pydantic.dev/)'s website:\n", + "\n", + "> PydanticAI is a Python agent framework designed to make it less painful to build production grade applications with Generative AI.\n", + "\n", + "PydanticAI allows us to define agents and tools easily to create Gen-AI apps in an innovative and painless manner. Some of its features are:\n", + "- Built by the Pydantic Team: Built by the team behind Pydantic (the validation layer of the OpenAI SDK, the Anthropic SDK, LangChain, LlamaIndex, AutoGPT, Transformers, CrewAI, Instructor and many more).\n", + "\n", + "- Model-agnostic: Supports OpenAI, Anthropic, Gemini, Deepseek, Ollama, Groq, Cohere, and Mistral, and there is a simple interface to implement support for other models.\n", + "\n", + "- Type-safe: Designed to make type checking as powerful and informative as possible for you.\n", + "\n", + "- Python-centric Design: Leverages Python's familiar control flow and agent composition to build your AI-driven projects, making it easy to apply standard Python best practices you'd use in any other (non-AI) project.\n", + "\n", + "- Structured Responses: Harnesses the power of Pydantic to validate and structure model outputs, ensuring responses are consistent across runs.\n", + "\n", + "- Dependency Injection System: Offers an optional dependency injection system to provide data and services to your agent's system prompts, tools and result validators. This is useful for testing and eval-driven iterative development.\n", + "\n", + "- Streamed Responses: Provides the ability to stream LLM outputs continuously, with immediate validation, ensuring rapid and accurate results.\n", + "\n", + "- Graph Support: Pydantic Graph provides a powerful way to define graphs using typing hints, this is useful in complex applications where standard control flow can degrade to spaghetti code.\n", + "\n", + "# Building a RAG Agent using PydanticAI\n", + "\n", + "PydanticAI makes heavy use of dependency injection to provide data and services to your agent's system prompts and tools. We define dependencies using a `dataclass`, which serves as a container for our dependencies.\n", + "\n", + "In our case, the only dependency for our agent to work in the `CouchbaseQueryVectorStore` instance. However, we will still use a `dataclass` as it is good practice. In the future, in case we wish to add more dependencies, we can just add more fields to the `dataclass` `Deps`.\n", + "\n", + "We also initialize an agent as a GPT-4o model. PydanticAI supports many different LLM providers, including Anthropic, Google, Cohere, etc. which can also be used. While initializing the agent, we also pass the type of the dependencies. This is mainly used for type checking, and not actually used at runtime.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "ec663fb6", + "metadata": {}, + "outputs": [], + "source": [ + "@dataclass\n", + "class Deps:\n", + " vector_store: CouchbaseQueryVectorStore\n", + "\n", + "agent = Agent(\"openai:gpt-4o\", deps_type=Deps)\n" + ] + }, + { + "cell_type": "markdown", + "id": "00e3ceef", + "metadata": {}, + "source": [ + "# Defining the Vector Store as a Tool\n", + "PydanticAI has the concept of `function tools`, which are functions that can be called by LLMs to retrieve extra information that can help form a better response.\n", + "\n", + "We can perform RAG by creating a tool which retrieves documents that are semantically similar to the query, and allowing the agent to call the tool when required. We can add the function as a tool using the `@agent.tool` decorator.\n", + "\n", + "Notice that we also add the `context` parameter, which contains the dependencies that are passed to the tool (in this case, the only dependency is the vector store).\n" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "fab753b9", + "metadata": {}, + "outputs": [], + "source": [ + "@agent.tool\n", + "async def retrieve(context: RunContext[Deps], search_query: str) -> str:\n", + " \"\"\"Retrieve news data based on a search query.\n", + "\n", + " Args:\n", + " context: The call context\n", + " search_query: The search query\n", + " \"\"\"\n", + " search_results = context.deps.vector_store.similarity_search_with_score(search_query, k=5)\n", + " return \"\\n\\n\".join(\n", + " f\"# Documents:\\n{doc.page_content}\"\n", + " for doc, score in search_results\n", + " )\n" + ] + }, + { + "cell_type": "markdown", + "id": "157070fb", + "metadata": {}, + "source": [ + "Finally, we create a function that allows us to define our dependencies and run our agent.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "e9a034ab", + "metadata": {}, + "outputs": [], + "source": [ + "async def run_agent(question: str):\n", + " deps = Deps(\n", + " vector_store=vector_store,\n", + " )\n", + " answer = await agent.run(question, deps=deps)\n", + " return answer\n" + ] + }, + { + "cell_type": "markdown", + "id": "7b1f7319", + "metadata": {}, + "source": [ + "# Running our Agent\n", + "We have now finished setting up our vector store and agent! The system is now ready to accept queries.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "799e9fb5", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "==================== Agent Output ====================\n", + "Pep Guardiola has expressed significant concerns about Manchester City's current form, which he describes as the worst run of results in his managerial career. Despite being satisfied with his role at City, Guardiola is personally affected by the team's performance, admitting issues with sleep and diet due to stress. City have lost nine of their past 12 matches, leading to a decline in their Premier League standings. Guardiola acknowledges the difficulty of solving current challenges, particularly in defending and avoiding mistakes at crucial moments.\n", + "\n", + "The absence of key players like midfielder Rodri has been a critical issue, likened to removing a crucial piece in a game of Jenga, leading to the collapse of the team's structure. Injuries have compounded the challenge, with errors from usually dependable international players adding to the crisis. Guardiola has opted to protect and support his players rather than criticize, allowing them time to reset with more days off.\n", + "\n", + "Guardiola sees the current tumult as the end of an era and is planning changes that potentially involve a substantial squad overhaul in the upcoming transfer windows. Despite these difficulties, Guardiola remains committed and is using this period to prepare for a new phase at Manchester City under his management.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/var/folders/zd/hmksn4rj5kd_pj7w7bw48y9m0000gn/T/ipykernel_22630/1716427794.py:5: DeprecationWarning: `result.data` is deprecated, use `result.output` instead.\n", + " print(output.data)\n" + ] + } + ], + "source": [ + "query = \"What was manchester city manager pep guardiola's reaction to the team's current form?\"\n", + "output = await run_agent(query)\n", + "\n", + "print(\"=\" * 20, \"Agent Output\", \"=\" * 20)\n", + "print(output.data)\n" + ] + }, + { + "cell_type": "markdown", + "id": "56444500", + "metadata": {}, + "source": [ + "# Inspecting the Agent\n", + "We can use the `all_messages()` method in the output object to observe how the agent and tools work.\n", + "\n", + "In the cell below, we see an extremely detailed list of all the model's messages and tool calls, which happens step by step:\n", + "1. The `UserPromptPart`, which consists of the query the user sends to the agent.\n", + "2. The agent calls the `retrieve` tool in the `ToolCallPart` message. This includes the `search_query` argument. Couchbase uses this `search_query` to perform semantic search over all the ingested news articles.\n", + "3. The `retrieve` tool returns a `ToolReturnPart` object with all the context required for the model to answer the user's query. The retrieve documents were truncated, because a large amount of context was retrieved. \n", + "4. The final message is the LLM generated response with the added context, which is sent back to the user.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "3ea69905", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Step 1:\n", + "('ModelRequest(parts=[UserPromptPart(content=\"What was manchester city manager '\n", + " 'pep guardiola\\'s reaction to the team\\'s current form?\", '\n", + " 'timestamp=datetime.datetime(2025, 9, 23, 6, 17, 27, 631157, '\n", + " \"tzinfo=datetime.timezone.utc), part_kind='user-prompt')], instructions=None, \"\n", + " \"kind='request')\")\n", + "==================================================\n", + "Step 2:\n", + "(\"ModelResponse(parts=[ToolCallPart(tool_name='retrieve', \"\n", + " 'args=\\'{\"search_query\":\"Manchester City Pep Guardiola reaction current form '\n", + " 'October 2023\"}\\', tool_call_id=\\'call_kUqRR8fk7bPUzWxzpkGzt2AP\\', '\n", + " \"part_kind='tool-call')], model_name='gpt-4o-2024-08-06', \"\n", + " 'timestamp=datetime.datetime(2025, 9, 23, 6, 17, 28, '\n", + " \"tzinfo=datetime.timezone.utc), kind='response')\")\n", + "==================================================\n", + "Step 3:\n", + "(\"ModelRequest(parts=[ToolReturnPart(tool_name='retrieve', content='# \"\n", + " 'Documents:\\\\nPep Guardiola has said Manchester City will be his final '\n", + " 'managerial job in club football before he \"maybe\" coaches a national '\n", + " 'team.\\\\n\\\\nThe former Barcelona and Bayern Munich boss has won 15 major '\n", + " 'trophies since taking charge of City in 2016.\\\\n\\\\nThe 53-year-old Spaniard '\n", + " 'was approached in the summer about the possibility of becoming England '\n", + " 'manager, but last month signed a two-year contract extension with City until '\n", + " '2027.\\\\n\\\\nSpeaking to celebrity chef Dani Garcia on YouTube, Guardiola did '\n", + " 'not indicate when he intends to step down at City but said he would not '\n", + " 'return to club football - in the Premier League or overseas.\\\\n\\\\n\"I\\\\\\'m '\n", + " 'not going to manage another team,\" he said.\\\\n\\\\n\"I\\\\\\'m not talking about '\n", + " \"the long-term future, but what I\\\\'m not going to do is leave Manchester \"\n", + " 'City, go to another country, and do the same thing as now.\\\\n\\\\n\"I '\n", + " \"wouldn\\\\'t have the energy. The thought of starting somewhere else, all the \"\n", + " 'process of training and so on. No, no, no. Maybe a national team, but '\n", + " 'that\\\\\\'s different.\\\\n\\\\n\"I want to leave it and go and play golf, but I '\n", + " \"can\\\\'t [if he takes a club job]. I think stopping would do me \"\n", + " 'good.\"\\\\n\\\\nCity have won just once since Guardiola extended his contract - '\n", + " 'and once in nine games since beating Southampton on 26 October.\\\\n\\\\nThat '\n", + " 'victory came at home to Nottingham Forest last Wednesday, but was followed '\n", + " 'by a 2-2 draw at Crystal Palace at the weekend.\\\\n\\\\nThe Blues visit '\n", + " 'Juventus next in the Champions League on Wednesday (20:00 GMT), before '\n", + " 'hosting Manchester United in the Premier League on Sunday '\n", + " '(16:30).\\\\n\\\\n\"Right now we are not in the position - when we have had the '\n", + " 'results of the last seven, eight games - to talk about winning games in '\n", + " 'plural,\" said Guardiola at his pre-match news conference.\\\\n\\\\n\"We have to '\n", + " 'win the game and not look at what happens in the next one yet.\"\\\\n\\\\n# '\n", + " \"Documents:\\\\n\\\\'Self-doubt, errors & big changes\\\\' - inside the crisis at \"\n", + " 'Man City\\\\n\\\\nPep Guardiola has not been through a moment like this in his '\n", + " 'managerial career. Manchester City have lost nine matches in their past 12 - '\n", + " 'as many defeats as they had suffered in their previous 106 fixtures. At the '\n", + " 'end of October, City were still unbeaten at the top of the Premier League '\n", + " 'and favourites to win a fifth successive title. Now they are seventh, 12 '\n", + " 'points behind leaders Liverpool having played a game more. It has been an '\n", + " 'incredible fall from grace and left people trying to work out what has '\n", + " 'happened - and whether Guardiola can make it right. After discussing the '\n", + " 'situation with those who know him best, I have taken a closer look at the '\n", + " 'future - both short and long term - and how the current crisis at Man City '\n", + " \"is going to be solved.\\\\n\\\\nPep Guardiola\\\\'s Man City have lost nine of \"\n", + " 'their past 12 matches\\\\n\\\\nGuardiola has also been giving it a lot of '\n", + " 'thought. He has not been sleeping very well, as he has said, and has not '\n", + " 'been himself at times when talking to the media. He has been talking to a '\n", + " 'lot of people about what is going on as he tries to work out the reasons for '\n", + " \"City\\\\'s demise. Some reasons he knows, others he still doesn\\\\'t. What \"\n", + " 'people perhaps do not realise is Guardiola hugely doubts himself and always '\n", + " 'has. He will be thinking \"I\\\\\\'m not going to be able to get us out of this\" '\n", + " 'and needs the support of people close to him to push away those insecurities '\n", + " '- and he has that. He is protected by his people who are very aware, like he '\n", + " 'is, that there are a lot of people that want City to fail. It has been a '\n", + " 'turbulent time for Guardiola. Remember those marks he had on his head after '\n", + " 'the 3-3 draw with Feyenoord in the Champions League? He always scratches his '\n", + " 'head, it is a gesture of nervousness. Normally nothing happens but on that '\n", + " 'day one of his nails was far too sharp so, after talking to the players in '\n", + " 'the changing room where he scratched his head because of his usual agitated '\n", + " 'gesturing, he went to the news conference. His right-hand man Manel Estiarte '\n", + " 'sent him photos in a message saying \"what have you got on your head?\", but '\n", + " 'by the time Guardiola returned to the coaching room there was hardly '\n", + " 'anything there again. He started that day with a cover on his nose after the '\n", + " 'same thing happened at the training ground the day before. Guardiola was '\n", + " 'having a footballing debate with Kyle Walker about positional stuff and '\n", + " 'marked his nose with that same nail. There was also that remarkable news '\n", + " 'conference after the Manchester derby when he said \"I don\\\\\\'t know what to '\n", + " 'do\". That is partly true and partly not true. Ignore the fact Guardiola '\n", + " 'suggested he was \"not good enough\". He actually meant he was not good enough '\n", + " 'to resolve the situation with the group of players he has available and with '\n", + " 'all the other current difficulties. There are obviously logical explanations '\n", + " 'for the crisis and the first one has been talked about many times - the '\n", + " 'absence of injured midfielder Rodri. You know the game Jenga? When you take '\n", + " 'the wrong piece out, the whole tower collapses. That is what has happened '\n", + " 'here. It is normal for teams to have an over-reliance on one player if he is '\n", + " 'the best in the world in his position. And you cannot calculate the '\n", + " 'consequences of an injury that rules someone like Rodri out for the season. '\n", + " 'City are a team, like many modern ones, in which the holding midfielder is a '\n", + " 'key element to the construction. So, when you take Rodri out, it is '\n", + " 'difficult to hold it together. There were Plan Bs - John Stones, Manuel '\n", + " 'Akanji, even Nathan Ake - but injuries struck. The big injury list has been '\n", + " 'out of the ordinary and the busy calendar has also played a part in '\n", + " 'compounding the issues. However, one factor even Guardiola cannot explain is '\n", + " 'the big uncharacteristic errors in almost every game from international '\n", + " 'players. Why did Matheus Nunes make that challenge to give away the penalty '\n", + " 'against Manchester United? Jack Grealish is sent on at the end to keep the '\n", + " 'ball and cannot do that. There are errors from Walker and other defenders. '\n", + " \"These are some of the best players in the world. Of course the players\\\\' \"\n", + " 'mindset is important, and confidence is diminishing. Wrong decisions get '\n", + " 'taken so there is almost panic on the pitch instead of calm. There are also '\n", + " 'players badly out of form who are having to play because of injuries. Walker '\n", + " \"is now unable to hide behind his pace, I\\\\'m not sure Kevin de Bruyne is \"\n", + " 'ever getting back to the level he used to be at, Bernardo Silva and Ilkay '\n", + " 'Gundogan do not have time to rest, Grealish is not playing at his best. Some '\n", + " 'of these players were only meant to be playing one game a week but, because '\n", + " 'of injuries, have played 12 games in 40 days. It all has a domino effect. '\n", + " \"One consequence is that Erling Haaland isn\\\\'t getting the service to score. \"\n", + " \"But the Norwegian still remains City\\\\'s top-scorer with 13. Defender Josko \"\n", + " 'Gvardiol is next on the list with just four. The way their form has been '\n", + " 'analysed inside the City camp is there have only been three games where they '\n", + " 'deserved to lose (Liverpool, Bournemouth and Aston Villa). But of course it '\n", + " 'is time to change the dynamic.\\\\n\\\\nGuardiola has never protected his '\n", + " 'players so much. He has not criticised them and is not going to do so. They '\n", + " 'have won everything with him. Instead of doing more with them, he has tried '\n", + " 'doing less. He has sometimes given them more days off to clear their heads, '\n", + " 'so they can reset - two days this week for instance. Perhaps the time to '\n", + " 'change a team is when you are winning, but no-one was suggesting Man City '\n", + " 'were about to collapse when they were top and unbeaten after nine league '\n", + " 'games. Some people have asked how bad it has to get before City make a '\n", + " 'decision on Guardiola. The answer is that there is no decision to be made. '\n", + " 'Maybe if this was Real Madrid, Barcelona or Juventus, the pressure from '\n", + " 'outside would be massive and the argument would be made that Guardiola has '\n", + " 'to go. At City he has won the lot, so how can anyone say he is failing? Yes, '\n", + " \"this is a crisis. But given all their problems, City\\\\'s renewed target is \"\n", + " 'finishing in the top four. That is what is in all their heads now. The idea '\n", + " 'is to recover their essence by improving defensive concepts that are not '\n", + " 'there and re-establishing the intensity they are known for. Guardiola is '\n", + " 'planning to use the next two years of his contract, which is expected to be '\n", + " 'his last as a club manager, to prepare a new Manchester City. When he was at '\n", + " 'the end of his four years at Barcelona, he asked two managers what to do '\n", + " 'when you feel people are not responding to your instructions. Do you go or '\n", + " 'do the players go? Sir Alex Ferguson and Rafael Benitez both told him that '\n", + " 'the players need to go. Guardiola did not listen because of his emotional '\n", + " 'attachment to his players back then and he decided to leave the Camp Nou '\n", + " 'because he felt the cycle was over. He will still protect his players now '\n", + " 'but there is not the same emotional attachment - so it is the players who '\n", + " 'are going to leave this time. It is likely City will look to replace five or '\n", + " 'six regular starters. Guardiola knows it is the end of an era and the start '\n", + " 'of a new one. Changes will not be immediate and the majority of the work '\n", + " 'will be done in the summer. But they are open to any opportunities in '\n", + " 'January - and a holding midfielder is one thing they need. In the summer '\n", + " \"City might want to get Spain\\\\'s Martin Zubimendi from Real Sociedad and \"\n", + " 'they know 60m euros (£50m) will get him. He said no to Liverpool last summer '\n", + " 'even though everything was agreed, but he now wants to move on and the '\n", + " 'Premier League is the target. Even if they do not get Zubimendi, that is the '\n", + " 'calibre of footballer they are after. A new Manchester City is on its way - '\n", + " 'with changes driven by Guardiola, incoming sporting director Hugo Viana and '\n", + " 'the football department.\\\\n\\\\n# Documents:\\\\nManchester City boss Pep '\n", + " 'Guardiola has won 18 trophies since he arrived at the club in '\n", + " '2016\\\\n\\\\nManchester City boss Pep Guardiola says he is \"fine\" despite '\n", + " 'admitting his sleep and diet are being affected by the worst run of results '\n", + " 'in his entire managerial career. In an interview with former Italy '\n", + " \"international Luca Toni for Amazon Prime Sport before Wednesday\\\\'s \"\n", + " 'Champions League defeat by Juventus, Guardiola touched on the personal '\n", + " \"impact City\\\\'s sudden downturn in form has had. Guardiola said his state of \"\n", + " 'mind was \"ugly\", that his sleep was \"worse\" and he was eating lighter as his '\n", + " \"digestion had suffered. City go into Sunday\\\\'s derby against Manchester \"\n", + " 'United at Etihad Stadium having won just one of their past 10 games. The '\n", + " 'Juventus loss means there is a chance they may not even secure a play-off '\n", + " 'spot in the Champions League. Asked to elaborate on his comments to Toni, '\n", + " 'Guardiola said: \"I\\\\\\'m fine. \"In our jobs we always want to do our best or '\n", + " \"the best as possible. When that doesn\\\\'t happen you are more uncomfortable \"\n", + " 'than when the situation is going well, always that happened. \"In good '\n", + " 'moments I am happier but when I get to the next game I am still concerned '\n", + " 'about what I have to do. There is no human being that makes an activity and '\n", + " 'it doesn\\\\\\'t matter how they do.\" Guardiola said City have to defend better '\n", + " 'and \"avoid making mistakes at both ends\". To emphasise his point, Guardiola '\n", + " \"referred back to the third game of City\\\\'s current run, against a Sporting \"\n", + " 'side managed by Ruben Amorim, who will be in the United dugout at the '\n", + " 'weekend. City dominated the first half in Lisbon, led thanks to Phil '\n", + " \"Foden\\\\'s early effort and looked to be cruising. Instead, they conceded \"\n", + " 'three times in 11 minutes either side of half-time as Sporting eventually '\n", + " 'ran out 4-1 winners. \"I would like to play the game like we played in Lisbon '\n", + " 'on Sunday, believe me,\" said Guardiola, who is facing the prospect of only '\n", + " 'having three fit defenders for the derby as Nathan Ake and Manuel Akanji try '\n", + " 'to overcome injury concerns. If there is solace for City, it comes from the '\n", + " 'knowledge United are not exactly flying. Their comeback Europa League '\n", + " \"victory against Viktoria Plzen on Thursday was their third win of Amorim\\\\'s \"\n", + " 'short reign so far but only one of those successes has come in the Premier '\n", + " 'League, where United have lost their past two games against Arsenal and '\n", + " 'Nottingham Forest. Nevertheless, Guardiola can see improvements already on '\n", + " 'the red side of the city. \"It\\\\\\'s already there,\" he said. \"You see all the '\n", + " 'patterns, the movements, the runners and the pace. He will do a good job at '\n", + " 'United, I\\\\\\'m pretty sure of that.\"\\\\n\\\\nGuardiola says skipper Kyle Walker '\n", + " 'has been offered support by the club after the City defender highlighted the '\n", + " 'racial abuse he had received on social media in the wake of the Juventus '\n", + " 'trip. \"It\\\\\\'s unacceptable,\" he said. \"Not because it\\\\\\'s Kyle - for any '\n", + " 'human being. \"Unfortunately it happens many times in the real world. It is '\n", + " 'not necessary to say he has the support of the entire club. It is completely '\n", + " 'unacceptable and we give our support to him.\"\\\\n\\\\n# Documents:\\\\n\\\\\\'I am '\n", + " \"not good enough\\\\' - Guardiola faces daunting and major rebuild\\\\n\\\\nThis \"\n", + " 'video can not be played To play this video you need to enable JavaScript in '\n", + " \"your browser. \\\\'I am not good enough\\\\' - Guardiola says he must find a \"\n", + " \"\\\\'solution\\\\' after derby loss\\\\n\\\\nPep Guardiola says his sleep has \"\n", + " \"suffered during Manchester City\\\\'s deepening crisis, so he will not be \"\n", + " 'helped by a nightmarish conclusion to one of the most stunning defeats of '\n", + " 'his long reign. Guardiola looked agitated, animated and on edge even after '\n", + " \"City led the Manchester derby through Josko Gvardiol\\\\'s 36th-minute header, \"\n", + " 'his reaction to the goal one of almost disdain that it came via a deflected '\n", + " 'cross as opposed to in his purist style. He sat alone with his eyes closed '\n", + " 'sipping from a water bottle before the resumption of the second half, then '\n", + " 'was denied even the respite of victory when Manchester United gave this '\n", + " 'largely dismal derby a dramatic conclusion it barely deserved with a '\n", + " 'remarkable late comeback. First, with 88 minutes on the clock, Matheus Nunes '\n", + " 'presented Amad Diallo with the ball before compounding his error by '\n", + " 'flattening the forward as he made an attempt to recover his mistake. Bruno '\n", + " 'Fernandes completed the formalities from the penalty spot. Worse was to come '\n", + " \"two minutes later when Lisandro Martinez\\\\'s routine long ball caught \"\n", + " \"City\\\\'s defence inexplicably statuesque. Goalkeeper Ederson\\\\'s positioning \"\n", + " 'was awry, allowing the lively Diallo to pounce from an acute angle to leave '\n", + " 'Guardiola and his players stunned. It was the latest into any game, 88 '\n", + " 'minutes, that reigning Premier League champions had led then lost. It was '\n", + " 'also the first time City had lost a game they were leading so late on. And '\n", + " \"in a sign of City\\\\'s previous excellence that is now being challenged, they \"\n", + " 'have only lost four of 105 Premier League home games under Guardiola in '\n", + " 'which they have been ahead at half-time, winning 94 and drawing seven. '\n", + " 'Guardiola delivered a brutal self-analysis as he told Match of the Day: \"I '\n", + " 'am not good enough. I am the boss. I am the manager. I have to find '\n", + " 'solutions and so far I haven\\\\\\'t. That\\\\\\'s the reality. \"Not much else to '\n", + " 'say. No defence. Manchester United were incredibly persistent. We have not '\n", + " 'lost eight games in two seasons. We can\\\\\\'t defend that.\"\\\\n\\\\nManchester '\n", + " 'City manager Pep Guardiola in despair during the derby defeat to Manchester '\n", + " 'United\\\\n\\\\nGuardiola suggested the serious renewal will wait until the '\n", + " 'summer but the red flags have been appearing for weeks in the sudden and '\n", + " 'shocking decline of a team that has lost the aura of invincibility that left '\n", + " 'many opponents beaten before kick-off in previous years. He has had stated '\n", + " 'City must \"survive\" this season - whatever qualifies as survival for a club '\n", + " 'of such rich ambition - but the quest for a record fifth successive Premier '\n", + " 'League title is surely over as they lie nine points behind leaders Liverpool '\n", + " 'having played a game more. Their Champions League aspirations are also in '\n", + " \"jeopardy after another loss, this time against Juventus in Turin. City\\\\'s \"\n", + " 'squad has been allowed to grow too old together. The insatiable thirst for '\n", + " 'success seems to have gone, the scales of superiority have fallen away and '\n", + " 'opponents now sense vulnerability right until the final whistle, as United '\n", + " 'did here. The manner in which United were able, and felt able, to snatch '\n", + " 'this victory drove right to the heart of how City, and Guardiola, are '\n", + " 'allowing opponents to prey on their downfall. Guardiola has every reason to '\n", + " 'cite injuries, most significantly to Rodri and also John Stones as well as '\n", + " 'others, but this cannot be used an excuse for such a dramatic decline in '\n", + " 'standards, allied to the appearance of a soft underbelly that is so easily '\n", + " \"exploited. And City\\\\'s rebuild will not be a quick fix. With every \"\n", + " 'performance, every defeat, the scale of what lies in front of Guardiola '\n", + " \"becomes more obvious - and daunting. Manchester City\\\\'s fans did their best \"\n", + " 'to reassure Guardiola of their faith in him with a giant Barcelona-inspired '\n", + " 'banner draped from the stands before kick-off emblazoned with his image '\n", + " 'reading \"Més que un entrenador\" - \"More Than A Coach\". And Guardiola will '\n", + " 'now need to be more than a coach than at any time in his career. He will '\n", + " \"have the finances but it will be done with City\\\\'s challengers also \"\n", + " 'strengthening. Kevin de Bruyne, 34 in June, lasted 68 minutes here before he '\n", + " 'was substituted. Age and injuries are catching up with one of the greatest '\n", + " 'players of the Premier League era and he is unlikely to be at City next '\n", + " 'season. Mateo Kovacic, who replaced De Bruyne, is also 31 in May. Kyle '\n", + " 'Walker, 34, is being increasingly exposed. His most notable contribution '\n", + " 'here was an embarrassing collapse to the ground after the mildest '\n", + " 'head-to-head collision with Rasmus Hojlund. Ilkay Gundogan, another '\n", + " \"34-year-old and a previous pillar of Guardiola\\\\'s great successes, no \"\n", + " 'longer has the legs or energy to exert influence. This looks increasingly '\n", + " 'like a season too far following his return from Barcelona. Flaws are also '\n", + " 'being exposed elsewhere, with previously reliable performers failing to hit '\n", + " 'previous standards. Phil Foden scored 27 goals and had 12 assists when he '\n", + " 'was Premier League Player of the Season last term. This year he has just '\n", + " 'three goals and two assists in 18 appearances in all competitions. He has no '\n", + " 'goals and just one assist in 11 Premier League games. Jack Grealish, who '\n", + " 'came on after 77 minutes against United, has not scored in a year for '\n", + " 'Manchester City, his last goal coming in a 2-2 draw against Crystal Palace '\n", + " 'on 16 December last year. He has, in the meantime, scored twice for England. '\n", + " 'Erling Haaland is also struggling as City lack creativity and cutting edge. '\n", + " 'He has three goals in his past 11 Premier League games after scoring 10 in '\n", + " \"his first five. And in another indication of City\\\\'s impotence, and their \"\n", + " \"reliance on Haaland, defender Gvardiol\\\\'s goal against United was his \"\n", + " 'fourth this season, making him their second highest scorer in all '\n", + " 'competitions behind the Norwegian striker, who has 18. Goalkeeper Ederson, '\n", + " 'so reliable for so long, has already been dropped once this season and did '\n", + " \"not cover himself in glory for United\\\\'s winner. Guardiola, with that \"\n", + " 'freshly signed two-year contract, insists he \"wants it\" as he treads on this '\n", + " 'alien territory of failure. He will be under no illusions about the size of '\n", + " 'the job in front of him as he placed his head in his hands in anguish after '\n", + " 'yet another damaging and deeply revealing defeat. City and Guardiola are in '\n", + " 'new, unforgiving territory.\\\\n\\\\n# Documents:\\\\nMan City might miss out on '\n", + " 'Champions League - Guardiola\\\\n\\\\nErling Haaland was part of the Manchester '\n", + " 'City side that won the Champions League for the first time in '\n", + " '2023\\\\n\\\\nManchester City boss Pep Guardiola says the club are in danger of '\n", + " \"missing out on a place in next season\\\\'s Champions League. City are \"\n", + " \"currently in their 14th consecutive season in European football\\\\'s most \"\n", + " 'prestigious club competition. Only Arsenal between 1998 and 2017, and '\n", + " 'Manchester United between 1996 and 2014, have a longer record of qualifying '\n", + " 'among English clubs. City are seventh in the Premier League after 17 '\n", + " 'matches, four points behind Nottingham Forest in fourth and a point behind '\n", + " \"fifth-placed Bournemouth. England are currently top of Uefa\\\\'s European \"\n", + " 'Performance Spot table and well placed to secure a fifth place in next '\n", + " \"season\\\\'s Champions League, although City would still not qualify on \"\n", + " 'current standings. \"When I said before, people laughed,\" said Guardiola. '\n", + " '\"They said, \\\\\\'qualifying for the Champions League is not a big '\n", + " 'success\\\\\\'. \"But I know it because it happens with clubs in this country. '\n", + " 'They were dominant for many years and after they were many years not '\n", + " 'qualifying for the Champions League.\"\\\\n\\\\nGuardiola\\\\\\'s side host Everton '\n", + " 'on Boxing Day, before a trip to Leicester on 29 December and a home match '\n", + " 'against West Ham on 4 January. Given all three opponents are in the bottom '\n", + " 'seven, it offers City a chance to improve on an appalling recent record of '\n", + " 'four points from eight games, which Guardiola acknowledges has left their '\n", + " 'lofty European ambitions in doubt. \"The one team that has been in the '\n", + " 'Champions League for the past years has been Manchester City,\" he added. '\n", + " '\"Now we are at risk, of course we are. Definitely.\" Arsenal, Chelsea, '\n", + " \"Liverpool and Manchester United finished in the Premier League\\\\'s top four \"\n", + " 'from the 2005-06 season to 2008-09. At least three of them also occupied the '\n", + " 'top four spots for 15 successive campaigns until 2012. But United have spent '\n", + " 'five out of the past 11 seasons outside the Champions League. Arsenal spent '\n", + " 'six seasons out of the competition before returning last term. Liverpool '\n", + " 'missed out all but one year in seven from 2010, while Chelsea are in their '\n", + " \"second successive campaign outside Europe\\\\'s elite. This term the threat to \"\n", + " 'City comes from unexpected sources. As well as Forest and Bournemouth, Aston '\n", + " 'Villa are ahead of City, while Newcastle, Fulham and Brighton are also '\n", + " 'within a couple of points. \"There are a lot of contenders,\" said Guardiola, '\n", + " 'whose side have lost nine of their last 12 games in all competitions. \"For '\n", + " 'every club it is so important and if we are not winning games, we will be '\n", + " 'out. \"If we don\\\\\\'t qualify it is because we don\\\\\\'t deserve it, because '\n", + " \"we were not prepared and because we had a lot of problems and didn\\\\'t solve \"\n", + " 'them.\"\\', tool_call_id=\\'call_kUqRR8fk7bPUzWxzpkGzt2AP\\', '\n", + " 'timestamp=datetime.datetime(2025, 9, 23, 6, 17, 29, 776114, '\n", + " \"tzinfo=datetime.timezone.utc), part_kind='tool-return')], instructions=None, \"\n", + " \"kind='request')\")\n", + "==================================================\n", + "Step 4:\n", + "('ModelResponse(parts=[TextPart(content=\"Pep Guardiola has expressed '\n", + " \"significant concerns about Manchester City's current form, which he \"\n", + " 'describes as the worst run of results in his managerial career. Despite '\n", + " 'being satisfied with his role at City, Guardiola is personally affected by '\n", + " \"the team's performance, admitting issues with sleep and diet due to stress. \"\n", + " 'City have lost nine of their past 12 matches, leading to a decline in their '\n", + " 'Premier League standings. Guardiola acknowledges the difficulty of solving '\n", + " 'current challenges, particularly in defending and avoiding mistakes at '\n", + " 'crucial moments.\\\\n\\\\nThe absence of key players like midfielder Rodri has '\n", + " 'been a critical issue, likened to removing a crucial piece in a game of '\n", + " \"Jenga, leading to the collapse of the team's structure. Injuries have \"\n", + " 'compounded the challenge, with errors from usually dependable international '\n", + " 'players adding to the crisis. Guardiola has opted to protect and support his '\n", + " 'players rather than criticize, allowing them time to reset with more days '\n", + " 'off.\\\\n\\\\nGuardiola sees the current tumult as the end of an era and is '\n", + " 'planning changes that potentially involve a substantial squad overhaul in '\n", + " 'the upcoming transfer windows. Despite these difficulties, Guardiola remains '\n", + " 'committed and is using this period to prepare for a new phase at Manchester '\n", + " 'City under his management.\", part_kind=\\'text\\')], '\n", + " \"model_name='gpt-4o-2024-08-06', timestamp=datetime.datetime(2025, 9, 23, 6, \"\n", + " \"17, 30, tzinfo=datetime.timezone.utc), kind='response')\")\n", + "==================================================\n" + ] + } + ], + "source": [ + "from pprint import pprint\n", + "\n", + "for idx, message in enumerate(output.all_messages(), start=1):\n", + " print(f\"Step {idx}:\")\n", + " pprint(message.__repr__())\n", + " print(\"=\" * 50)\n" + ] + }, + { + "cell_type": "markdown", + "id": "a2f4387f", + "metadata": {}, + "source": [ + "## Conclusion\n", + "By following these steps, you'll have a fully functional semantic search engine that leverages the strengths of Couchbase and PydanticAI. This guide is designed not just to show you how to build the system, but also to explain why each step is necessary, giving you a deeper understanding of the principles behind semantic search and how it improves querying data more efficiently using GSI which can significantly improve your RAG performance. Whether you're a newcomer to software development or an experienced developer looking to expand your skills, this guide will provide you with the knowledge and tools you need to create a powerful, AI-driven search engine using PydanticAI's agent-based approach." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/pydantic_ai/gsi/frontmatter.md b/pydantic_ai/gsi/frontmatter.md index 307f717e..7609ee47 100644 --- a/pydantic_ai/gsi/frontmatter.md +++ b/pydantic_ai/gsi/frontmatter.md @@ -1,12 +1,12 @@ --- # frontmatter -path: "/tutorial-pydantic-ai-couchbase-rag" +path: "/tutorial-pydantic-ai-couchbase-rag-with-global-secondary-index" title: Retrieval-Augmented Generation (RAG) with Couchbase and PydanticAI using GSI -short_title: RAG with Couchbase and PydanticAI +short_title: RAG with Couchbase and PydanticAI using GSI description: - Learn how to build a semantic search engine using Couchbase and PydanticAI using GSI. - - This tutorial demonstrates how to integrate Couchbase's vector search capabilities with PydanticAI using tool calling. - - You'll understand how to perform Retrieval-Augmented Generation (RAG) using PydanticAI and Couchbase. + - This tutorial demonstrates how to integrate Couchbase's vector search capabilities with PydanticAI using GSI indexes. + - You'll understand how to perform Retrieval-Augmented Generation (RAG) using PydanticAI and Couchbase with GSI optimization. content_type: tutorial filter: sdk technology: From b6f27f023a22ea782ab6ea9c3ba1068bc4e4acdc Mon Sep 17 00:00:00 2001 From: AayushTyagi1 Date: Thu, 23 Oct 2025 12:30:31 +0530 Subject: [PATCH 4/9] Frontmatter Modified and altenative tutorial lin put --- pydantic_ai/fts/RAG_with_Couchbase_and_PydanticAI.ipynb | 2 +- pydantic_ai/fts/frontmatter.md | 2 +- pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/pydantic_ai/fts/RAG_with_Couchbase_and_PydanticAI.ipynb b/pydantic_ai/fts/RAG_with_Couchbase_and_PydanticAI.ipynb index 17d7f137..ae779c70 100644 --- a/pydantic_ai/fts/RAG_with_Couchbase_and_PydanticAI.ipynb +++ b/pydantic_ai/fts/RAG_with_Couchbase_and_PydanticAI.ipynb @@ -7,7 +7,7 @@ }, "source": [ "# Introduction\n", - "In this guide, we will walk you through building a powerful semantic search engine using Couchbase as the backend database, [OpenAI](https://openai.com) as the embedding and LLM provider, and [PydanticAI](https://ai.pydantic.dev) as an agent orchestrator. Semantic search goes beyond simple keyword matching by understanding the context and meaning behind the words in a query, making it an essential tool for applications that require intelligent information retrieval. This tutorial is designed to be beginner-friendly, with clear, step-by-step instructions that will equip you with the knowledge to create a fully functional semantic search system from scratch." + "In this guide, we will walk you through building a powerful semantic search engine using Couchbase as the backend database, [OpenAI](https://openai.com) as the embedding and LLM provider, and [PydanticAI](https://ai.pydantic.dev) as an agent orchestrator. Semantic search goes beyond simple keyword matching by understanding the context and meaning behind the words in a query, making it an essential tool for applications that require intelligent information retrieval. This tutorial is designed to be beginner-friendly, with clear, step-by-step instructions that will equip you with the knowledge to create a fully functional semantic search system from scratch. Alternatively if you want to perform semantic search using the GSI index, please take a look at [this.](https://developer.couchbase.com/tutorial-pydantic-ai-couchbase-rag-with-global-secondary-index/)" ] }, { diff --git a/pydantic_ai/fts/frontmatter.md b/pydantic_ai/fts/frontmatter.md index 74ad650e..7cfad333 100644 --- a/pydantic_ai/fts/frontmatter.md +++ b/pydantic_ai/fts/frontmatter.md @@ -1,6 +1,6 @@ --- # frontmatter -path: "/tutorial-pydantic-ai-couchbase-rag" +path: "/tutorial-pydantic-ai-couchbase-rag-with-fts" title: Retrieval-Augmented Generation (RAG) with Couchbase and PydanticAI using FTS short_title: RAG with Couchbase and PydanticAI description: diff --git a/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb b/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb index fe680020..5a389585 100644 --- a/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb +++ b/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb @@ -6,7 +6,7 @@ "metadata": {}, "source": [ "# Introduction\n", - "In this guide, we will walk you through building a powerful semantic search engine using Couchbase as the backend database, [OpenAI](https://openai.com) as the embedding and LLM provider, and [PydanticAI](https://ai.pydantic.dev) as an agent orchestrator. Semantic search goes beyond simple keyword matching by understanding the context and meaning behind the words in a query, making it an essential tool for applications that require intelligent information retrieval. This tutorial is designed to be beginner-friendly, with clear, step-by-step instructions that will equip you with the knowledge to create a fully functional semantic search system using GSI (Global Secondary Index) from scratch. Alternatively if you want to perform semantic search using the FTS index, please take a look at [this.](https://developer.couchbase.com/tutorial-pydantic-ai-couchbase-rag/)\n" + "In this guide, we will walk you through building a powerful semantic search engine using Couchbase as the backend database, [OpenAI](https://openai.com) as the embedding and LLM provider, and [PydanticAI](https://ai.pydantic.dev) as an agent orchestrator. Semantic search goes beyond simple keyword matching by understanding the context and meaning behind the words in a query, making it an essential tool for applications that require intelligent information retrieval. This tutorial is designed to be beginner-friendly, with clear, step-by-step instructions that will equip you with the knowledge to create a fully functional semantic search system using GSI (Global Secondary Index) from scratch. Alternatively if you want to perform semantic search using the FTS index, please take a look at [this.](https://developer.couchbase.com/tutorial-pydantic-ai-couchbase-rag-with-fts/)\n" ] }, { From f92f46b9a1fe7403ae85a6e04a6d05a8033a86cc Mon Sep 17 00:00:00 2001 From: AayushTyagi1 Date: Thu, 23 Oct 2025 12:38:11 +0530 Subject: [PATCH 5/9] Alt index in markdown --- .../gsi/RAG_with_Couchbase_and_PydanticAI.ipynb | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb b/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb index 5a389585..34d9e389 100644 --- a/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb +++ b/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb @@ -782,11 +782,18 @@ "metadata": {}, "outputs": [], "source": [ - "vector_store.create_index(index_type=IndexType.BHIVE, index_name=\"pydantic_ai_bhive_index\", index_description=\"IVF,SQ8\")\n", + "vector_store.create_index(index_type=IndexType.BHIVE, index_name=\"pydantic_ai_bhive_index\", index_description=\"IVF,SQ8\")" + ] + }, + { + "cell_type": "markdown", + "id": "e864010e", + "metadata": {}, + "source": [ + "Note: To create a COMPOSITE index, the below code can be used.\n", + "Choose based on your specific use case and query patterns. For this tutorial's news search scenario, either index type would work, but BHIVE might be more efficient for pure semantic search across news articles.\n", "\n", - "# Note: To create a COMPOSITE index, the below code can be used.\n", - "# Choose based on your specific use case and query patterns. For this tutorial's news search scenario, either index type would work, but BHIVE might be more efficient for pure semantic search across news articles.\n", - "# vector_store.create_index(index_type=IndexType.COMPOSITE, index_name=\"pydantic_ai_composite_index\", index_description=\"IVF,SQ8\")" + "vector_store.create_index(index_type=IndexType.COMPOSITE, index_name=\"pydantic_ai_composite_index\", index_description=\"IVF,SQ8\")" ] }, { From 2ed206dd9790136b8ba29954feed0ef6b6d123e3 Mon Sep 17 00:00:00 2001 From: AayushTyagi1 Date: Thu, 6 Nov 2025 12:53:26 +0530 Subject: [PATCH 6/9] Add Description and Change Score to Distance --- .../RAG_with_Couchbase_and_PydanticAI.ipynb | 132 +++++++++++++++++- 1 file changed, 126 insertions(+), 6 deletions(-) diff --git a/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb b/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb index 34d9e389..ea6cc07f 100644 --- a/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb +++ b/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb @@ -362,6 +362,126 @@ " raise ValueError(f\"Error creating OpenAIEmbeddings: {str(e)}\")\n" ] }, + { + "cell_type": "markdown", + "id": "b4402378", + "metadata": {}, + "source": [ + "# Understanding GSI Vector Search\n", + "\n", + "### Optimizing Vector Search with Global Secondary Index (GSI)\n", + "\n", + "With Couchbase 8.0+, you can leverage the power of GSI-based vector search, which offers significant performance improvements over traditional Full-Text Search (FTS) approaches for vector-first workloads. GSI vector search provides high-performance vector similarity search with advanced filtering capabilities and is designed to scale to billions of vectors.\n", + "\n", + "#### GSI vs FTS: Choosing the Right Approach\n", + "\n", + "| Feature | GSI Vector Search | FTS Vector Search |\n", + "| --------------------- | --------------------------------------------------------------- | ----------------------------------------- |\n", + "| **Best For** | Vector-first workloads, complex filtering, high QPS performance| Hybrid search and high recall rates |\n", + "| **Couchbase Version** | 8.0.0+ | 7.6+ |\n", + "| **Filtering** | Pre-filtering with `WHERE` clauses (Composite) or post-filtering (BHIVE) | Pre-filtering with flexible ordering |\n", + "| **Scalability** | Up to billions of vectors (BHIVE) | Up to 10 million vectors |\n", + "| **Performance** | Optimized for concurrent operations with low memory footprint | Good for mixed text and vector queries |\n", + "\n", + "\n", + "#### GSI Vector Index Types\n", + "\n", + "Couchbase offers two distinct GSI vector index types, each optimized for different use cases:\n", + "\n", + "##### Hyperscale Vector Indexes (BHIVE)\n", + "\n", + "- **Best for**: Pure vector searches like content discovery, recommendations, and semantic search\n", + "- **Use when**: You primarily perform vector-only queries without complex scalar filtering\n", + "- **Features**: \n", + " - High performance with low memory footprint\n", + " - Optimized for concurrent operations\n", + " - Designed to scale to billions of vectors\n", + " - Supports post-scan filtering for basic metadata filtering\n", + "\n", + "##### Composite Vector Indexes\n", + "\n", + " - **Best for**: Filtered vector searches that combine vector similarity with scalar value filtering\n", + "- **Use when**: Your queries combine vector similarity with scalar filters that eliminate large portions of data\n", + "- **Features**: \n", + " - Efficient pre-filtering where scalar attributes reduce the vector comparison scope\n", + " - Best for well-defined workloads requiring complex filtering using GSI features\n", + " - Supports range lookups combined with vector search\n", + "\n", + "#### Index Type Selection for This Tutorial\n", + "\n", + "In this tutorial, we'll demonstrate creating a **BHIVE index** and running vector similarity queries using GSI. BHIVE is ideal for semantic search scenarios where you want:\n", + "\n", + "1. **High-performance vector search** across large datasets\n", + "2. **Low latency** for real-time applications\n", + "3. **Scalability** to handle growing vector collections\n", + "4. **Concurrent operations** for multi-user environments\n", + "\n", + "The BHIVE index will provide optimal performance for our Hugging Face embedding-based semantic search implementation.\n", + "\n", + "#### Alternative: Composite Vector Index\n", + "\n", + "If your use case requires complex filtering with scalar attributes, you may want to consider using a **Composite Vector Index** instead:\n", + "\n", + "```python\n", + "# Alternative: Create a Composite index for filtered searches\n", + "vector_store.create_index(\n", + " index_type=IndexType.COMPOSITE,\n", + " index_description=\"IVF,SQ8\",\n", + " distance_metric=DistanceStrategy.COSINE,\n", + " index_name=\"pydantic_composite_index\",\n", + ")\n", + "```\n", + "\n", + "**Use Composite indexes when:**\n", + "- You need to filter by document metadata or attributes before vector similarity\n", + "- Your queries combine vector search with WHERE clauses\n", + "- You have well-defined filtering requirements that can reduce the search space\n", + "\n", + "**Note**: Composite indexes enable pre-filtering with scalar attributes, making them ideal for applications where you need to search within specific categories, date ranges, or user-specific data segments.\n", + "\n", + "#### Understanding GSI Index Configuration (Couchbase 8.0 Feature)\n", + "\n", + "Before creating our BHIVE index, it's important to understand the configuration parameters that optimize vector storage and search performance. The `index_description` parameter controls how Couchbase optimizes vector storage through centroids and quantization.\n", + "\n", + "##### Index Description Format: `'IVF[],{PQ|SQ}'`\n", + "\n", + "##### Centroids (IVF - Inverted File)\n", + "\n", + "- Controls how the dataset is subdivided for faster searches\n", + "- **More centroids** = faster search, slower training time\n", + "- **Fewer centroids** = slower search, faster training time\n", + "- If omitted (like `IVF,SQ8`), Couchbase auto-selects based on dataset size\n", + "\n", + "###### Quantization Options\n", + "\n", + "**Scalar Quantization (SQ):**\n", + "- `SQ4`, `SQ6`, `SQ8` (4, 6, or 8 bits per dimension)\n", + "- Lower memory usage, faster search, slightly reduced accuracy\n", + "\n", + "**Product Quantization (PQ):**\n", + "- Format: `PQx` (e.g., `PQ32x8`)\n", + "- Better compression for very large datasets\n", + "- More complex but can maintain accuracy with smaller index size\n", + "\n", + "##### Common Configuration Examples\n", + "\n", + "- **`IVF,SQ8`** - Auto centroids, 8-bit scalar quantization (good default)\n", + "- **`IVF1000,SQ6`** - 1000 centroids, 6-bit scalar quantization\n", + "- **`IVF,PQ32x8`** - Auto centroids, 32 subquantizers with 8 bits\n", + "\n", + "For detailed configuration options, see the [Quantization & Centroid Settings](https://docs.couchbase.com/cloud/vector-index/hyperscale-vector-index.html#algo_settings).\n", + "\n", + "For more information on GSI vector indexes, see [Couchbase GSI Vector Documentation](https://docs.couchbase.com/cloud/vector-index/use-vector-indexes.html).\n", + "\n", + "##### Our Configuration Choice\n", + "\n", + "In this tutorial, we use `IVF,SQ8` which provides:\n", + "- **Auto-selected centroids** optimized for our dataset size\n", + "- **8-bit scalar quantization** for good balance of speed, memory usage, and accuracy\n", + "- **COSINE distance metric** ideal for semantic similarity search\n", + "- **Optimal performance** for most semantic search use cases" + ] + }, { "cell_type": "markdown", "id": "2f67cece", @@ -548,7 +668,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": null, "id": "be49ce6f", "metadata": {}, "outputs": [ @@ -705,8 +825,8 @@ " # Display search results\n", " print(f\"\\nSemantic Search Results (completed in {search_elapsed_time:.2f} seconds):\")\n", " print(\"-\" * 80) # Add separator line\n", - " for doc, score in search_results:\n", - " print(f\"Score: {score:.4f}, Text: {doc.page_content}\")\n", + " for doc, distance in search_results:\n", + " print(f\"Vector Distance: {distance:.4f}, Text: {doc.page_content}\")\n", " print(\"-\" * 80) # Add separator between results\n", "\n", "except CouchbaseException as e:\n", @@ -798,7 +918,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": null, "id": "d1f4812c", "metadata": {}, "outputs": [ @@ -955,8 +1075,8 @@ " # Display search results\n", " print(f\"\\nSemantic Search Results (completed in {search_elapsed_time:.2f} seconds):\")\n", " print(\"-\" * 80) # Add separator line\n", - " for doc, score in search_results:\n", - " print(f\"Score: {score:.4f}, Text: {doc.page_content}\")\n", + " for doc, distance in search_results:\n", + " print(f\"Vector Distance: {distance:.4f}, Text: {doc.page_content}\")\n", " print(\"-\" * 80) # Add separator between results\n", "\n", "except CouchbaseException as e:\n", From f6042d81b91cbbc24932864de68c457d22d6643e Mon Sep 17 00:00:00 2001 From: AayushTyagi1 Date: Thu, 6 Nov 2025 13:12:16 +0530 Subject: [PATCH 7/9] DA-1075 Pydantic AI Tutorial --- .../RAG_with_Couchbase_and_PydanticAI.ipynb | 1080 ++++++++++++----- 1 file changed, 758 insertions(+), 322 deletions(-) diff --git a/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb b/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb index ea6cc07f..2d81056a 100644 --- a/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb +++ b/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb @@ -82,16 +82,7 @@ "execution_count": 2, "id": "bafc8b3c", "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/Users/aayush.tyagi/Documents/couchbaselabs-vector-search-cookbook/vector-search-cookbook/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", - " from .autonotebook import tqdm as notebook_tqdm\n" - ] - } - ], + "outputs": [], "source": [ "import getpass\n", "import json\n", @@ -197,7 +188,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-09-23 11:44:21,729 - INFO - Successfully connected to Couchbase\n" + "2025-11-06 13:07:15,968 - INFO - Successfully connected to Couchbase\n" ] } ], @@ -250,15 +241,18 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-09-23 11:44:23,574 - INFO - Bucket 'query-vector-search-testing' exists.\n", - "2025-09-23 11:44:23,581 - INFO - Collection 'pydantic_ai' already exists. Skipping creation.\n", - "2025-09-23 11:44:25,657 - INFO - All documents cleared from the collection.\n" + "2025-11-06 13:07:20,020 - INFO - Bucket 'travel-sample' exists.\n", + "2025-11-06 13:07:20,024 - INFO - Scope 'shared' does not exist. Creating it...\n", + "2025-11-06 13:07:20,069 - INFO - Scope 'shared' created successfully.\n", + "2025-11-06 13:07:20,078 - INFO - Collection 'demo' does not exist. Creating it...\n", + "2025-11-06 13:07:20,124 - INFO - Collection 'demo' created successfully.\n", + "2025-11-06 13:07:22,198 - INFO - All documents cleared from the collection.\n" ] }, { "data": { "text/plain": [ - "" + "" ] }, "execution_count": 6, @@ -350,7 +344,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-09-23 11:44:28,895 - INFO - Successfully created OpenAIEmbeddings\n" + "2025-11-06 13:07:27,643 - INFO - Successfully created OpenAIEmbeddings\n" ] } ], @@ -503,7 +497,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-09-23 11:44:32,820 - INFO - Successfully created vector store\n" + "2025-11-06 13:07:38,469 - INFO - Successfully created vector store\n" ] } ], @@ -543,7 +537,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-09-23 11:44:53,772 - INFO - Successfully loaded the BBC News dataset with 2687 rows.\n" + "2025-11-06 13:07:47,468 - INFO - Successfully loaded the BBC News dataset with 2687 rows.\n" ] }, { @@ -635,7 +629,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-09-23 11:46:08,521 - INFO - Document ingestion completed successfully.\n" + "2025-11-06 13:09:09,453 - INFO - Document ingestion completed successfully.\n" ] } ], @@ -668,7 +662,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 12, "id": "be49ce6f", "metadata": {}, "outputs": [ @@ -676,7 +670,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-09-23 11:46:56,960 - INFO - Semantic search completed in 1.69 seconds\n" + "2025-11-06 13:09:15,449 - INFO - Semantic search completed in 1.47 seconds\n" ] }, { @@ -684,15 +678,15 @@ "output_type": "stream", "text": [ "\n", - "Semantic Search Results (completed in 1.69 seconds):\n", + "Semantic Search Results (completed in 1.47 seconds):\n", "--------------------------------------------------------------------------------\n", - "Score: 0.2955, Text: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", + "Vector Distance: 0.2956, Text: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", "\n", "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affected by the worst run of results in his entire managerial career. In an interview with former Italy international Luca Toni for Amazon Prime Sport before Wednesday's Champions League defeat by Juventus, Guardiola touched on the personal impact City's sudden downturn in form has had. Guardiola said his state of mind was \"ugly\", that his sleep was \"worse\" and he was eating lighter as his digestion had suffered. City go into Sunday's derby against Manchester United at Etihad Stadium having won just one of their past 10 games. The Juventus loss means there is a chance they may not even secure a play-off spot in the Champions League. Asked to elaborate on his comments to Toni, Guardiola said: \"I'm fine. \"In our jobs we always want to do our best or the best as possible. When that doesn't happen you are more uncomfortable than when the situation is going well, always that happened. \"In good moments I am happier but when I get to the next game I am still concerned about what I have to do. There is no human being that makes an activity and it doesn't matter how they do.\" Guardiola said City have to defend better and \"avoid making mistakes at both ends\". To emphasise his point, Guardiola referred back to the third game of City's current run, against a Sporting side managed by Ruben Amorim, who will be in the United dugout at the weekend. City dominated the first half in Lisbon, led thanks to Phil Foden's early effort and looked to be cruising. Instead, they conceded three times in 11 minutes either side of half-time as Sporting eventually ran out 4-1 winners. \"I would like to play the game like we played in Lisbon on Sunday, believe me,\" said Guardiola, who is facing the prospect of only having three fit defenders for the derby as Nathan Ake and Manuel Akanji try to overcome injury concerns. If there is solace for City, it comes from the knowledge United are not exactly flying. Their comeback Europa League victory against Viktoria Plzen on Thursday was their third win of Amorim's short reign so far but only one of those successes has come in the Premier League, where United have lost their past two games against Arsenal and Nottingham Forest. Nevertheless, Guardiola can see improvements already on the red side of the city. \"It's already there,\" he said. \"You see all the patterns, the movements, the runners and the pace. He will do a good job at United, I'm pretty sure of that.\"\n", "\n", "Guardiola says skipper Kyle Walker has been offered support by the club after the City defender highlighted the racial abuse he had received on social media in the wake of the Juventus trip. \"It's unacceptable,\" he said. \"Not because it's Kyle - for any human being. \"Unfortunately it happens many times in the real world. It is not necessary to say he has the support of the entire club. It is completely unacceptable and we give our support to him.\"\n", "--------------------------------------------------------------------------------\n", - "Score: 0.3100, Text: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", + "Vector Distance: 0.3100, Text: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", "\n", "The former Barcelona and Bayern Munich boss has won 15 major trophies since taking charge of City in 2016.\n", "\n", @@ -718,7 +712,7 @@ "\n", "\"We have to win the game and not look at what happens in the next one yet.\"\n", "--------------------------------------------------------------------------------\n", - "Score: 0.3311, Text: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", + "Vector Distance: 0.3311, Text: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", "\n", "This video can not be played To play this video you need to enable JavaScript in your browser. 'I am not good enough' - Guardiola says he must find a 'solution' after derby loss\n", "\n", @@ -728,7 +722,7 @@ "\n", "Guardiola suggested the serious renewal will wait until the summer but the red flags have been appearing for weeks in the sudden and shocking decline of a team that has lost the aura of invincibility that left many opponents beaten before kick-off in previous years. He has had stated City must \"survive\" this season - whatever qualifies as survival for a club of such rich ambition - but the quest for a record fifth successive Premier League title is surely over as they lie nine points behind leaders Liverpool having played a game more. Their Champions League aspirations are also in jeopardy after another loss, this time against Juventus in Turin. City's squad has been allowed to grow too old together. The insatiable thirst for success seems to have gone, the scales of superiority have fallen away and opponents now sense vulnerability right until the final whistle, as United did here. The manner in which United were able, and felt able, to snatch this victory drove right to the heart of how City, and Guardiola, are allowing opponents to prey on their downfall. Guardiola has every reason to cite injuries, most significantly to Rodri and also John Stones as well as others, but this cannot be used an excuse for such a dramatic decline in standards, allied to the appearance of a soft underbelly that is so easily exploited. And City's rebuild will not be a quick fix. With every performance, every defeat, the scale of what lies in front of Guardiola becomes more obvious - and daunting. Manchester City's fans did their best to reassure Guardiola of their faith in him with a giant Barcelona-inspired banner draped from the stands before kick-off emblazoned with his image reading \"Més que un entrenador\" - \"More Than A Coach\". And Guardiola will now need to be more than a coach than at any time in his career. He will have the finances but it will be done with City's challengers also strengthening. Kevin de Bruyne, 34 in June, lasted 68 minutes here before he was substituted. Age and injuries are catching up with one of the greatest players of the Premier League era and he is unlikely to be at City next season. Mateo Kovacic, who replaced De Bruyne, is also 31 in May. Kyle Walker, 34, is being increasingly exposed. His most notable contribution here was an embarrassing collapse to the ground after the mildest head-to-head collision with Rasmus Hojlund. Ilkay Gundogan, another 34-year-old and a previous pillar of Guardiola's great successes, no longer has the legs or energy to exert influence. This looks increasingly like a season too far following his return from Barcelona. Flaws are also being exposed elsewhere, with previously reliable performers failing to hit previous standards. Phil Foden scored 27 goals and had 12 assists when he was Premier League Player of the Season last term. This year he has just three goals and two assists in 18 appearances in all competitions. He has no goals and just one assist in 11 Premier League games. Jack Grealish, who came on after 77 minutes against United, has not scored in a year for Manchester City, his last goal coming in a 2-2 draw against Crystal Palace on 16 December last year. He has, in the meantime, scored twice for England. Erling Haaland is also struggling as City lack creativity and cutting edge. He has three goals in his past 11 Premier League games after scoring 10 in his first five. And in another indication of City's impotence, and their reliance on Haaland, defender Gvardiol's goal against United was his fourth this season, making him their second highest scorer in all competitions behind the Norwegian striker, who has 18. Goalkeeper Ederson, so reliable for so long, has already been dropped once this season and did not cover himself in glory for United's winner. Guardiola, with that freshly signed two-year contract, insists he \"wants it\" as he treads on this alien territory of failure. He will be under no illusions about the size of the job in front of him as he placed his head in his hands in anguish after yet another damaging and deeply revealing defeat. City and Guardiola are in new, unforgiving territory.\n", "--------------------------------------------------------------------------------\n", - "Score: 0.3474, Text: 'Self-doubt, errors & big changes' - inside the crisis at Man City\n", + "Vector Distance: 0.3474, Text: 'Self-doubt, errors & big changes' - inside the crisis at Man City\n", "\n", "Pep Guardiola has not been through a moment like this in his managerial career. Manchester City have lost nine matches in their past 12 - as many defeats as they had suffered in their previous 106 fixtures. At the end of October, City were still unbeaten at the top of the Premier League and favourites to win a fifth successive title. Now they are seventh, 12 points behind leaders Liverpool having played a game more. It has been an incredible fall from grace and left people trying to work out what has happened - and whether Guardiola can make it right. After discussing the situation with those who know him best, I have taken a closer look at the future - both short and long term - and how the current crisis at Man City is going to be solved.\n", "\n", @@ -738,7 +732,7 @@ "\n", "Guardiola has never protected his players so much. He has not criticised them and is not going to do so. They have won everything with him. Instead of doing more with them, he has tried doing less. He has sometimes given them more days off to clear their heads, so they can reset - two days this week for instance. Perhaps the time to change a team is when you are winning, but no-one was suggesting Man City were about to collapse when they were top and unbeaten after nine league games. Some people have asked how bad it has to get before City make a decision on Guardiola. The answer is that there is no decision to be made. Maybe if this was Real Madrid, Barcelona or Juventus, the pressure from outside would be massive and the argument would be made that Guardiola has to go. At City he has won the lot, so how can anyone say he is failing? Yes, this is a crisis. But given all their problems, City's renewed target is finishing in the top four. That is what is in all their heads now. The idea is to recover their essence by improving defensive concepts that are not there and re-establishing the intensity they are known for. Guardiola is planning to use the next two years of his contract, which is expected to be his last as a club manager, to prepare a new Manchester City. When he was at the end of his four years at Barcelona, he asked two managers what to do when you feel people are not responding to your instructions. Do you go or do the players go? Sir Alex Ferguson and Rafael Benitez both told him that the players need to go. Guardiola did not listen because of his emotional attachment to his players back then and he decided to leave the Camp Nou because he felt the cycle was over. He will still protect his players now but there is not the same emotional attachment - so it is the players who are going to leave this time. It is likely City will look to replace five or six regular starters. Guardiola knows it is the end of an era and the start of a new one. Changes will not be immediate and the majority of the work will be done in the summer. But they are open to any opportunities in January - and a holding midfielder is one thing they need. In the summer City might want to get Spain's Martin Zubimendi from Real Sociedad and they know 60m euros (£50m) will get him. He said no to Liverpool last summer even though everything was agreed, but he now wants to move on and the Premier League is the target. Even if they do not get Zubimendi, that is the calibre of footballer they are after. A new Manchester City is on its way - with changes driven by Guardiola, incoming sporting director Hugo Viana and the football department.\n", "--------------------------------------------------------------------------------\n", - "Score: 0.3665, Text: Man City's Dias ruled out for 'three or four weeks'\n", + "Vector Distance: 0.3665, Text: Man City's Dias ruled out for 'three or four weeks'\n", "\n", "Ruben Dias has won 10 major trophies during his time at Manchester City\n", "\n", @@ -746,7 +740,7 @@ "\n", "Guardiola's mood was so downbeat in the immediate aftermath of the United defeat it was easy to imagine he might conclude he was no longer capable of doing the job. He gave his players a couple of days off afterwards and was brighter when he spoke to journalists in his scheduled briefing before the Villa trip. \"We'd just finished a game that we lost in the circumstances and I was not happy,\" he said. \"I try to be honest about the feelings of my teams. We fell down six times [number of Premier League games without a win], we have to stand up seven. There is no alternative. \"I'm fine. I'm a normal person with feelings like all of us. When the situation is going well we are better but it's normal. I would not go to the press conference if we were 1-0 up and expressing something that I didn't feel.\" Former Villa forward Jack Grealish has not scored for City in over a year but Guardiola pointed out he is not the only attacking player struggling this season. He added: \"We are struggling to create a little bit up front, but always I am optimistic about my players that they are going to turn and perform well.\"\n", "--------------------------------------------------------------------------------\n", - "Score: 0.3818, Text: 'We have to find a way' - Guardiola vows to end relegation form\n", + "Vector Distance: 0.3818, Text: 'We have to find a way' - Guardiola vows to end relegation form\n", "\n", "This video can not be played To play this video you need to enable JavaScript in your browser. 'Worrying' and 'staggering' - Why do Manchester City keep conceding?\n", "\n", @@ -760,7 +754,7 @@ "\n", "There are chinks in the armour never seen before at City under Guardiola and Erling Haaland conceded belief within the squad is low. He told TNT after the game: \"Of course, [confidence levels are] not the best. We know how important confidence is and you can see that it affects every human being. That is how it is, we have to continue and stay positive even though it is difficult.\" Haaland, with 76 goals in 83 Premier League appearances since joining City from Borussia Dortmund in 2022, had one shot and one touch in the Villa box. His 18 touches in the whole game were the lowest of all starting players and he has been self critical, despite scoring 13 goals in the top flight this season. Over City's last eight games he has netted just twice though, but Guardiola refused to criticise his star striker. He said: \"Without him we will be even worse but I like the players feeling that way. I don't agree with Erling. He needs to have the balls delivered in the right spots but he will fight for the next one.\"\n", "--------------------------------------------------------------------------------\n", - "Score: 0.4157, Text: Man City might miss out on Champions League - Guardiola\n", + "Vector Distance: 0.4157, Text: Man City might miss out on Champions League - Guardiola\n", "\n", "Erling Haaland was part of the Manchester City side that won the Champions League for the first time in 2023\n", "\n", @@ -768,7 +762,7 @@ "\n", "Guardiola's side host Everton on Boxing Day, before a trip to Leicester on 29 December and a home match against West Ham on 4 January. Given all three opponents are in the bottom seven, it offers City a chance to improve on an appalling recent record of four points from eight games, which Guardiola acknowledges has left their lofty European ambitions in doubt. \"The one team that has been in the Champions League for the past years has been Manchester City,\" he added. \"Now we are at risk, of course we are. Definitely.\" Arsenal, Chelsea, Liverpool and Manchester United finished in the Premier League's top four from the 2005-06 season to 2008-09. At least three of them also occupied the top four spots for 15 successive campaigns until 2012. But United have spent five out of the past 11 seasons outside the Champions League. Arsenal spent six seasons out of the competition before returning last term. Liverpool missed out all but one year in seven from 2010, while Chelsea are in their second successive campaign outside Europe's elite. This term the threat to City comes from unexpected sources. As well as Forest and Bournemouth, Aston Villa are ahead of City, while Newcastle, Fulham and Brighton are also within a couple of points. \"There are a lot of contenders,\" said Guardiola, whose side have lost nine of their last 12 games in all competitions. \"For every club it is so important and if we are not winning games, we will be out. \"If we don't qualify it is because we don't deserve it, because we were not prepared and because we had a lot of problems and didn't solve them.\"\n", "--------------------------------------------------------------------------------\n", - "Score: 0.4705, Text: 'So happy he is back' - 'integral' De Bruyne 'one of best we've seen'\n", + "Vector Distance: 0.4705, Text: 'So happy he is back' - 'integral' De Bruyne 'one of best we've seen'\n", "\n", "This video can not be played To play this video you need to enable JavaScript in your browser. Match of the Day: How Kevin de Bruyne inspired Man City back to winning ways\n", "\n", @@ -780,7 +774,7 @@ "\n", "City are now unbeaten in their past 31 Premier League games with De Bruyne starting. He has been involved in 25 goals (nine goals, 16 assists) in those games. De Bruyne said: \"There have never been issues between me and Pep. He knows I've been struggling. It's painful and uncomfortable. \"Hopefully I can get back to my body with not much pain and then I'll be fine.\" However, a bit like the victory itself - tarnished by an injury to Manuel Akanji that may rule the Switzerland defender out of the weekend trip to Crystal Palace, and a hamstring problem for Nathan Ake who has already missed five weeks with a similar injury this season that Guardiola said \"doesn’t look good\" and makes him feel \"sad\" for the Dutchman - there was a caveat as the City boss assessed De Bruyne's contribution. \"He fought a lot and he prepared himself,\" he said. \"He is back to his physicality. The minutes he played at Anfield were really good. \"Last season he was out for many months, this season as well. We will see how he recovers after a long time injured and how he feels in three days.\" De Bruyne recorded four or more shots and created four or more chances for the third time in a Premier League game this season. Despite only starting five Premier League games, only Arsenal's Bukayo Saka has done so more often this term. Guardiola's fear must be that if he pushes De Bruyne too far too quickly, his body will let him down. Former Man City defender Micah Richards told BBC Match of the Day: \"He is a top-quality player and one of the best we have seen. He always manages to find space on the pitch.\" \"He has been integral to Man City's success over a number of years,\" added former City boss Stuart Pearce on Amazon Prime. \"He is the go-to player that sets Erling Haaland alight with his passing. He creates goals, he scores goals. \"If you were to pick one player out over the last eight or nine years De Bruyne would be at the top of almost everyone's list.\" It is a delicate balance given his team are still nine points adrift of Premier League leaders Liverpool and are also outside the Champions League top eight before next week's trip to Italy and a meeting with Juventus, after which City will only have January first-phase games remaining to ensure they secure qualification for the last-16 without needing to be bothered by February's play-off round. However, as with his team, De Bruyne's recovery had to start somewhere.\n", "--------------------------------------------------------------------------------\n", - "Score: 0.4831, Text: 'Life is not easy' - Haaland penalty miss sums up Man City crisis\n", + "Vector Distance: 0.4831, Text: 'Life is not easy' - Haaland penalty miss sums up Man City crisis\n", "\n", "Manchester City striker Erling Haaland has now missed two of his 17 penalties taken in the Premier League\n", "\n", @@ -796,7 +790,7 @@ "\n", "This video can not be played To play this video you need to enable JavaScript in your browser. Dyche 'very pleased' with Everton defence in Man City draw\n", "--------------------------------------------------------------------------------\n", - "Score: 0.5382, Text: Amorim knows job in 'danger' without victories\n", + "Vector Distance: 0.5382, Text: Amorim knows job in 'danger' without victories\n", "\n", "This video can not be played To play this video you need to enable JavaScript in your browser. 'I know that every manager is in danger'\n", "\n", @@ -918,7 +912,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 14, "id": "d1f4812c", "metadata": {}, "outputs": [ @@ -926,7 +920,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-09-23 11:47:17,010 - INFO - Semantic search completed in 1.28 seconds\n" + "2025-11-06 13:09:36,866 - INFO - Semantic search completed in 0.56 seconds\n" ] }, { @@ -934,15 +928,15 @@ "output_type": "stream", "text": [ "\n", - "Semantic Search Results (completed in 1.28 seconds):\n", + "Semantic Search Results (completed in 0.56 seconds):\n", "--------------------------------------------------------------------------------\n", - "Score: 0.2955, Text: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", + "Vector Distance: 0.2956, Text: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", "\n", "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affected by the worst run of results in his entire managerial career. In an interview with former Italy international Luca Toni for Amazon Prime Sport before Wednesday's Champions League defeat by Juventus, Guardiola touched on the personal impact City's sudden downturn in form has had. Guardiola said his state of mind was \"ugly\", that his sleep was \"worse\" and he was eating lighter as his digestion had suffered. City go into Sunday's derby against Manchester United at Etihad Stadium having won just one of their past 10 games. The Juventus loss means there is a chance they may not even secure a play-off spot in the Champions League. Asked to elaborate on his comments to Toni, Guardiola said: \"I'm fine. \"In our jobs we always want to do our best or the best as possible. When that doesn't happen you are more uncomfortable than when the situation is going well, always that happened. \"In good moments I am happier but when I get to the next game I am still concerned about what I have to do. There is no human being that makes an activity and it doesn't matter how they do.\" Guardiola said City have to defend better and \"avoid making mistakes at both ends\". To emphasise his point, Guardiola referred back to the third game of City's current run, against a Sporting side managed by Ruben Amorim, who will be in the United dugout at the weekend. City dominated the first half in Lisbon, led thanks to Phil Foden's early effort and looked to be cruising. Instead, they conceded three times in 11 minutes either side of half-time as Sporting eventually ran out 4-1 winners. \"I would like to play the game like we played in Lisbon on Sunday, believe me,\" said Guardiola, who is facing the prospect of only having three fit defenders for the derby as Nathan Ake and Manuel Akanji try to overcome injury concerns. If there is solace for City, it comes from the knowledge United are not exactly flying. Their comeback Europa League victory against Viktoria Plzen on Thursday was their third win of Amorim's short reign so far but only one of those successes has come in the Premier League, where United have lost their past two games against Arsenal and Nottingham Forest. Nevertheless, Guardiola can see improvements already on the red side of the city. \"It's already there,\" he said. \"You see all the patterns, the movements, the runners and the pace. He will do a good job at United, I'm pretty sure of that.\"\n", "\n", "Guardiola says skipper Kyle Walker has been offered support by the club after the City defender highlighted the racial abuse he had received on social media in the wake of the Juventus trip. \"It's unacceptable,\" he said. \"Not because it's Kyle - for any human being. \"Unfortunately it happens many times in the real world. It is not necessary to say he has the support of the entire club. It is completely unacceptable and we give our support to him.\"\n", "--------------------------------------------------------------------------------\n", - "Score: 0.3100, Text: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", + "Vector Distance: 0.3100, Text: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", "\n", "The former Barcelona and Bayern Munich boss has won 15 major trophies since taking charge of City in 2016.\n", "\n", @@ -968,7 +962,7 @@ "\n", "\"We have to win the game and not look at what happens in the next one yet.\"\n", "--------------------------------------------------------------------------------\n", - "Score: 0.3311, Text: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", + "Vector Distance: 0.3311, Text: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", "\n", "This video can not be played To play this video you need to enable JavaScript in your browser. 'I am not good enough' - Guardiola says he must find a 'solution' after derby loss\n", "\n", @@ -978,7 +972,7 @@ "\n", "Guardiola suggested the serious renewal will wait until the summer but the red flags have been appearing for weeks in the sudden and shocking decline of a team that has lost the aura of invincibility that left many opponents beaten before kick-off in previous years. He has had stated City must \"survive\" this season - whatever qualifies as survival for a club of such rich ambition - but the quest for a record fifth successive Premier League title is surely over as they lie nine points behind leaders Liverpool having played a game more. Their Champions League aspirations are also in jeopardy after another loss, this time against Juventus in Turin. City's squad has been allowed to grow too old together. The insatiable thirst for success seems to have gone, the scales of superiority have fallen away and opponents now sense vulnerability right until the final whistle, as United did here. The manner in which United were able, and felt able, to snatch this victory drove right to the heart of how City, and Guardiola, are allowing opponents to prey on their downfall. Guardiola has every reason to cite injuries, most significantly to Rodri and also John Stones as well as others, but this cannot be used an excuse for such a dramatic decline in standards, allied to the appearance of a soft underbelly that is so easily exploited. And City's rebuild will not be a quick fix. With every performance, every defeat, the scale of what lies in front of Guardiola becomes more obvious - and daunting. Manchester City's fans did their best to reassure Guardiola of their faith in him with a giant Barcelona-inspired banner draped from the stands before kick-off emblazoned with his image reading \"Més que un entrenador\" - \"More Than A Coach\". And Guardiola will now need to be more than a coach than at any time in his career. He will have the finances but it will be done with City's challengers also strengthening. Kevin de Bruyne, 34 in June, lasted 68 minutes here before he was substituted. Age and injuries are catching up with one of the greatest players of the Premier League era and he is unlikely to be at City next season. Mateo Kovacic, who replaced De Bruyne, is also 31 in May. Kyle Walker, 34, is being increasingly exposed. His most notable contribution here was an embarrassing collapse to the ground after the mildest head-to-head collision with Rasmus Hojlund. Ilkay Gundogan, another 34-year-old and a previous pillar of Guardiola's great successes, no longer has the legs or energy to exert influence. This looks increasingly like a season too far following his return from Barcelona. Flaws are also being exposed elsewhere, with previously reliable performers failing to hit previous standards. Phil Foden scored 27 goals and had 12 assists when he was Premier League Player of the Season last term. This year he has just three goals and two assists in 18 appearances in all competitions. He has no goals and just one assist in 11 Premier League games. Jack Grealish, who came on after 77 minutes against United, has not scored in a year for Manchester City, his last goal coming in a 2-2 draw against Crystal Palace on 16 December last year. He has, in the meantime, scored twice for England. Erling Haaland is also struggling as City lack creativity and cutting edge. He has three goals in his past 11 Premier League games after scoring 10 in his first five. And in another indication of City's impotence, and their reliance on Haaland, defender Gvardiol's goal against United was his fourth this season, making him their second highest scorer in all competitions behind the Norwegian striker, who has 18. Goalkeeper Ederson, so reliable for so long, has already been dropped once this season and did not cover himself in glory for United's winner. Guardiola, with that freshly signed two-year contract, insists he \"wants it\" as he treads on this alien territory of failure. He will be under no illusions about the size of the job in front of him as he placed his head in his hands in anguish after yet another damaging and deeply revealing defeat. City and Guardiola are in new, unforgiving territory.\n", "--------------------------------------------------------------------------------\n", - "Score: 0.3474, Text: 'Self-doubt, errors & big changes' - inside the crisis at Man City\n", + "Vector Distance: 0.3474, Text: 'Self-doubt, errors & big changes' - inside the crisis at Man City\n", "\n", "Pep Guardiola has not been through a moment like this in his managerial career. Manchester City have lost nine matches in their past 12 - as many defeats as they had suffered in their previous 106 fixtures. At the end of October, City were still unbeaten at the top of the Premier League and favourites to win a fifth successive title. Now they are seventh, 12 points behind leaders Liverpool having played a game more. It has been an incredible fall from grace and left people trying to work out what has happened - and whether Guardiola can make it right. After discussing the situation with those who know him best, I have taken a closer look at the future - both short and long term - and how the current crisis at Man City is going to be solved.\n", "\n", @@ -988,7 +982,7 @@ "\n", "Guardiola has never protected his players so much. He has not criticised them and is not going to do so. They have won everything with him. Instead of doing more with them, he has tried doing less. He has sometimes given them more days off to clear their heads, so they can reset - two days this week for instance. Perhaps the time to change a team is when you are winning, but no-one was suggesting Man City were about to collapse when they were top and unbeaten after nine league games. Some people have asked how bad it has to get before City make a decision on Guardiola. The answer is that there is no decision to be made. Maybe if this was Real Madrid, Barcelona or Juventus, the pressure from outside would be massive and the argument would be made that Guardiola has to go. At City he has won the lot, so how can anyone say he is failing? Yes, this is a crisis. But given all their problems, City's renewed target is finishing in the top four. That is what is in all their heads now. The idea is to recover their essence by improving defensive concepts that are not there and re-establishing the intensity they are known for. Guardiola is planning to use the next two years of his contract, which is expected to be his last as a club manager, to prepare a new Manchester City. When he was at the end of his four years at Barcelona, he asked two managers what to do when you feel people are not responding to your instructions. Do you go or do the players go? Sir Alex Ferguson and Rafael Benitez both told him that the players need to go. Guardiola did not listen because of his emotional attachment to his players back then and he decided to leave the Camp Nou because he felt the cycle was over. He will still protect his players now but there is not the same emotional attachment - so it is the players who are going to leave this time. It is likely City will look to replace five or six regular starters. Guardiola knows it is the end of an era and the start of a new one. Changes will not be immediate and the majority of the work will be done in the summer. But they are open to any opportunities in January - and a holding midfielder is one thing they need. In the summer City might want to get Spain's Martin Zubimendi from Real Sociedad and they know 60m euros (£50m) will get him. He said no to Liverpool last summer even though everything was agreed, but he now wants to move on and the Premier League is the target. Even if they do not get Zubimendi, that is the calibre of footballer they are after. A new Manchester City is on its way - with changes driven by Guardiola, incoming sporting director Hugo Viana and the football department.\n", "--------------------------------------------------------------------------------\n", - "Score: 0.3665, Text: Man City's Dias ruled out for 'three or four weeks'\n", + "Vector Distance: 0.3665, Text: Man City's Dias ruled out for 'three or four weeks'\n", "\n", "Ruben Dias has won 10 major trophies during his time at Manchester City\n", "\n", @@ -996,7 +990,7 @@ "\n", "Guardiola's mood was so downbeat in the immediate aftermath of the United defeat it was easy to imagine he might conclude he was no longer capable of doing the job. He gave his players a couple of days off afterwards and was brighter when he spoke to journalists in his scheduled briefing before the Villa trip. \"We'd just finished a game that we lost in the circumstances and I was not happy,\" he said. \"I try to be honest about the feelings of my teams. We fell down six times [number of Premier League games without a win], we have to stand up seven. There is no alternative. \"I'm fine. I'm a normal person with feelings like all of us. When the situation is going well we are better but it's normal. I would not go to the press conference if we were 1-0 up and expressing something that I didn't feel.\" Former Villa forward Jack Grealish has not scored for City in over a year but Guardiola pointed out he is not the only attacking player struggling this season. He added: \"We are struggling to create a little bit up front, but always I am optimistic about my players that they are going to turn and perform well.\"\n", "--------------------------------------------------------------------------------\n", - "Score: 0.3818, Text: 'We have to find a way' - Guardiola vows to end relegation form\n", + "Vector Distance: 0.3818, Text: 'We have to find a way' - Guardiola vows to end relegation form\n", "\n", "This video can not be played To play this video you need to enable JavaScript in your browser. 'Worrying' and 'staggering' - Why do Manchester City keep conceding?\n", "\n", @@ -1010,7 +1004,7 @@ "\n", "There are chinks in the armour never seen before at City under Guardiola and Erling Haaland conceded belief within the squad is low. He told TNT after the game: \"Of course, [confidence levels are] not the best. We know how important confidence is and you can see that it affects every human being. That is how it is, we have to continue and stay positive even though it is difficult.\" Haaland, with 76 goals in 83 Premier League appearances since joining City from Borussia Dortmund in 2022, had one shot and one touch in the Villa box. His 18 touches in the whole game were the lowest of all starting players and he has been self critical, despite scoring 13 goals in the top flight this season. Over City's last eight games he has netted just twice though, but Guardiola refused to criticise his star striker. He said: \"Without him we will be even worse but I like the players feeling that way. I don't agree with Erling. He needs to have the balls delivered in the right spots but he will fight for the next one.\"\n", "--------------------------------------------------------------------------------\n", - "Score: 0.4157, Text: Man City might miss out on Champions League - Guardiola\n", + "Vector Distance: 0.4157, Text: Man City might miss out on Champions League - Guardiola\n", "\n", "Erling Haaland was part of the Manchester City side that won the Champions League for the first time in 2023\n", "\n", @@ -1018,7 +1012,7 @@ "\n", "Guardiola's side host Everton on Boxing Day, before a trip to Leicester on 29 December and a home match against West Ham on 4 January. Given all three opponents are in the bottom seven, it offers City a chance to improve on an appalling recent record of four points from eight games, which Guardiola acknowledges has left their lofty European ambitions in doubt. \"The one team that has been in the Champions League for the past years has been Manchester City,\" he added. \"Now we are at risk, of course we are. Definitely.\" Arsenal, Chelsea, Liverpool and Manchester United finished in the Premier League's top four from the 2005-06 season to 2008-09. At least three of them also occupied the top four spots for 15 successive campaigns until 2012. But United have spent five out of the past 11 seasons outside the Champions League. Arsenal spent six seasons out of the competition before returning last term. Liverpool missed out all but one year in seven from 2010, while Chelsea are in their second successive campaign outside Europe's elite. This term the threat to City comes from unexpected sources. As well as Forest and Bournemouth, Aston Villa are ahead of City, while Newcastle, Fulham and Brighton are also within a couple of points. \"There are a lot of contenders,\" said Guardiola, whose side have lost nine of their last 12 games in all competitions. \"For every club it is so important and if we are not winning games, we will be out. \"If we don't qualify it is because we don't deserve it, because we were not prepared and because we had a lot of problems and didn't solve them.\"\n", "--------------------------------------------------------------------------------\n", - "Score: 0.4705, Text: 'So happy he is back' - 'integral' De Bruyne 'one of best we've seen'\n", + "Vector Distance: 0.4705, Text: 'So happy he is back' - 'integral' De Bruyne 'one of best we've seen'\n", "\n", "This video can not be played To play this video you need to enable JavaScript in your browser. Match of the Day: How Kevin de Bruyne inspired Man City back to winning ways\n", "\n", @@ -1030,7 +1024,7 @@ "\n", "City are now unbeaten in their past 31 Premier League games with De Bruyne starting. He has been involved in 25 goals (nine goals, 16 assists) in those games. De Bruyne said: \"There have never been issues between me and Pep. He knows I've been struggling. It's painful and uncomfortable. \"Hopefully I can get back to my body with not much pain and then I'll be fine.\" However, a bit like the victory itself - tarnished by an injury to Manuel Akanji that may rule the Switzerland defender out of the weekend trip to Crystal Palace, and a hamstring problem for Nathan Ake who has already missed five weeks with a similar injury this season that Guardiola said \"doesn’t look good\" and makes him feel \"sad\" for the Dutchman - there was a caveat as the City boss assessed De Bruyne's contribution. \"He fought a lot and he prepared himself,\" he said. \"He is back to his physicality. The minutes he played at Anfield were really good. \"Last season he was out for many months, this season as well. We will see how he recovers after a long time injured and how he feels in three days.\" De Bruyne recorded four or more shots and created four or more chances for the third time in a Premier League game this season. Despite only starting five Premier League games, only Arsenal's Bukayo Saka has done so more often this term. Guardiola's fear must be that if he pushes De Bruyne too far too quickly, his body will let him down. Former Man City defender Micah Richards told BBC Match of the Day: \"He is a top-quality player and one of the best we have seen. He always manages to find space on the pitch.\" \"He has been integral to Man City's success over a number of years,\" added former City boss Stuart Pearce on Amazon Prime. \"He is the go-to player that sets Erling Haaland alight with his passing. He creates goals, he scores goals. \"If you were to pick one player out over the last eight or nine years De Bruyne would be at the top of almost everyone's list.\" It is a delicate balance given his team are still nine points adrift of Premier League leaders Liverpool and are also outside the Champions League top eight before next week's trip to Italy and a meeting with Juventus, after which City will only have January first-phase games remaining to ensure they secure qualification for the last-16 without needing to be bothered by February's play-off round. However, as with his team, De Bruyne's recovery had to start somewhere.\n", "--------------------------------------------------------------------------------\n", - "Score: 0.4831, Text: 'Life is not easy' - Haaland penalty miss sums up Man City crisis\n", + "Vector Distance: 0.4831, Text: 'Life is not easy' - Haaland penalty miss sums up Man City crisis\n", "\n", "Manchester City striker Erling Haaland has now missed two of his 17 penalties taken in the Premier League\n", "\n", @@ -1046,7 +1040,7 @@ "\n", "This video can not be played To play this video you need to enable JavaScript in your browser. Dyche 'very pleased' with Everton defence in Man City draw\n", "--------------------------------------------------------------------------------\n", - "Score: 0.5382, Text: Amorim knows job in 'danger' without victories\n", + "Vector Distance: 0.5382, Text: Amorim knows job in 'danger' without victories\n", "\n", "This video can not be played To play this video you need to enable JavaScript in your browser. 'I know that every manager is in danger'\n", "\n", @@ -1085,6 +1079,479 @@ " raise RuntimeError(f\"Unexpected error: {str(e)}\")" ] }, + { + "cell_type": "markdown", + "id": "bd651112", + "metadata": {}, + "source": [ + "# Vector Search Performance Optimization\n", + "\n", + "Now let's demonstrate the performance benefits of different optimization approaches available in Couchbase. We'll compare three optimization levels to show how each contributes to building a production-ready semantic search system:\n", + "\n", + "1. **Baseline (Raw Search)**: Basic vector similarity search without GSI optimization\n", + "2. **GSI-Optimized Search**: High-performance search using BHIVE GSI index\n", + "3. **Cache Benefits**: Show how caching can be applied on top of any search approach\n", + "\n", + "**Important**: Caching is orthogonal to index types - you can apply caching benefits to both raw searches and GSI-optimized searches to improve repeated query performance.\n" + ] + }, + { + "cell_type": "markdown", + "id": "6eef087c", + "metadata": {}, + "source": [ + "## Understanding Vector Search Results\n", + "\n", + "Before we start our performance comparisons, let's understand what the search results mean:\n", + "\n", + "When you perform a search query with vector search:\n", + "\n", + "1. **Query Embedding**: Your search text is converted into a vector embedding using the OpenAI model\n", + "2. **Vector Similarity Calculation**: The system compares your query vector against all stored document vectors\n", + "3. **Distance Computation**: Using the COSINE distance metric, the system calculates similarity distances\n", + "4. **Result Ranking**: Documents are ranked by their distance values (lower = more similar)\n", + "5. **Post-processing**: Results include both the document content and metadata\n", + "\n", + "**Note**: The returned value represents the vector distance between query and document embeddings. Lower distance values indicate higher similarity.\n" + ] + }, + { + "cell_type": "markdown", + "id": "9f0b75c4", + "metadata": {}, + "source": [ + "## Performance Testing Search Function\n", + "\n", + "Let's create a comprehensive search function for our performance comparison:\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "b901a559", + "metadata": {}, + "outputs": [], + "source": [ + "def search_with_performance_metrics(query_text, stage_name, k=3):\n", + " \"\"\"Perform optimized semantic search with detailed performance metrics\"\"\"\n", + " print(f\"\\n=== {stage_name.upper()} ===\")\n", + " print(f\"Query: \\\"{query_text}\\\"\")\n", + " \n", + " start_time = time.time()\n", + " results = vector_store.similarity_search_with_score(query_text, k=k)\n", + " end_time = time.time()\n", + " \n", + " search_time = end_time - start_time\n", + " print(f\"Search Time: {search_time:.4f} seconds\")\n", + " print(f\"Results Found: {len(results)} documents\")\n", + " \n", + " for i, (doc, distance) in enumerate(results, 1):\n", + " print(f\"\\n[Result {i}]\")\n", + " print(f\"Vector Distance: {distance:.6f} (lower = more similar)\")\n", + " # Truncate long documents for readability\n", + " content_preview = doc.page_content[:200] + \"...\" if len(doc.page_content) > 200 else doc.page_content\n", + " print(f\"Document Content: {content_preview}\")\n", + " if hasattr(doc, 'metadata') and doc.metadata:\n", + " print(f\"Metadata: {doc.metadata}\")\n", + " \n", + " return search_time, results\n" + ] + }, + { + "cell_type": "markdown", + "id": "206089f3", + "metadata": {}, + "source": [ + "## Phase 1: Baseline Performance (Raw Vector Search)\n", + "\n", + "First, let's establish baseline performance with raw vector search - no GSI optimization yet.\n", + "\n", + "**Note**: For this demonstration, we'll temporarily remove any existing GSI indexes to show the baseline performance without optimization.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "5d1a1d19", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2025-11-06 13:09:54,294 - INFO - No existing index to drop or error: ParsingFailedException()\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Testing baseline performance without GSI optimization...\n", + "\n", + "=== PHASE 1: BASELINE VECTOR SEARCH ===\n", + "Query: \"What was manchester city manager pep guardiola's reaction to the team's current form?\"\n", + "Search Time: 0.4738 seconds\n", + "Results Found: 3 documents\n", + "\n", + "[Result 1]\n", + "Vector Distance: 0.295601 (lower = more similar)\n", + "Document Content: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", + "\n", + "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affecte...\n", + "\n", + "[Result 2]\n", + "Vector Distance: 0.309984 (lower = more similar)\n", + "Document Content: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", + "\n", + "The former Barcelona and Bayern Munich boss has won 15 major trophi...\n", + "\n", + "[Result 3]\n", + "Vector Distance: 0.331093 (lower = more similar)\n", + "Document Content: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. 'I am not good enough' - Guardiola s...\n" + ] + } + ], + "source": [ + "# Drop any existing GSI index to demonstrate baseline performance\n", + "try:\n", + " cluster.query(f\"DROP INDEX `{CB_BUCKET_NAME}`.`{SCOPE_NAME}`.`{COLLECTION_NAME}`.pydantic_ai_bhive_index\").execute()\n", + " logging.info(\"Dropped existing BHIVE index for baseline testing\")\n", + " time.sleep(2) # Wait for index drop to complete\n", + "except Exception as e:\n", + " logging.info(f\"No existing index to drop or error: {str(e)}\")\n", + "\n", + "test_query = \"What was manchester city manager pep guardiola's reaction to the team's current form?\"\n", + "print(\"Testing baseline performance without GSI optimization...\")\n", + "baseline_time, baseline_results = search_with_performance_metrics(\n", + " test_query, \"Phase 1: Baseline Vector Search\"\n", + ")\n" + ] + }, + { + "cell_type": "markdown", + "id": "5d528ee6", + "metadata": {}, + "source": [ + "## Phase 2: Create BHIVE GSI Index and Test Performance\n", + "\n", + "Now let's create the BHIVE GSI index and measure the performance improvement:\n" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "73e9a28a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Creating BHIVE GSI vector index...\n", + "✓ BHIVE GSI vector index already exists, proceeding...\n", + "\n", + "Testing performance with BHIVE GSI optimization...\n", + "\n", + "=== PHASE 2: GSI-OPTIMIZED SEARCH ===\n", + "Query: \"What was manchester city manager pep guardiola's reaction to the team's current form?\"\n", + "Search Time: 0.5026 seconds\n", + "Results Found: 3 documents\n", + "\n", + "[Result 1]\n", + "Vector Distance: 0.295612 (lower = more similar)\n", + "Document Content: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", + "\n", + "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affecte...\n", + "\n", + "[Result 2]\n", + "Vector Distance: 0.310004 (lower = more similar)\n", + "Document Content: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", + "\n", + "The former Barcelona and Bayern Munich boss has won 15 major trophi...\n", + "\n", + "[Result 3]\n", + "Vector Distance: 0.331096 (lower = more similar)\n", + "Document Content: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. 'I am not good enough' - Guardiola s...\n" + ] + } + ], + "source": [ + "# Create BHIVE index for optimized vector search\n", + "print(\"Creating BHIVE GSI vector index...\")\n", + "try:\n", + " vector_store.create_index(\n", + " index_type=IndexType.BHIVE,\n", + " index_description=\"IVF,SQ8\",\n", + " distance_metric=DistanceStrategy.COSINE,\n", + " index_name=\"pydantic_ai_bhive_index\",\n", + " )\n", + " print(\"✓ BHIVE GSI vector index created successfully!\")\n", + " \n", + " # Wait for index to become available\n", + " print(\"Waiting for index to become available...\")\n", + " time.sleep(3)\n", + " \n", + "except Exception as e:\n", + " if \"already exists\" in str(e).lower():\n", + " print(\"✓ BHIVE GSI vector index already exists, proceeding...\")\n", + " else:\n", + " print(f\"Error creating GSI index: {str(e)}\")\n", + "\n", + "# Test the same query with GSI optimization\n", + "print(\"\\nTesting performance with BHIVE GSI optimization...\")\n", + "gsi_time, gsi_results = search_with_performance_metrics(\n", + " test_query, \"Phase 2: GSI-Optimized Search\"\n", + ")\n" + ] + }, + { + "cell_type": "markdown", + "id": "01132754", + "metadata": {}, + "source": [ + "## Phase 3: Demonstrate Cache Benefits\n", + "\n", + "Now let's show how caching can improve performance for repeated queries. **Note**: Caching benefits apply to both raw searches and GSI-optimized searches.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "c8b93f92", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Setting up Couchbase cache for improved performance on repeated queries...\n", + "✓ Couchbase cache enabled!\n", + "\n", + "Testing cache benefits with a different query...\n", + "First execution (cache miss):\n", + "\n", + "=== PHASE 3A: FIRST QUERY (CACHE MISS) ===\n", + "Query: \"How does Manchester United's performance compare to Manchester City?\"\n", + "Search Time: 0.7791 seconds\n", + "Results Found: 2 documents\n", + "\n", + "[Result 1]\n", + "Vector Distance: 0.461303 (lower = more similar)\n", + "Document Content: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. 'I am not good enough' - Guardiola s...\n", + "\n", + "[Result 2]\n", + "Vector Distance: 0.472090 (lower = more similar)\n", + "Document Content: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", + "\n", + "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affecte...\n", + "\n", + "Second execution (cache hit):\n", + "\n", + "=== PHASE 3B: REPEATED QUERY (CACHE HIT) ===\n", + "Query: \"How does Manchester United's performance compare to Manchester City?\"\n", + "Search Time: 0.7957 seconds\n", + "Results Found: 2 documents\n", + "\n", + "[Result 1]\n", + "Vector Distance: 0.461303 (lower = more similar)\n", + "Document Content: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", + "\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. 'I am not good enough' - Guardiola s...\n", + "\n", + "[Result 2]\n", + "Vector Distance: 0.472090 (lower = more similar)\n", + "Document Content: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", + "\n", + "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affecte...\n" + ] + } + ], + "source": [ + "# Import cache utilities\n", + "from langchain_core.globals import set_llm_cache\n", + "from langchain_couchbase.cache import CouchbaseCache\n", + "\n", + "# Set up Couchbase cache (can be applied to any search approach)\n", + "print(\"Setting up Couchbase cache for improved performance on repeated queries...\")\n", + "cache = CouchbaseCache(\n", + " cluster=cluster,\n", + " bucket_name=CB_BUCKET_NAME,\n", + " scope_name=SCOPE_NAME,\n", + " collection_name=COLLECTION_NAME,\n", + ")\n", + "set_llm_cache(cache)\n", + "print(\"✓ Couchbase cache enabled!\")\n", + "\n", + "# Test cache benefits with the same query (should show improvement on second run)\n", + "cache_query = \"How does Manchester United's performance compare to Manchester City?\"\n", + "\n", + "print(\"\\nTesting cache benefits with a different query...\")\n", + "print(\"First execution (cache miss):\")\n", + "cache_time_1, _ = search_with_performance_metrics(\n", + " cache_query, \"Phase 3a: First Query (Cache Miss)\", k=2\n", + ")\n", + "\n", + "print(\"\\nSecond execution (cache hit):\")\n", + "cache_time_2, _ = search_with_performance_metrics(\n", + " cache_query, \"Phase 3b: Repeated Query (Cache Hit)\", k=2\n", + ")\n" + ] + }, + { + "cell_type": "markdown", + "id": "f800b0c0", + "metadata": {}, + "source": [ + "## Complete Performance Analysis\n", + "\n", + "Let's analyze the complete performance improvements across all optimization levels:\n" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "6a6e909f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "================================================================================\n", + "VECTOR SEARCH PERFORMANCE OPTIMIZATION SUMMARY\n", + "================================================================================\n", + "Phase 1 - Baseline (Raw Search): 0.4738 seconds\n", + "Phase 2 - GSI-Optimized Search: 0.5026 seconds\n", + "Phase 3 - Cache Benefits:\n", + " First execution (cache miss): 0.7791 seconds\n", + " Second execution (cache hit): 0.7957 seconds\n", + "\n", + "--------------------------------------------------------------------------------\n", + "OPTIMIZATION IMPACT ANALYSIS:\n", + "--------------------------------------------------------------------------------\n", + "GSI Index Benefit: Performance similar to baseline (may vary with dataset size)\n", + "Cache Benefit: No significant improvement (results may be cached already)\n", + "\n", + "Key Insights:\n", + "• GSI optimization provides consistent performance benefits, especially with larger datasets\n", + "• Caching benefits apply to both raw and GSI-optimized searches\n", + "• Combined GSI + Cache provides the best performance for production applications\n", + "• BHIVE indexes scale to billions of vectors with optimized concurrent operations\n" + ] + } + ], + "source": [ + "print(\"\\n\" + \"=\"*80)\n", + "print(\"VECTOR SEARCH PERFORMANCE OPTIMIZATION SUMMARY\")\n", + "print(\"=\"*80)\n", + "\n", + "print(f\"Phase 1 - Baseline (Raw Search): {baseline_time:.4f} seconds\")\n", + "print(f\"Phase 2 - GSI-Optimized Search: {gsi_time:.4f} seconds\")\n", + "print(f\"Phase 3 - Cache Benefits:\")\n", + "print(f\" First execution (cache miss): {cache_time_1:.4f} seconds\")\n", + "print(f\" Second execution (cache hit): {cache_time_2:.4f} seconds\")\n", + "\n", + "print(\"\\n\" + \"-\"*80)\n", + "print(\"OPTIMIZATION IMPACT ANALYSIS:\")\n", + "print(\"-\"*80)\n", + "\n", + "# GSI improvement analysis\n", + "if gsi_time and baseline_time and gsi_time < baseline_time:\n", + " gsi_speedup = baseline_time / gsi_time\n", + " gsi_improvement = ((baseline_time - gsi_time) / baseline_time) * 100\n", + " print(f\"GSI Index Benefit: {gsi_speedup:.2f}x faster ({gsi_improvement:.1f}% improvement)\")\n", + "else:\n", + " print(f\"GSI Index Benefit: Performance similar to baseline (may vary with dataset size)\")\n", + "\n", + "# Cache improvement analysis\n", + "if cache_time_2 and cache_time_1 and cache_time_2 < cache_time_1:\n", + " cache_speedup = cache_time_1 / cache_time_2\n", + " cache_improvement = ((cache_time_1 - cache_time_2) / cache_time_1) * 100\n", + " print(f\"Cache Benefit: {cache_speedup:.2f}x faster ({cache_improvement:.1f}% improvement)\")\n", + "else:\n", + " print(f\"Cache Benefit: No significant improvement (results may be cached already)\")\n", + "\n", + "print(f\"\\nKey Insights:\")\n", + "print(f\"• GSI optimization provides consistent performance benefits, especially with larger datasets\")\n", + "print(f\"• Caching benefits apply to both raw and GSI-optimized searches\")\n", + "print(f\"• Combined GSI + Cache provides the best performance for production applications\")\n", + "print(f\"• BHIVE indexes scale to billions of vectors with optimized concurrent operations\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "898161ed", + "metadata": {}, + "source": [ + "## Interactive Testing\n", + "\n", + "Try your own queries with the optimized search system:\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "00a3146f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "=== INTERACTIVE GSI-OPTIMIZED SEARCH ===\n", + "Query: \"Tell me about Manchester City Uniform\"\n", + "Search Time: 0.5068 seconds\n", + "Results Found: 3 documents\n", + "\n", + "[Result 1]\n", + "Vector Distance: 0.469613 (lower = more similar)\n", + "Document Content: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", + "\n", + "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affecte...\n", + "\n", + "[Result 2]\n", + "Vector Distance: 0.491903 (lower = more similar)\n", + "Document Content: 'Self-doubt, errors & big changes' - inside the crisis at Man City\n", + "\n", + "Pep Guardiola has not been through a moment like this in his managerial career. Manchester City have lost nine matches in their past...\n", + "\n", + "[Result 3]\n", + "Vector Distance: 0.515922 (lower = more similar)\n", + "Document Content: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", + "\n", + "The former Barcelona and Bayern Munich boss has won 15 major trophi...\n" + ] + }, + { + "data": { + "text/plain": [ + "(0.5067887306213379,\n", + " [(Document(id='33e94f118f7946ab8b8c3a41bf886184', metadata={}, page_content='Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\\n\\nManchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affected by the worst run of results in his entire managerial career. In an interview with former Italy international Luca Toni for Amazon Prime Sport before Wednesday\\'s Champions League defeat by Juventus, Guardiola touched on the personal impact City\\'s sudden downturn in form has had. Guardiola said his state of mind was \"ugly\", that his sleep was \"worse\" and he was eating lighter as his digestion had suffered. City go into Sunday\\'s derby against Manchester United at Etihad Stadium having won just one of their past 10 games. The Juventus loss means there is a chance they may not even secure a play-off spot in the Champions League. Asked to elaborate on his comments to Toni, Guardiola said: \"I\\'m fine. \"In our jobs we always want to do our best or the best as possible. When that doesn\\'t happen you are more uncomfortable than when the situation is going well, always that happened. \"In good moments I am happier but when I get to the next game I am still concerned about what I have to do. There is no human being that makes an activity and it doesn\\'t matter how they do.\" Guardiola said City have to defend better and \"avoid making mistakes at both ends\". To emphasise his point, Guardiola referred back to the third game of City\\'s current run, against a Sporting side managed by Ruben Amorim, who will be in the United dugout at the weekend. City dominated the first half in Lisbon, led thanks to Phil Foden\\'s early effort and looked to be cruising. Instead, they conceded three times in 11 minutes either side of half-time as Sporting eventually ran out 4-1 winners. \"I would like to play the game like we played in Lisbon on Sunday, believe me,\" said Guardiola, who is facing the prospect of only having three fit defenders for the derby as Nathan Ake and Manuel Akanji try to overcome injury concerns. If there is solace for City, it comes from the knowledge United are not exactly flying. Their comeback Europa League victory against Viktoria Plzen on Thursday was their third win of Amorim\\'s short reign so far but only one of those successes has come in the Premier League, where United have lost their past two games against Arsenal and Nottingham Forest. Nevertheless, Guardiola can see improvements already on the red side of the city. \"It\\'s already there,\" he said. \"You see all the patterns, the movements, the runners and the pace. He will do a good job at United, I\\'m pretty sure of that.\"\\n\\nGuardiola says skipper Kyle Walker has been offered support by the club after the City defender highlighted the racial abuse he had received on social media in the wake of the Juventus trip. \"It\\'s unacceptable,\" he said. \"Not because it\\'s Kyle - for any human being. \"Unfortunately it happens many times in the real world. It is not necessary to say he has the support of the entire club. It is completely unacceptable and we give our support to him.\"'),\n", + " 0.46961301382250065),\n", + " (Document(id='20b142f6e1f54a568942aac9b4bcdd81', metadata={}, page_content='\\'Self-doubt, errors & big changes\\' - inside the crisis at Man City\\n\\nPep Guardiola has not been through a moment like this in his managerial career. Manchester City have lost nine matches in their past 12 - as many defeats as they had suffered in their previous 106 fixtures. At the end of October, City were still unbeaten at the top of the Premier League and favourites to win a fifth successive title. Now they are seventh, 12 points behind leaders Liverpool having played a game more. It has been an incredible fall from grace and left people trying to work out what has happened - and whether Guardiola can make it right. After discussing the situation with those who know him best, I have taken a closer look at the future - both short and long term - and how the current crisis at Man City is going to be solved.\\n\\nPep Guardiola\\'s Man City have lost nine of their past 12 matches\\n\\nGuardiola has also been giving it a lot of thought. He has not been sleeping very well, as he has said, and has not been himself at times when talking to the media. He has been talking to a lot of people about what is going on as he tries to work out the reasons for City\\'s demise. Some reasons he knows, others he still doesn\\'t. What people perhaps do not realise is Guardiola hugely doubts himself and always has. He will be thinking \"I\\'m not going to be able to get us out of this\" and needs the support of people close to him to push away those insecurities - and he has that. He is protected by his people who are very aware, like he is, that there are a lot of people that want City to fail. It has been a turbulent time for Guardiola. Remember those marks he had on his head after the 3-3 draw with Feyenoord in the Champions League? He always scratches his head, it is a gesture of nervousness. Normally nothing happens but on that day one of his nails was far too sharp so, after talking to the players in the changing room where he scratched his head because of his usual agitated gesturing, he went to the news conference. His right-hand man Manel Estiarte sent him photos in a message saying \"what have you got on your head?\", but by the time Guardiola returned to the coaching room there was hardly anything there again. He started that day with a cover on his nose after the same thing happened at the training ground the day before. Guardiola was having a footballing debate with Kyle Walker about positional stuff and marked his nose with that same nail. There was also that remarkable news conference after the Manchester derby when he said \"I don\\'t know what to do\". That is partly true and partly not true. Ignore the fact Guardiola suggested he was \"not good enough\". He actually meant he was not good enough to resolve the situation with the group of players he has available and with all the other current difficulties. There are obviously logical explanations for the crisis and the first one has been talked about many times - the absence of injured midfielder Rodri. You know the game Jenga? When you take the wrong piece out, the whole tower collapses. That is what has happened here. It is normal for teams to have an over-reliance on one player if he is the best in the world in his position. And you cannot calculate the consequences of an injury that rules someone like Rodri out for the season. City are a team, like many modern ones, in which the holding midfielder is a key element to the construction. So, when you take Rodri out, it is difficult to hold it together. There were Plan Bs - John Stones, Manuel Akanji, even Nathan Ake - but injuries struck. The big injury list has been out of the ordinary and the busy calendar has also played a part in compounding the issues. However, one factor even Guardiola cannot explain is the big uncharacteristic errors in almost every game from international players. Why did Matheus Nunes make that challenge to give away the penalty against Manchester United? Jack Grealish is sent on at the end to keep the ball and cannot do that. There are errors from Walker and other defenders. These are some of the best players in the world. Of course the players\\' mindset is important, and confidence is diminishing. Wrong decisions get taken so there is almost panic on the pitch instead of calm. There are also players badly out of form who are having to play because of injuries. Walker is now unable to hide behind his pace, I\\'m not sure Kevin de Bruyne is ever getting back to the level he used to be at, Bernardo Silva and Ilkay Gundogan do not have time to rest, Grealish is not playing at his best. Some of these players were only meant to be playing one game a week but, because of injuries, have played 12 games in 40 days. It all has a domino effect. One consequence is that Erling Haaland isn\\'t getting the service to score. But the Norwegian still remains City\\'s top-scorer with 13. Defender Josko Gvardiol is next on the list with just four. The way their form has been analysed inside the City camp is there have only been three games where they deserved to lose (Liverpool, Bournemouth and Aston Villa). But of course it is time to change the dynamic.\\n\\nGuardiola has never protected his players so much. He has not criticised them and is not going to do so. They have won everything with him. Instead of doing more with them, he has tried doing less. He has sometimes given them more days off to clear their heads, so they can reset - two days this week for instance. Perhaps the time to change a team is when you are winning, but no-one was suggesting Man City were about to collapse when they were top and unbeaten after nine league games. Some people have asked how bad it has to get before City make a decision on Guardiola. The answer is that there is no decision to be made. Maybe if this was Real Madrid, Barcelona or Juventus, the pressure from outside would be massive and the argument would be made that Guardiola has to go. At City he has won the lot, so how can anyone say he is failing? Yes, this is a crisis. But given all their problems, City\\'s renewed target is finishing in the top four. That is what is in all their heads now. The idea is to recover their essence by improving defensive concepts that are not there and re-establishing the intensity they are known for. Guardiola is planning to use the next two years of his contract, which is expected to be his last as a club manager, to prepare a new Manchester City. When he was at the end of his four years at Barcelona, he asked two managers what to do when you feel people are not responding to your instructions. Do you go or do the players go? Sir Alex Ferguson and Rafael Benitez both told him that the players need to go. Guardiola did not listen because of his emotional attachment to his players back then and he decided to leave the Camp Nou because he felt the cycle was over. He will still protect his players now but there is not the same emotional attachment - so it is the players who are going to leave this time. It is likely City will look to replace five or six regular starters. Guardiola knows it is the end of an era and the start of a new one. Changes will not be immediate and the majority of the work will be done in the summer. But they are open to any opportunities in January - and a holding midfielder is one thing they need. In the summer City might want to get Spain\\'s Martin Zubimendi from Real Sociedad and they know 60m euros (£50m) will get him. He said no to Liverpool last summer even though everything was agreed, but he now wants to move on and the Premier League is the target. Even if they do not get Zubimendi, that is the calibre of footballer they are after. A new Manchester City is on its way - with changes driven by Guardiola, incoming sporting director Hugo Viana and the football department.'),\n", + " 0.4919030863142537),\n", + " (Document(id='77b8b9fea1984454b49a9b566c0d893d', metadata={}, page_content='Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\\n\\nThe former Barcelona and Bayern Munich boss has won 15 major trophies since taking charge of City in 2016.\\n\\nThe 53-year-old Spaniard was approached in the summer about the possibility of becoming England manager, but last month signed a two-year contract extension with City until 2027.\\n\\nSpeaking to celebrity chef Dani Garcia on YouTube, Guardiola did not indicate when he intends to step down at City but said he would not return to club football - in the Premier League or overseas.\\n\\n\"I\\'m not going to manage another team,\" he said.\\n\\n\"I\\'m not talking about the long-term future, but what I\\'m not going to do is leave Manchester City, go to another country, and do the same thing as now.\\n\\n\"I wouldn\\'t have the energy. The thought of starting somewhere else, all the process of training and so on. No, no, no. Maybe a national team, but that\\'s different.\\n\\n\"I want to leave it and go and play golf, but I can\\'t [if he takes a club job]. I think stopping would do me good.\"\\n\\nCity have won just once since Guardiola extended his contract - and once in nine games since beating Southampton on 26 October.\\n\\nThat victory came at home to Nottingham Forest last Wednesday, but was followed by a 2-2 draw at Crystal Palace at the weekend.\\n\\nThe Blues visit Juventus next in the Champions League on Wednesday (20:00 GMT), before hosting Manchester United in the Premier League on Sunday (16:30).\\n\\n\"Right now we are not in the position - when we have had the results of the last seven, eight games - to talk about winning games in plural,\" said Guardiola at his pre-match news conference.\\n\\n\"We have to win the game and not look at what happens in the next one yet.\"'),\n", + " 0.5159221949345671)])" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "custom_query = input(\"Enter your search query: \")\n", + "search_with_performance_metrics(custom_query, \"Interactive GSI-Optimized Search\")\n" + ] + }, { "cell_type": "markdown", "id": "bc188844", @@ -1123,7 +1590,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 21, "id": "ec663fb6", "metadata": {}, "outputs": [], @@ -1150,7 +1617,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 22, "id": "fab753b9", "metadata": {}, "outputs": [], @@ -1180,7 +1647,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 23, "id": "e9a034ab", "metadata": {}, "outputs": [], @@ -1204,7 +1671,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 24, "id": "799e9fb5", "metadata": {}, "outputs": [ @@ -1213,18 +1680,20 @@ "output_type": "stream", "text": [ "==================== Agent Output ====================\n", - "Pep Guardiola has expressed significant concerns about Manchester City's current form, which he describes as the worst run of results in his managerial career. Despite being satisfied with his role at City, Guardiola is personally affected by the team's performance, admitting issues with sleep and diet due to stress. City have lost nine of their past 12 matches, leading to a decline in their Premier League standings. Guardiola acknowledges the difficulty of solving current challenges, particularly in defending and avoiding mistakes at crucial moments.\n", + "Manchester City manager Pep Guardiola has been candid about the team's current form, admitting to personal impacts such as affected sleep and diet due to stress over what he describes as the \"worst run of results\" in his managerial career. He expressed that his state of mind was \"ugly\" and acknowledged that the team's downturn has made him uncomfortable. Guardiola emphasized the team's need to improve defensively and reduce mistakes.\n", "\n", - "The absence of key players like midfielder Rodri has been a critical issue, likened to removing a crucial piece in a game of Jenga, leading to the collapse of the team's structure. Injuries have compounded the challenge, with errors from usually dependable international players adding to the crisis. Guardiola has opted to protect and support his players rather than criticize, allowing them time to reset with more days off.\n", + "In addition, he openly analyzed his own role in the situation, stating, \"I am not good enough. I am the boss. I am the manager. I have to find solutions and so far I haven't.\" Despite this brutal self-criticism, Guardiola remains committed to finding a way to turn the team's fortunes around.\n", "\n", - "Guardiola sees the current tumult as the end of an era and is planning changes that potentially involve a substantial squad overhaul in the upcoming transfer windows. Despite these difficulties, Guardiola remains committed and is using this period to prepare for a new phase at Manchester City under his management.\n" + "Recently, Guardiola noted the need for serious rebuilding, while acknowledging the challenges of an aging squad and injuries to key players. However, he continues to express optimism and trust in his players, believing in their desire and pride to overcome current difficulties.\n", + "\n", + "In summary, Guardiola's reaction to Manchester City's current form reflects a mix of personal distress, professional determination to find solutions, and a realistic acknowledgment of the challenges the team faces.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ - "/var/folders/zd/hmksn4rj5kd_pj7w7bw48y9m0000gn/T/ipykernel_22630/1716427794.py:5: DeprecationWarning: `result.data` is deprecated, use `result.output` instead.\n", + "/var/folders/zd/hmksn4rj5kd_pj7w7bw48y9m0000gn/T/ipykernel_55688/1716427794.py:5: DeprecationWarning: `result.data` is deprecated, use `result.output` instead.\n", " print(output.data)\n" ] } @@ -1254,7 +1723,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 25, "id": "3ea69905", "metadata": {}, "outputs": [ @@ -1265,222 +1734,90 @@ "Step 1:\n", "('ModelRequest(parts=[UserPromptPart(content=\"What was manchester city manager '\n", " 'pep guardiola\\'s reaction to the team\\'s current form?\", '\n", - " 'timestamp=datetime.datetime(2025, 9, 23, 6, 17, 27, 631157, '\n", + " 'timestamp=datetime.datetime(2025, 11, 6, 7, 41, 32, 361643, '\n", " \"tzinfo=datetime.timezone.utc), part_kind='user-prompt')], instructions=None, \"\n", " \"kind='request')\")\n", "==================================================\n", "Step 2:\n", "(\"ModelResponse(parts=[ToolCallPart(tool_name='retrieve', \"\n", - " 'args=\\'{\"search_query\":\"Manchester City Pep Guardiola reaction current form '\n", - " 'October 2023\"}\\', tool_call_id=\\'call_kUqRR8fk7bPUzWxzpkGzt2AP\\', '\n", + " 'args=\\'{\"search_query\":\"Manchester City Pep Guardiola reaction to team\\\\\\'s '\n", + " 'current form\"}\\', tool_call_id=\\'call_kLchsPmUPdNovuMcE2StuBxU\\', '\n", " \"part_kind='tool-call')], model_name='gpt-4o-2024-08-06', \"\n", - " 'timestamp=datetime.datetime(2025, 9, 23, 6, 17, 28, '\n", + " 'timestamp=datetime.datetime(2025, 11, 6, 7, 41, 33, '\n", " \"tzinfo=datetime.timezone.utc), kind='response')\")\n", "==================================================\n", "Step 3:\n", "(\"ModelRequest(parts=[ToolReturnPart(tool_name='retrieve', content='# \"\n", - " 'Documents:\\\\nPep Guardiola has said Manchester City will be his final '\n", - " 'managerial job in club football before he \"maybe\" coaches a national '\n", - " 'team.\\\\n\\\\nThe former Barcelona and Bayern Munich boss has won 15 major '\n", - " 'trophies since taking charge of City in 2016.\\\\n\\\\nThe 53-year-old Spaniard '\n", - " 'was approached in the summer about the possibility of becoming England '\n", - " 'manager, but last month signed a two-year contract extension with City until '\n", - " '2027.\\\\n\\\\nSpeaking to celebrity chef Dani Garcia on YouTube, Guardiola did '\n", - " 'not indicate when he intends to step down at City but said he would not '\n", - " 'return to club football - in the Premier League or overseas.\\\\n\\\\n\"I\\\\\\'m '\n", - " 'not going to manage another team,\" he said.\\\\n\\\\n\"I\\\\\\'m not talking about '\n", - " \"the long-term future, but what I\\\\'m not going to do is leave Manchester \"\n", - " 'City, go to another country, and do the same thing as now.\\\\n\\\\n\"I '\n", - " \"wouldn\\\\'t have the energy. The thought of starting somewhere else, all the \"\n", - " 'process of training and so on. No, no, no. Maybe a national team, but '\n", - " 'that\\\\\\'s different.\\\\n\\\\n\"I want to leave it and go and play golf, but I '\n", - " \"can\\\\'t [if he takes a club job]. I think stopping would do me \"\n", - " 'good.\"\\\\n\\\\nCity have won just once since Guardiola extended his contract - '\n", - " 'and once in nine games since beating Southampton on 26 October.\\\\n\\\\nThat '\n", - " 'victory came at home to Nottingham Forest last Wednesday, but was followed '\n", - " 'by a 2-2 draw at Crystal Palace at the weekend.\\\\n\\\\nThe Blues visit '\n", - " 'Juventus next in the Champions League on Wednesday (20:00 GMT), before '\n", - " 'hosting Manchester United in the Premier League on Sunday '\n", - " '(16:30).\\\\n\\\\n\"Right now we are not in the position - when we have had the '\n", - " 'results of the last seven, eight games - to talk about winning games in '\n", - " 'plural,\" said Guardiola at his pre-match news conference.\\\\n\\\\n\"We have to '\n", - " 'win the game and not look at what happens in the next one yet.\"\\\\n\\\\n# '\n", - " \"Documents:\\\\n\\\\'Self-doubt, errors & big changes\\\\' - inside the crisis at \"\n", - " 'Man City\\\\n\\\\nPep Guardiola has not been through a moment like this in his '\n", - " 'managerial career. Manchester City have lost nine matches in their past 12 - '\n", - " 'as many defeats as they had suffered in their previous 106 fixtures. At the '\n", - " 'end of October, City were still unbeaten at the top of the Premier League '\n", - " 'and favourites to win a fifth successive title. Now they are seventh, 12 '\n", - " 'points behind leaders Liverpool having played a game more. It has been an '\n", - " 'incredible fall from grace and left people trying to work out what has '\n", - " 'happened - and whether Guardiola can make it right. After discussing the '\n", - " 'situation with those who know him best, I have taken a closer look at the '\n", - " 'future - both short and long term - and how the current crisis at Man City '\n", - " \"is going to be solved.\\\\n\\\\nPep Guardiola\\\\'s Man City have lost nine of \"\n", - " 'their past 12 matches\\\\n\\\\nGuardiola has also been giving it a lot of '\n", - " 'thought. He has not been sleeping very well, as he has said, and has not '\n", - " 'been himself at times when talking to the media. He has been talking to a '\n", - " 'lot of people about what is going on as he tries to work out the reasons for '\n", - " \"City\\\\'s demise. Some reasons he knows, others he still doesn\\\\'t. What \"\n", - " 'people perhaps do not realise is Guardiola hugely doubts himself and always '\n", - " 'has. He will be thinking \"I\\\\\\'m not going to be able to get us out of this\" '\n", - " 'and needs the support of people close to him to push away those insecurities '\n", - " '- and he has that. He is protected by his people who are very aware, like he '\n", - " 'is, that there are a lot of people that want City to fail. It has been a '\n", - " 'turbulent time for Guardiola. Remember those marks he had on his head after '\n", - " 'the 3-3 draw with Feyenoord in the Champions League? He always scratches his '\n", - " 'head, it is a gesture of nervousness. Normally nothing happens but on that '\n", - " 'day one of his nails was far too sharp so, after talking to the players in '\n", - " 'the changing room where he scratched his head because of his usual agitated '\n", - " 'gesturing, he went to the news conference. His right-hand man Manel Estiarte '\n", - " 'sent him photos in a message saying \"what have you got on your head?\", but '\n", - " 'by the time Guardiola returned to the coaching room there was hardly '\n", - " 'anything there again. He started that day with a cover on his nose after the '\n", - " 'same thing happened at the training ground the day before. Guardiola was '\n", - " 'having a footballing debate with Kyle Walker about positional stuff and '\n", - " 'marked his nose with that same nail. There was also that remarkable news '\n", - " 'conference after the Manchester derby when he said \"I don\\\\\\'t know what to '\n", - " 'do\". That is partly true and partly not true. Ignore the fact Guardiola '\n", - " 'suggested he was \"not good enough\". He actually meant he was not good enough '\n", - " 'to resolve the situation with the group of players he has available and with '\n", - " 'all the other current difficulties. There are obviously logical explanations '\n", - " 'for the crisis and the first one has been talked about many times - the '\n", - " 'absence of injured midfielder Rodri. You know the game Jenga? When you take '\n", - " 'the wrong piece out, the whole tower collapses. That is what has happened '\n", - " 'here. It is normal for teams to have an over-reliance on one player if he is '\n", - " 'the best in the world in his position. And you cannot calculate the '\n", - " 'consequences of an injury that rules someone like Rodri out for the season. '\n", - " 'City are a team, like many modern ones, in which the holding midfielder is a '\n", - " 'key element to the construction. So, when you take Rodri out, it is '\n", - " 'difficult to hold it together. There were Plan Bs - John Stones, Manuel '\n", - " 'Akanji, even Nathan Ake - but injuries struck. The big injury list has been '\n", - " 'out of the ordinary and the busy calendar has also played a part in '\n", - " 'compounding the issues. However, one factor even Guardiola cannot explain is '\n", - " 'the big uncharacteristic errors in almost every game from international '\n", - " 'players. Why did Matheus Nunes make that challenge to give away the penalty '\n", - " 'against Manchester United? Jack Grealish is sent on at the end to keep the '\n", - " 'ball and cannot do that. There are errors from Walker and other defenders. '\n", - " \"These are some of the best players in the world. Of course the players\\\\' \"\n", - " 'mindset is important, and confidence is diminishing. Wrong decisions get '\n", - " 'taken so there is almost panic on the pitch instead of calm. There are also '\n", - " 'players badly out of form who are having to play because of injuries. Walker '\n", - " \"is now unable to hide behind his pace, I\\\\'m not sure Kevin de Bruyne is \"\n", - " 'ever getting back to the level he used to be at, Bernardo Silva and Ilkay '\n", - " 'Gundogan do not have time to rest, Grealish is not playing at his best. Some '\n", - " 'of these players were only meant to be playing one game a week but, because '\n", - " 'of injuries, have played 12 games in 40 days. It all has a domino effect. '\n", - " \"One consequence is that Erling Haaland isn\\\\'t getting the service to score. \"\n", - " \"But the Norwegian still remains City\\\\'s top-scorer with 13. Defender Josko \"\n", - " 'Gvardiol is next on the list with just four. The way their form has been '\n", - " 'analysed inside the City camp is there have only been three games where they '\n", - " 'deserved to lose (Liverpool, Bournemouth and Aston Villa). But of course it '\n", - " 'is time to change the dynamic.\\\\n\\\\nGuardiola has never protected his '\n", - " 'players so much. He has not criticised them and is not going to do so. They '\n", - " 'have won everything with him. Instead of doing more with them, he has tried '\n", - " 'doing less. He has sometimes given them more days off to clear their heads, '\n", - " 'so they can reset - two days this week for instance. Perhaps the time to '\n", - " 'change a team is when you are winning, but no-one was suggesting Man City '\n", - " 'were about to collapse when they were top and unbeaten after nine league '\n", - " 'games. Some people have asked how bad it has to get before City make a '\n", - " 'decision on Guardiola. The answer is that there is no decision to be made. '\n", - " 'Maybe if this was Real Madrid, Barcelona or Juventus, the pressure from '\n", - " 'outside would be massive and the argument would be made that Guardiola has '\n", - " 'to go. At City he has won the lot, so how can anyone say he is failing? Yes, '\n", - " \"this is a crisis. But given all their problems, City\\\\'s renewed target is \"\n", - " 'finishing in the top four. That is what is in all their heads now. The idea '\n", - " 'is to recover their essence by improving defensive concepts that are not '\n", - " 'there and re-establishing the intensity they are known for. Guardiola is '\n", - " 'planning to use the next two years of his contract, which is expected to be '\n", - " 'his last as a club manager, to prepare a new Manchester City. When he was at '\n", - " 'the end of his four years at Barcelona, he asked two managers what to do '\n", - " 'when you feel people are not responding to your instructions. Do you go or '\n", - " 'do the players go? Sir Alex Ferguson and Rafael Benitez both told him that '\n", - " 'the players need to go. Guardiola did not listen because of his emotional '\n", - " 'attachment to his players back then and he decided to leave the Camp Nou '\n", - " 'because he felt the cycle was over. He will still protect his players now '\n", - " 'but there is not the same emotional attachment - so it is the players who '\n", - " 'are going to leave this time. It is likely City will look to replace five or '\n", - " 'six regular starters. Guardiola knows it is the end of an era and the start '\n", - " 'of a new one. Changes will not be immediate and the majority of the work '\n", - " 'will be done in the summer. But they are open to any opportunities in '\n", - " 'January - and a holding midfielder is one thing they need. In the summer '\n", - " \"City might want to get Spain\\\\'s Martin Zubimendi from Real Sociedad and \"\n", - " 'they know 60m euros (£50m) will get him. He said no to Liverpool last summer '\n", - " 'even though everything was agreed, but he now wants to move on and the '\n", - " 'Premier League is the target. Even if they do not get Zubimendi, that is the '\n", - " 'calibre of footballer they are after. A new Manchester City is on its way - '\n", - " 'with changes driven by Guardiola, incoming sporting director Hugo Viana and '\n", - " 'the football department.\\\\n\\\\n# Documents:\\\\nManchester City boss Pep '\n", - " 'Guardiola has won 18 trophies since he arrived at the club in '\n", - " '2016\\\\n\\\\nManchester City boss Pep Guardiola says he is \"fine\" despite '\n", - " 'admitting his sleep and diet are being affected by the worst run of results '\n", - " 'in his entire managerial career. In an interview with former Italy '\n", - " \"international Luca Toni for Amazon Prime Sport before Wednesday\\\\'s \"\n", - " 'Champions League defeat by Juventus, Guardiola touched on the personal '\n", - " \"impact City\\\\'s sudden downturn in form has had. Guardiola said his state of \"\n", - " 'mind was \"ugly\", that his sleep was \"worse\" and he was eating lighter as his '\n", - " \"digestion had suffered. City go into Sunday\\\\'s derby against Manchester \"\n", - " 'United at Etihad Stadium having won just one of their past 10 games. The '\n", - " 'Juventus loss means there is a chance they may not even secure a play-off '\n", - " 'spot in the Champions League. Asked to elaborate on his comments to Toni, '\n", - " 'Guardiola said: \"I\\\\\\'m fine. \"In our jobs we always want to do our best or '\n", - " \"the best as possible. When that doesn\\\\'t happen you are more uncomfortable \"\n", - " 'than when the situation is going well, always that happened. \"In good '\n", - " 'moments I am happier but when I get to the next game I am still concerned '\n", - " 'about what I have to do. There is no human being that makes an activity and '\n", - " 'it doesn\\\\\\'t matter how they do.\" Guardiola said City have to defend better '\n", - " 'and \"avoid making mistakes at both ends\". To emphasise his point, Guardiola '\n", - " \"referred back to the third game of City\\\\'s current run, against a Sporting \"\n", - " 'side managed by Ruben Amorim, who will be in the United dugout at the '\n", - " 'weekend. City dominated the first half in Lisbon, led thanks to Phil '\n", - " \"Foden\\\\'s early effort and looked to be cruising. Instead, they conceded \"\n", - " 'three times in 11 minutes either side of half-time as Sporting eventually '\n", - " 'ran out 4-1 winners. \"I would like to play the game like we played in Lisbon '\n", - " 'on Sunday, believe me,\" said Guardiola, who is facing the prospect of only '\n", - " 'having three fit defenders for the derby as Nathan Ake and Manuel Akanji try '\n", - " 'to overcome injury concerns. If there is solace for City, it comes from the '\n", - " 'knowledge United are not exactly flying. Their comeback Europa League '\n", - " \"victory against Viktoria Plzen on Thursday was their third win of Amorim\\\\'s \"\n", - " 'short reign so far but only one of those successes has come in the Premier '\n", - " 'League, where United have lost their past two games against Arsenal and '\n", - " 'Nottingham Forest. Nevertheless, Guardiola can see improvements already on '\n", - " 'the red side of the city. \"It\\\\\\'s already there,\" he said. \"You see all the '\n", - " 'patterns, the movements, the runners and the pace. He will do a good job at '\n", - " 'United, I\\\\\\'m pretty sure of that.\"\\\\n\\\\nGuardiola says skipper Kyle Walker '\n", - " 'has been offered support by the club after the City defender highlighted the '\n", - " 'racial abuse he had received on social media in the wake of the Juventus '\n", - " 'trip. \"It\\\\\\'s unacceptable,\" he said. \"Not because it\\\\\\'s Kyle - for any '\n", - " 'human being. \"Unfortunately it happens many times in the real world. It is '\n", - " 'not necessary to say he has the support of the entire club. It is completely '\n", - " 'unacceptable and we give our support to him.\"\\\\n\\\\n# Documents:\\\\n\\\\\\'I am '\n", - " \"not good enough\\\\' - Guardiola faces daunting and major rebuild\\\\n\\\\nThis \"\n", - " 'video can not be played To play this video you need to enable JavaScript in '\n", - " \"your browser. \\\\'I am not good enough\\\\' - Guardiola says he must find a \"\n", - " \"\\\\'solution\\\\' after derby loss\\\\n\\\\nPep Guardiola says his sleep has \"\n", - " \"suffered during Manchester City\\\\'s deepening crisis, so he will not be \"\n", - " 'helped by a nightmarish conclusion to one of the most stunning defeats of '\n", - " 'his long reign. Guardiola looked agitated, animated and on edge even after '\n", - " \"City led the Manchester derby through Josko Gvardiol\\\\'s 36th-minute header, \"\n", - " 'his reaction to the goal one of almost disdain that it came via a deflected '\n", - " 'cross as opposed to in his purist style. He sat alone with his eyes closed '\n", - " 'sipping from a water bottle before the resumption of the second half, then '\n", - " 'was denied even the respite of victory when Manchester United gave this '\n", - " 'largely dismal derby a dramatic conclusion it barely deserved with a '\n", - " 'remarkable late comeback. First, with 88 minutes on the clock, Matheus Nunes '\n", - " 'presented Amad Diallo with the ball before compounding his error by '\n", - " 'flattening the forward as he made an attempt to recover his mistake. Bruno '\n", - " 'Fernandes completed the formalities from the penalty spot. Worse was to come '\n", - " \"two minutes later when Lisandro Martinez\\\\'s routine long ball caught \"\n", - " \"City\\\\'s defence inexplicably statuesque. Goalkeeper Ederson\\\\'s positioning \"\n", - " 'was awry, allowing the lively Diallo to pounce from an acute angle to leave '\n", - " 'Guardiola and his players stunned. It was the latest into any game, 88 '\n", - " 'minutes, that reigning Premier League champions had led then lost. It was '\n", - " 'also the first time City had lost a game they were leading so late on. And '\n", - " \"in a sign of City\\\\'s previous excellence that is now being challenged, they \"\n", - " 'have only lost four of 105 Premier League home games under Guardiola in '\n", - " 'which they have been ahead at half-time, winning 94 and drawing seven. '\n", - " 'Guardiola delivered a brutal self-analysis as he told Match of the Day: \"I '\n", - " 'am not good enough. I am the boss. I am the manager. I have to find '\n", + " 'Documents:\\\\nManchester City boss Pep Guardiola has won 18 trophies since he '\n", + " 'arrived at the club in 2016\\\\n\\\\nManchester City boss Pep Guardiola says he '\n", + " 'is \"fine\" despite admitting his sleep and diet are being affected by the '\n", + " 'worst run of results in his entire managerial career. In an interview with '\n", + " 'former Italy international Luca Toni for Amazon Prime Sport before '\n", + " \"Wednesday\\\\'s Champions League defeat by Juventus, Guardiola touched on the \"\n", + " \"personal impact City\\\\'s sudden downturn in form has had. Guardiola said his \"\n", + " 'state of mind was \"ugly\", that his sleep was \"worse\" and he was eating '\n", + " \"lighter as his digestion had suffered. City go into Sunday\\\\'s derby against \"\n", + " 'Manchester United at Etihad Stadium having won just one of their past 10 '\n", + " 'games. The Juventus loss means there is a chance they may not even secure a '\n", + " 'play-off spot in the Champions League. Asked to elaborate on his comments to '\n", + " 'Toni, Guardiola said: \"I\\\\\\'m fine. \"In our jobs we always want to do our '\n", + " \"best or the best as possible. When that doesn\\\\'t happen you are more \"\n", + " 'uncomfortable than when the situation is going well, always that happened. '\n", + " '\"In good moments I am happier but when I get to the next game I am still '\n", + " 'concerned about what I have to do. There is no human being that makes an '\n", + " 'activity and it doesn\\\\\\'t matter how they do.\" Guardiola said City have to '\n", + " 'defend better and \"avoid making mistakes at both ends\". To emphasise his '\n", + " \"point, Guardiola referred back to the third game of City\\\\'s current run, \"\n", + " 'against a Sporting side managed by Ruben Amorim, who will be in the United '\n", + " 'dugout at the weekend. City dominated the first half in Lisbon, led thanks '\n", + " \"to Phil Foden\\\\'s early effort and looked to be cruising. Instead, they \"\n", + " 'conceded three times in 11 minutes either side of half-time as Sporting '\n", + " 'eventually ran out 4-1 winners. \"I would like to play the game like we '\n", + " 'played in Lisbon on Sunday, believe me,\" said Guardiola, who is facing the '\n", + " 'prospect of only having three fit defenders for the derby as Nathan Ake and '\n", + " 'Manuel Akanji try to overcome injury concerns. If there is solace for City, '\n", + " 'it comes from the knowledge United are not exactly flying. Their comeback '\n", + " 'Europa League victory against Viktoria Plzen on Thursday was their third win '\n", + " \"of Amorim\\\\'s short reign so far but only one of those successes has come in \"\n", + " 'the Premier League, where United have lost their past two games against '\n", + " 'Arsenal and Nottingham Forest. Nevertheless, Guardiola can see improvements '\n", + " 'already on the red side of the city. \"It\\\\\\'s already there,\" he said. \"You '\n", + " 'see all the patterns, the movements, the runners and the pace. He will do a '\n", + " 'good job at United, I\\\\\\'m pretty sure of that.\"\\\\n\\\\nGuardiola says skipper '\n", + " 'Kyle Walker has been offered support by the club after the City defender '\n", + " 'highlighted the racial abuse he had received on social media in the wake of '\n", + " 'the Juventus trip. \"It\\\\\\'s unacceptable,\" he said. \"Not because it\\\\\\'s '\n", + " 'Kyle - for any human being. \"Unfortunately it happens many times in the real '\n", + " 'world. It is not necessary to say he has the support of the entire club. It '\n", + " 'is completely unacceptable and we give our support to him.\"\\\\n\\\\n# '\n", + " \"Documents:\\\\n\\\\'I am not good enough\\\\' - Guardiola faces daunting and major \"\n", + " 'rebuild\\\\n\\\\nThis video can not be played To play this video you need to '\n", + " \"enable JavaScript in your browser. \\\\'I am not good enough\\\\' - Guardiola \"\n", + " \"says he must find a \\\\'solution\\\\' after derby loss\\\\n\\\\nPep Guardiola says \"\n", + " \"his sleep has suffered during Manchester City\\\\'s deepening crisis, so he \"\n", + " 'will not be helped by a nightmarish conclusion to one of the most stunning '\n", + " 'defeats of his long reign. Guardiola looked agitated, animated and on edge '\n", + " \"even after City led the Manchester derby through Josko Gvardiol\\\\'s \"\n", + " '36th-minute header, his reaction to the goal one of almost disdain that it '\n", + " 'came via a deflected cross as opposed to in his purist style. He sat alone '\n", + " 'with his eyes closed sipping from a water bottle before the resumption of '\n", + " 'the second half, then was denied even the respite of victory when Manchester '\n", + " 'United gave this largely dismal derby a dramatic conclusion it barely '\n", + " 'deserved with a remarkable late comeback. First, with 88 minutes on the '\n", + " 'clock, Matheus Nunes presented Amad Diallo with the ball before compounding '\n", + " 'his error by flattening the forward as he made an attempt to recover his '\n", + " 'mistake. Bruno Fernandes completed the formalities from the penalty spot. '\n", + " \"Worse was to come two minutes later when Lisandro Martinez\\\\'s routine long \"\n", + " \"ball caught City\\\\'s defence inexplicably statuesque. Goalkeeper Ederson\\\\'s \"\n", + " 'positioning was awry, allowing the lively Diallo to pounce from an acute '\n", + " 'angle to leave Guardiola and his players stunned. It was the latest into any '\n", + " 'game, 88 minutes, that reigning Premier League champions had led then lost. '\n", + " 'It was also the first time City had lost a game they were leading so late '\n", + " \"on. And in a sign of City\\\\'s previous excellence that is now being \"\n", + " 'challenged, they have only lost four of 105 Premier League home games under '\n", + " 'Guardiola in which they have been ahead at half-time, winning 94 and drawing '\n", + " 'seven. Guardiola delivered a brutal self-analysis as he told Match of the '\n", + " 'Day: \"I am not good enough. I am the boss. I am the manager. I have to find '\n", " 'solutions and so far I haven\\\\\\'t. That\\\\\\'s the reality. \"Not much else to '\n", " 'say. No defence. Manchester United were incredibly persistent. We have not '\n", " 'lost eight games in two seasons. We can\\\\\\'t defend that.\"\\\\n\\\\nManchester '\n", @@ -1541,72 +1878,171 @@ " 'alien territory of failure. He will be under no illusions about the size of '\n", " 'the job in front of him as he placed his head in his hands in anguish after '\n", " 'yet another damaging and deeply revealing defeat. City and Guardiola are in '\n", - " 'new, unforgiving territory.\\\\n\\\\n# Documents:\\\\nMan City might miss out on '\n", - " 'Champions League - Guardiola\\\\n\\\\nErling Haaland was part of the Manchester '\n", - " 'City side that won the Champions League for the first time in '\n", - " '2023\\\\n\\\\nManchester City boss Pep Guardiola says the club are in danger of '\n", - " \"missing out on a place in next season\\\\'s Champions League. City are \"\n", - " \"currently in their 14th consecutive season in European football\\\\'s most \"\n", - " 'prestigious club competition. Only Arsenal between 1998 and 2017, and '\n", - " 'Manchester United between 1996 and 2014, have a longer record of qualifying '\n", - " 'among English clubs. City are seventh in the Premier League after 17 '\n", - " 'matches, four points behind Nottingham Forest in fourth and a point behind '\n", - " \"fifth-placed Bournemouth. England are currently top of Uefa\\\\'s European \"\n", - " 'Performance Spot table and well placed to secure a fifth place in next '\n", - " \"season\\\\'s Champions League, although City would still not qualify on \"\n", - " 'current standings. \"When I said before, people laughed,\" said Guardiola. '\n", - " '\"They said, \\\\\\'qualifying for the Champions League is not a big '\n", - " 'success\\\\\\'. \"But I know it because it happens with clubs in this country. '\n", - " 'They were dominant for many years and after they were many years not '\n", - " 'qualifying for the Champions League.\"\\\\n\\\\nGuardiola\\\\\\'s side host Everton '\n", - " 'on Boxing Day, before a trip to Leicester on 29 December and a home match '\n", - " 'against West Ham on 4 January. Given all three opponents are in the bottom '\n", - " 'seven, it offers City a chance to improve on an appalling recent record of '\n", - " 'four points from eight games, which Guardiola acknowledges has left their '\n", - " 'lofty European ambitions in doubt. \"The one team that has been in the '\n", - " 'Champions League for the past years has been Manchester City,\" he added. '\n", - " '\"Now we are at risk, of course we are. Definitely.\" Arsenal, Chelsea, '\n", - " \"Liverpool and Manchester United finished in the Premier League\\\\'s top four \"\n", - " 'from the 2005-06 season to 2008-09. At least three of them also occupied the '\n", - " 'top four spots for 15 successive campaigns until 2012. But United have spent '\n", - " 'five out of the past 11 seasons outside the Champions League. Arsenal spent '\n", - " 'six seasons out of the competition before returning last term. Liverpool '\n", - " 'missed out all but one year in seven from 2010, while Chelsea are in their '\n", - " \"second successive campaign outside Europe\\\\'s elite. This term the threat to \"\n", - " 'City comes from unexpected sources. As well as Forest and Bournemouth, Aston '\n", - " 'Villa are ahead of City, while Newcastle, Fulham and Brighton are also '\n", - " 'within a couple of points. \"There are a lot of contenders,\" said Guardiola, '\n", - " 'whose side have lost nine of their last 12 games in all competitions. \"For '\n", - " 'every club it is so important and if we are not winning games, we will be '\n", - " 'out. \"If we don\\\\\\'t qualify it is because we don\\\\\\'t deserve it, because '\n", - " \"we were not prepared and because we had a lot of problems and didn\\\\'t solve \"\n", - " 'them.\"\\', tool_call_id=\\'call_kUqRR8fk7bPUzWxzpkGzt2AP\\', '\n", - " 'timestamp=datetime.datetime(2025, 9, 23, 6, 17, 29, 776114, '\n", + " 'new, unforgiving territory.\\\\n\\\\n# Documents:\\\\nPep Guardiola has said '\n", + " 'Manchester City will be his final managerial job in club football before he '\n", + " '\"maybe\" coaches a national team.\\\\n\\\\nThe former Barcelona and Bayern Munich '\n", + " 'boss has won 15 major trophies since taking charge of City in 2016.\\\\n\\\\nThe '\n", + " '53-year-old Spaniard was approached in the summer about the possibility of '\n", + " 'becoming England manager, but last month signed a two-year contract '\n", + " 'extension with City until 2027.\\\\n\\\\nSpeaking to celebrity chef Dani Garcia '\n", + " 'on YouTube, Guardiola did not indicate when he intends to step down at City '\n", + " 'but said he would not return to club football - in the Premier League or '\n", + " 'overseas.\\\\n\\\\n\"I\\\\\\'m not going to manage another team,\" he '\n", + " 'said.\\\\n\\\\n\"I\\\\\\'m not talking about the long-term future, but what I\\\\\\'m '\n", + " 'not going to do is leave Manchester City, go to another country, and do the '\n", + " 'same thing as now.\\\\n\\\\n\"I wouldn\\\\\\'t have the energy. The thought of '\n", + " 'starting somewhere else, all the process of training and so on. No, no, no. '\n", + " 'Maybe a national team, but that\\\\\\'s different.\\\\n\\\\n\"I want to leave it and '\n", + " \"go and play golf, but I can\\\\'t [if he takes a club job]. I think stopping \"\n", + " 'would do me good.\"\\\\n\\\\nCity have won just once since Guardiola extended his '\n", + " 'contract - and once in nine games since beating Southampton on 26 '\n", + " 'October.\\\\n\\\\nThat victory came at home to Nottingham Forest last Wednesday, '\n", + " 'but was followed by a 2-2 draw at Crystal Palace at the weekend.\\\\n\\\\nThe '\n", + " 'Blues visit Juventus next in the Champions League on Wednesday (20:00 GMT), '\n", + " 'before hosting Manchester United in the Premier League on Sunday '\n", + " '(16:30).\\\\n\\\\n\"Right now we are not in the position - when we have had the '\n", + " 'results of the last seven, eight games - to talk about winning games in '\n", + " 'plural,\" said Guardiola at his pre-match news conference.\\\\n\\\\n\"We have to '\n", + " 'win the game and not look at what happens in the next one yet.\"\\\\n\\\\n# '\n", + " \"Documents:\\\\n\\\\'We have to find a way\\\\' - Guardiola vows to end relegation \"\n", + " 'form\\\\n\\\\nThis video can not be played To play this video you need to enable '\n", + " \"JavaScript in your browser. \\\\'Worrying\\\\' and \\\\'staggering\\\\' - Why do \"\n", + " 'Manchester City keep conceding?\\\\n\\\\nManchester City are currently in '\n", + " \"relegation form and there is little sign of it ending. Saturday\\\\'s 2-1 \"\n", + " 'defeat at Aston Villa left them joint bottom of the form table over the past '\n", + " 'eight games with just Southampton for company. Saints, at the foot of the '\n", + " 'Premier League, have the same number of points, four, as City over their '\n", + " 'past eight matches having won one, drawn one and lost six - the same record '\n", + " 'as the floundering champions. And if Southampton - who appointed Ivan Juric '\n", + " 'as their new manager on Saturday - get at least a point at Fulham on Sunday, '\n", + " 'City will be on the worst run in the division. Even Wolves, who sacked boss '\n", + " \"Gary O\\\\'Neil last Sunday and replaced him with Vitor Pereira, have earned \"\n", + " 'double the number of points during the same period having played a game '\n", + " 'fewer. They are damning statistics for Pep Guardiola, even if he does have '\n", + " 'some mitigating circumstances with injuries to Ederson, Nathan Ake and Ruben '\n", + " 'Dias - who all missed the loss at Villa Park - and the long-term loss of '\n", + " \"midfield powerhouse Rodri. Guardiola was happy with Saturday\\\\'s \"\n", + " 'performance, despite defeat in Birmingham, but there is little solace to '\n", + " 'take at slipping further out of the title race. He may have needed to field '\n", + " 'a half-fit Manuel Akanji and John Stones at Villa Park but that does not '\n", + " 'account for City looking a shadow of their former selves. That does not '\n", + " 'justify the error Josko Gvardiol made to gift Jhon Duran a golden chance '\n", + " 'inside the first 20 seconds, or £100m man Jack Grealish again failing to '\n", + " \"have an impact on a game. There may be legitimate reasons for City\\\\'s drop \"\n", + " 'off, whether that be injuries, mental fatigue or just simply a team coming '\n", + " 'to the end of its lifecycle, but their form, which has plunged off a cliff '\n", + " 'edge, would have been unthinkable as they strolled to a fourth straight '\n", + " 'title last season. \"The worrying thing is the number of goals conceded,\" '\n", + " 'said ex-England captain Alan Shearer on BBC Match of the Day. \"The number of '\n", + " 'times they were opened up because of the lack of protection and legs in '\n", + " 'midfield was staggering. There are so many things that are wrong at this '\n", + " 'moment in time.\"\\\\n\\\\nThis video can not be played To play this video you '\n", + " \"need to enable JavaScript in your browser. Man City \\\\'have to find a way\\\\' \"\n", + " 'to return to form - Guardiola\\\\n\\\\nAfterwards Guardiola was calm, so much so '\n", + " 'it was difficult to hear him in the news conference, a contrast to the '\n", + " 'frustrated figure he cut on the touchline. He said: \"It depends on us. The '\n", + " 'solution is bring the players back. We have just one central defender fit, '\n", + " 'that is difficult. We are going to try next game - another opportunity and '\n", + " 'we don\\\\\\'t think much further than that. \"Of course there are more reasons. '\n", + " \"We concede the goals we don\\\\'t concede in the past, we [don\\\\'t] score the \"\n", + " 'goals we score in the past. Football is not just one reason. There are a lot '\n", + " 'of little factors. \"Last season we won the Premier League, but we came here '\n", + " 'and lost. We have to think positive and I have incredible trust in the guys. '\n", + " 'Some of them have incredible pride and desire to do it. We have to find a '\n", + " 'way, step by step, sooner or later to find a way back.\" Villa boss Unai '\n", + " \"Emery highlighted City\\\\'s frailties, saying he felt Villa could seize on \"\n", + " 'the visitors\\\\\\' lack of belief. \"Manchester City are a little bit under the '\n", + " 'confidence they have normally,\" he said. \"The second half was different, we '\n", + " 'dominated and we scored. Through those circumstances they were feeling worse '\n", + " 'than even in the first half.\"\\\\n\\\\nErling Haaland had one touch in the Villa '\n", + " 'box\\\\n\\\\nThere are chinks in the armour never seen before at City under '\n", + " 'Guardiola and Erling Haaland conceded belief within the squad is low. He '\n", + " 'told TNT after the game: \"Of course, [confidence levels are] not the best. '\n", + " 'We know how important confidence is and you can see that it affects every '\n", + " 'human being. That is how it is, we have to continue and stay positive even '\n", + " 'though it is difficult.\" Haaland, with 76 goals in 83 Premier League '\n", + " 'appearances since joining City from Borussia Dortmund in 2022, had one shot '\n", + " 'and one touch in the Villa box. His 18 touches in the whole game were the '\n", + " 'lowest of all starting players and he has been self critical, despite '\n", + " \"scoring 13 goals in the top flight this season. Over City\\\\'s last eight \"\n", + " 'games he has netted just twice though, but Guardiola refused to criticise '\n", + " 'his star striker. He said: \"Without him we will be even worse but I like the '\n", + " \"players feeling that way. I don\\\\'t agree with Erling. He needs to have the \"\n", + " 'balls delivered in the right spots but he will fight for the next '\n", + " 'one.\"\\\\n\\\\n# Documents:\\\\nMan City\\\\\\'s Dias ruled out for \\\\\\'three or four '\n", + " \"weeks\\\\'\\\\n\\\\nRuben Dias has won 10 major trophies during his time at \"\n", + " 'Manchester City\\\\n\\\\nManchester City have suffered a fresh injury blow with '\n", + " 'manager Pep Guardiola confirming Portugal central defender Ruben Dias has '\n", + " 'been ruled out for \"three or four weeks\" with a muscle injury. Dias, who '\n", + " \"suffered the injury in Saturday\\\\'s 2-1 defeat by Manchester United, will \"\n", + " 'miss the entire festive programme and potentially the FA Cup third-round tie '\n", + " 'with Salford on 11 January. The 27-year-old also faces a battle to be fit '\n", + " \"for City\\\\'s crucial Champions League trip to Paris St-Germain on 22 \"\n", + " 'January. Dias has already missed seven games with a calf injury this season, '\n", + " 'adding to a defensive injury list that has seen John Stones, Nathan Ake, '\n", + " 'Manuel Akanji and Kyle Walker all ruled out at various points, while Ballon '\n", + " \"d\\\\'Or winner Rodri will miss the remainder of the domestic season after \"\n", + " 'suffering a cruciate knee ligament injury. \"It\\\\\\'s a muscular problem and '\n", + " 'he will be out for three to four weeks,\" said Guardiola. \"After 75 minutes '\n", + " \"against United he felt something. But he\\\\'s so strong and wanted to stay on \"\n", + " 'the pitch. Now he\\\\\\'s injured.\" Guardiola confirmed Stones, Akanji and '\n", + " 'midfielder Mateo Kovacic have all trained this week and could feature at '\n", + " 'Aston Villa on Saturday (12:30 GMT), but said goalkeeper Ederson was \"a '\n", + " 'doubt\" with an unspecified problem. \"Ederson has been struggling with some '\n", + " 'niggles in his leg, he doesn\\\\\\'t feel completely fine,\" said Guardiola. '\n", + " '\"Ederson is so important for us.\" Amid City\\\\\\'s current run of one win in '\n", + " \"11 games, surprise has been expressed about Guardiola\\\\'s use of youngsters \"\n", + " \"James McAtee and Nico O\\\\'Reilly. City made a point of keeping both players \"\n", + " 'despite numerous loan options. Yet McAtee has made just two substitute '\n", + " 'appearances - coming on in the last minute on both occasions - while '\n", + " \"O\\\\'Reilly is yet to make his league debut. But it seems they will stay at \"\n", + " 'the club for the second half of the season, with Guardiola replying \"I '\n", + " 'don\\\\\\'t think so\" when asked if players might leave during the January '\n", + " 'transfer window. The Spaniard said he is \"not a big fan\" of buying players '\n", + " 'in January but it is \"possible\" City will look to sign someone because \"the '\n", + " 'circumstances of this season have been special\".\\\\n\\\\nGuardiola\\\\\\'s mood '\n", + " 'was so downbeat in the immediate aftermath of the United defeat it was easy '\n", + " 'to imagine he might conclude he was no longer capable of doing the job. He '\n", + " 'gave his players a couple of days off afterwards and was brighter when he '\n", + " 'spoke to journalists in his scheduled briefing before the Villa trip. '\n", + " '\"We\\\\\\'d just finished a game that we lost in the circumstances and I was '\n", + " 'not happy,\" he said. \"I try to be honest about the feelings of my teams. We '\n", + " 'fell down six times [number of Premier League games without a win], we have '\n", + " 'to stand up seven. There is no alternative. \"I\\\\\\'m fine. I\\\\\\'m a normal '\n", + " 'person with feelings like all of us. When the situation is going well we are '\n", + " \"better but it\\\\'s normal. I would not go to the press conference if we were \"\n", + " '1-0 up and expressing something that I didn\\\\\\'t feel.\" Former Villa forward '\n", + " 'Jack Grealish has not scored for City in over a year but Guardiola pointed '\n", + " 'out he is not the only attacking player struggling this season. He added: '\n", + " '\"We are struggling to create a little bit up front, but always I am '\n", + " 'optimistic about my players that they are going to turn and perform '\n", + " 'well.\"\\', tool_call_id=\\'call_kLchsPmUPdNovuMcE2StuBxU\\', '\n", + " 'timestamp=datetime.datetime(2025, 11, 6, 7, 41, 35, 220049, '\n", " \"tzinfo=datetime.timezone.utc), part_kind='tool-return')], instructions=None, \"\n", " \"kind='request')\")\n", "==================================================\n", "Step 4:\n", - "('ModelResponse(parts=[TextPart(content=\"Pep Guardiola has expressed '\n", - " \"significant concerns about Manchester City's current form, which he \"\n", - " 'describes as the worst run of results in his managerial career. Despite '\n", - " 'being satisfied with his role at City, Guardiola is personally affected by '\n", - " \"the team's performance, admitting issues with sleep and diet due to stress. \"\n", - " 'City have lost nine of their past 12 matches, leading to a decline in their '\n", - " 'Premier League standings. Guardiola acknowledges the difficulty of solving '\n", - " 'current challenges, particularly in defending and avoiding mistakes at '\n", - " 'crucial moments.\\\\n\\\\nThe absence of key players like midfielder Rodri has '\n", - " 'been a critical issue, likened to removing a crucial piece in a game of '\n", - " \"Jenga, leading to the collapse of the team's structure. Injuries have \"\n", - " 'compounded the challenge, with errors from usually dependable international '\n", - " 'players adding to the crisis. Guardiola has opted to protect and support his '\n", - " 'players rather than criticize, allowing them time to reset with more days '\n", - " 'off.\\\\n\\\\nGuardiola sees the current tumult as the end of an era and is '\n", - " 'planning changes that potentially involve a substantial squad overhaul in '\n", - " 'the upcoming transfer windows. Despite these difficulties, Guardiola remains '\n", - " 'committed and is using this period to prepare for a new phase at Manchester '\n", - " 'City under his management.\", part_kind=\\'text\\')], '\n", - " \"model_name='gpt-4o-2024-08-06', timestamp=datetime.datetime(2025, 9, 23, 6, \"\n", - " \"17, 30, tzinfo=datetime.timezone.utc), kind='response')\")\n", + "(\"ModelResponse(parts=[TextPart(content='Manchester City manager Pep Guardiola \"\n", + " \"has been candid about the team\\\\'s current form, admitting to personal \"\n", + " 'impacts such as affected sleep and diet due to stress over what he describes '\n", + " 'as the \"worst run of results\" in his managerial career. He expressed that '\n", + " 'his state of mind was \"ugly\" and acknowledged that the team\\\\\\'s downturn '\n", + " \"has made him uncomfortable. Guardiola emphasized the team\\\\'s need to \"\n", + " 'improve defensively and reduce mistakes.\\\\n\\\\nIn addition, he openly '\n", + " 'analyzed his own role in the situation, stating, \"I am not good enough. I am '\n", + " 'the boss. I am the manager. I have to find solutions and so far I '\n", + " 'haven\\\\\\'t.\" Despite this brutal self-criticism, Guardiola remains committed '\n", + " \"to finding a way to turn the team\\\\'s fortunes around.\\\\n\\\\nRecently, \"\n", + " 'Guardiola noted the need for serious rebuilding, while acknowledging the '\n", + " 'challenges of an aging squad and injuries to key players. However, he '\n", + " 'continues to express optimism and trust in his players, believing in their '\n", + " 'desire and pride to overcome current difficulties.\\\\n\\\\nIn summary, '\n", + " \"Guardiola\\\\'s reaction to Manchester City\\\\'s current form reflects a mix of \"\n", + " 'personal distress, professional determination to find solutions, and a '\n", + " \"realistic acknowledgment of the challenges the team faces.', \"\n", + " \"part_kind='text')], model_name='gpt-4o-2024-08-06', \"\n", + " 'timestamp=datetime.datetime(2025, 11, 6, 7, 41, 36, '\n", + " \"tzinfo=datetime.timezone.utc), kind='response')\")\n", "==================================================\n" ] } From 6746ade86a00302ed370f8d6f17b78f51e96e53f Mon Sep 17 00:00:00 2001 From: AayushTyagi1 Date: Fri, 7 Nov 2025 13:19:13 +0530 Subject: [PATCH 8/9] Final Review Pydantic AI --- .../RAG_with_Couchbase_and_PydanticAI.ipynb | 1134 ++++++----------- 1 file changed, 360 insertions(+), 774 deletions(-) diff --git a/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb b/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb index 2d81056a..4bc2fcb0 100644 --- a/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb +++ b/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb @@ -38,7 +38,7 @@ "\n", "To learn more, please follow the [instructions](https://docs.couchbase.com/cloud/get-started/create-account.html).\n", "\n", - "Note: To run this this tutorial, you will need Capella with Couchbase Server version 8.0 or above as GSI vector search is supported only from version 8.0\n", + "Note: To run this tutorial, you will need Capella with Couchbase Server version 8.0 or above as GSI vector search is supported only from version 8.0\n", "\n", "### Couchbase Capella Configuration\n", "\n", @@ -188,7 +188,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-11-06 13:07:15,968 - INFO - Successfully connected to Couchbase\n" + "2025-11-07 13:08:54,718 - INFO - Successfully connected to Couchbase\n" ] } ], @@ -233,7 +233,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 16, "id": "d9494a6f", "metadata": {}, "outputs": [ @@ -241,21 +241,21 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-11-06 13:07:20,020 - INFO - Bucket 'travel-sample' exists.\n", - "2025-11-06 13:07:20,024 - INFO - Scope 'shared' does not exist. Creating it...\n", - "2025-11-06 13:07:20,069 - INFO - Scope 'shared' created successfully.\n", - "2025-11-06 13:07:20,078 - INFO - Collection 'demo' does not exist. Creating it...\n", - "2025-11-06 13:07:20,124 - INFO - Collection 'demo' created successfully.\n", - "2025-11-06 13:07:22,198 - INFO - All documents cleared from the collection.\n" + "2025-11-07 13:13:43,503 - INFO - Bucket 'travel-sample' exists.\n", + "2025-11-07 13:13:43,505 - INFO - Scope 'shared' does not exist. Creating it...\n", + "2025-11-07 13:13:43,540 - INFO - Scope 'shared' created successfully.\n", + "2025-11-07 13:13:43,543 - INFO - Collection 'demo' does not exist. Creating it...\n", + "2025-11-07 13:13:43,594 - INFO - Collection 'demo' created successfully.\n", + "2025-11-07 13:13:45,644 - INFO - All documents cleared from the collection.\n" ] }, { "data": { "text/plain": [ - "" + "" ] }, - "execution_count": 6, + "execution_count": 16, "metadata": {}, "output_type": "execute_result" } @@ -344,7 +344,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-11-06 13:07:27,643 - INFO - Successfully created OpenAIEmbeddings\n" + "2025-11-07 13:09:11,598 - INFO - Successfully created OpenAIEmbeddings\n" ] } ], @@ -410,7 +410,7 @@ "3. **Scalability** to handle growing vector collections\n", "4. **Concurrent operations** for multi-user environments\n", "\n", - "The BHIVE index will provide optimal performance for our Hugging Face embedding-based semantic search implementation.\n", + "The BHIVE index will provide optimal performance for our OpenAI embedding-based semantic search implementation.\n", "\n", "#### Alternative: Composite Vector Index\n", "\n", @@ -489,7 +489,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 20, "id": "2603a98a", "metadata": {}, "outputs": [ @@ -497,7 +497,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-11-06 13:07:38,469 - INFO - Successfully created vector store\n" + "2025-11-07 13:16:27,214 - INFO - Successfully created vector store\n" ] } ], @@ -529,7 +529,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 17, "id": "2bfb848c", "metadata": {}, "outputs": [ @@ -537,7 +537,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-11-06 13:07:47,468 - INFO - Successfully loaded the BBC News dataset with 2687 rows.\n" + "2025-11-07 13:14:26,898 - INFO - Successfully loaded the BBC News dataset with 2687 rows.\n" ] }, { @@ -572,7 +572,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 18, "id": "81457c3a", "metadata": {}, "outputs": [ @@ -621,7 +621,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 19, "id": "7d85286e", "metadata": {}, "outputs": [ @@ -629,7 +629,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-11-06 13:09:09,453 - INFO - Document ingestion completed successfully.\n" + "2025-11-07 13:15:32,204 - INFO - Document ingestion completed successfully.\n" ] } ], @@ -654,23 +654,67 @@ "id": "0b66ca05", "metadata": {}, "source": [ - "# Perform Semantic Search\n", - "Semantic search in Couchbase involves converting queries and documents into vector representations using an embeddings model. These vectors capture the semantic meaning of the text and are stored directly in Couchbase. When a query is made, Couchbase performs a similarity search by comparing the query vector against the stored document vectors. The similarity metric used for this comparison is configurable, allowing flexibility in how the relevance of documents is determined. Common metrics include cosine similarity, Euclidean distance, or dot product, but other metrics can be implemented based on specific use cases. Different embedding models like BERT, Word2Vec, or GloVe can also be used depending on the application's needs, with the vectors generated by these models stored and searched within Couchbase itself.\n", + "# Understanding Semantic Search in Couchbase\n", + "\n", + "Semantic search goes beyond traditional keyword matching by understanding the meaning and context behind queries. Here's how it works in Couchbase:\n", + "\n", + "## How Semantic Search Works\n", + "\n", + "1. **Vector Embeddings**: Documents and queries are converted into high-dimensional vectors using an embeddings model (in our case, OpenAI's text-embedding-3-small)\n", + "\n", + "2. **Similarity Calculation**: When a query is made, Couchbase compares the query vector against stored document vectors using the COSINE distance metric\n", + "\n", + "3. **Result Ranking**: Documents are ranked by their vector distance (lower distance = more similar meaning)\n", + "\n", + "4. **Flexible Configuration**: Different distance metrics (cosine, euclidean, dot product) and embedding models can be used based on your needs\n", "\n", - "In the provided code, the search process begins by recording the start time, followed by executing the `similarity_search_with_score` method of the `CouchbaseQueryVectorStore`. This method searches Couchbase for the most relevant documents based on the vector similarity to the query. The search results include the document content and the distance that reflects how closely each document aligns with the query in the defined semantic space. The time taken to perform this search is then calculated and logged, and the results are displayed, showing the most relevant documents along with their similarity scores. This approach leverages Couchbase as both a storage and retrieval engine for vector data, enabling efficient and scalable semantic searches. The integration of vector storage and search capabilities within Couchbase allows for sophisticated semantic search operations without relying on external services for vector storage or comparison." + "The `similarity_search_with_score` method performs this entire process, returning documents along with their similarity scores. This enables you to find semantically related content even when exact keywords don't match.\n", + "\n", + "Now let's see semantic search in action and measure its performance with different optimization strategies." + ] + }, + { + "cell_type": "markdown", + "id": "bee91224", + "metadata": {}, + "source": [ + "# Vector Search Performance Optimization\n", + "\n", + "Now let's measure and compare the performance benefits of different optimization strategies. We'll conduct a comprehensive performance analysis across three phases:\n", + "\n", + "## Performance Testing Phases\n", + "\n", + "1. **Phase 1 - Baseline Performance**: Test vector search without GSI indexes to establish baseline metrics\n", + "2. **Phase 2 - GSI-Optimized Search**: Create BHIVE index and measure performance improvements \n", + "3. **Phase 3 - LLM Cache Setup**: Configure caching for the PydanticAI agent to optimize LLM responses\n", + "\n", + "**Important Context:**\n", + "- GSI performance benefits scale with dataset size and concurrent load\n", + "- With our dataset (~1,700 articles), improvements may be modest\n", + "- Production environments with millions of vectors show significant GSI advantages\n", + "- The combination of GSI + LLM caching provides optimal RAG performance\n" ] }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 15, "id": "be49ce6f", "metadata": {}, "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "================================================================================\n", + "PHASE 1: BASELINE PERFORMANCE (NO GSI INDEX)\n", + "================================================================================\n" + ] + }, { "name": "stderr", "output_type": "stream", "text": [ - "2025-11-06 13:09:15,449 - INFO - Semantic search completed in 1.47 seconds\n" + "2025-11-07 13:12:25,470 - INFO - Baseline search completed in 1.09 seconds\n" ] }, { @@ -678,150 +722,88 @@ "output_type": "stream", "text": [ "\n", - "Semantic Search Results (completed in 1.47 seconds):\n", + "Baseline Search Results (completed in 1.0870 seconds):\n", "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.2956, Text: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", + "[Result 1] Vector Distance: 0.2956\n", + "Text: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", "\n", - "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affected by the worst run of results in his entire managerial career. In an interview with former Italy international Luca Toni for Amazon Prime Sport before Wednesday's Champions League defeat by Juventus, Guardiola touched on the personal impact City's sudden downturn in form has had. Guardiola said his state of mind was \"ugly\", that his sleep was \"worse\" and he was eating lighter as his digestion had suffered. City go into Sunday's derby against Manchester United at Etihad Stadium having won just one of their past 10 games. The Juventus loss means there is a chance they may not even secure a play-off spot in the Champions League. Asked to elaborate on his comments to Toni, Guardiola said: \"I'm fine. \"In our jobs we always want to do our best or the best as possible. When that doesn't happen you are more uncomfortable than when the situation is going well, always that happened. \"In good moments I am happier but when I get to the next game I am still concerned about what I have to do. There is no human being that makes an activity and it doesn't matter how they do.\" Guardiola said City have to defend better and \"avoid making mistakes at both ends\". To emphasise his point, Guardiola referred back to the third game of City's current run, against a Sporting side managed by Ruben Amorim, who will be in the United dugout at the weekend. City dominated the first half in Lisbon, led thanks to Phil Foden's early effort and looked to be cruising. Instead, they conceded three times in 11 minutes either side of half-time as Sporting eventually ran out 4-1 winners. \"I would like to play the game like we played in Lisbon on Sunday, believe me,\" said Guardiola, who is facing the prospect of only having three fit defenders for the derby as Nathan Ake and Manuel Akanji try to overcome injury concerns. If there is solace for City, it comes from the knowledge United are not exactly flying. Their comeback Europa League victory against Viktoria Plzen on Thursday was their third win of Amorim's short reign so far but only one of those successes has come in the Premier League, where United have lost their past two games against Arsenal and Nottingham Forest. Nevertheless, Guardiola can see improvements already on the red side of the city. \"It's already there,\" he said. \"You see all the patterns, the movements, the runners and the pace. He will do a good job at United, I'm pretty sure of that.\"\n", - "\n", - "Guardiola says skipper Kyle Walker has been offered support by the club after the City defender highlighted the racial abuse he had received on social media in the wake of the Juventus trip. \"It's unacceptable,\" he said. \"Not because it's Kyle - for any human being. \"Unfortunately it happens many times in the real world. It is not necessary to say he has the support of the entire club. It is completely unacceptable and we give our support to him.\"\n", + "Manchester City boss Pep Guardiola says he is \"fine\" desp...\n", "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.3100, Text: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", - "\n", - "The former Barcelona and Bayern Munich boss has won 15 major trophies since taking charge of City in 2016.\n", - "\n", - "The 53-year-old Spaniard was approached in the summer about the possibility of becoming England manager, but last month signed a two-year contract extension with City until 2027.\n", - "\n", - "Speaking to celebrity chef Dani Garcia on YouTube, Guardiola did not indicate when he intends to step down at City but said he would not return to club football - in the Premier League or overseas.\n", - "\n", - "\"I'm not going to manage another team,\" he said.\n", - "\n", - "\"I'm not talking about the long-term future, but what I'm not going to do is leave Manchester City, go to another country, and do the same thing as now.\n", - "\n", - "\"I wouldn't have the energy. The thought of starting somewhere else, all the process of training and so on. No, no, no. Maybe a national team, but that's different.\n", - "\n", - "\"I want to leave it and go and play golf, but I can't [if he takes a club job]. I think stopping would do me good.\"\n", - "\n", - "City have won just once since Guardiola extended his contract - and once in nine games since beating Southampton on 26 October.\n", - "\n", - "That victory came at home to Nottingham Forest last Wednesday, but was followed by a 2-2 draw at Crystal Palace at the weekend.\n", - "\n", - "The Blues visit Juventus next in the Champions League on Wednesday (20:00 GMT), before hosting Manchester United in the Premier League on Sunday (16:30).\n", - "\n", - "\"Right now we are not in the position - when we have had the results of the last seven, eight games - to talk about winning games in plural,\" said Guardiola at his pre-match news conference.\n", + "[Result 2] Vector Distance: 0.3100\n", + "Text: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", "\n", - "\"We have to win the game and not look at what happens in the next one yet.\"\n", + "The former Barcel...\n", "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.3311, Text: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", + "[Result 3] Vector Distance: 0.3311\n", + "Text: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. 'I am not good enough' - Guardiola says he must find a 'solution' after derby loss\n", - "\n", - "Pep Guardiola says his sleep has suffered during Manchester City's deepening crisis, so he will not be helped by a nightmarish conclusion to one of the most stunning defeats of his long reign. Guardiola looked agitated, animated and on edge even after City led the Manchester derby through Josko Gvardiol's 36th-minute header, his reaction to the goal one of almost disdain that it came via a deflected cross as opposed to in his purist style. He sat alone with his eyes closed sipping from a water bottle before the resumption of the second half, then was denied even the respite of victory when Manchester United gave this largely dismal derby a dramatic conclusion it barely deserved with a remarkable late comeback. First, with 88 minutes on the clock, Matheus Nunes presented Amad Diallo with the ball before compounding his error by flattening the forward as he made an attempt to recover his mistake. Bruno Fernandes completed the formalities from the penalty spot. Worse was to come two minutes later when Lisandro Martinez's routine long ball caught City's defence inexplicably statuesque. Goalkeeper Ederson's positioning was awry, allowing the lively Diallo to pounce from an acute angle to leave Guardiola and his players stunned. It was the latest into any game, 88 minutes, that reigning Premier League champions had led then lost. It was also the first time City had lost a game they were leading so late on. And in a sign of City's previous excellence that is now being challenged, they have only lost four of 105 Premier League home games under Guardiola in which they have been ahead at half-time, winning 94 and drawing seven. Guardiola delivered a brutal self-analysis as he told Match of the Day: \"I am not good enough. I am the boss. I am the manager. I have to find solutions and so far I haven't. That's the reality. \"Not much else to say. No defence. Manchester United were incredibly persistent. We have not lost eight games in two seasons. We can't defend that.\"\n", - "\n", - "Manchester City manager Pep Guardiola in despair during the derby defeat to Manchester United\n", - "\n", - "Guardiola suggested the serious renewal will wait until the summer but the red flags have been appearing for weeks in the sudden and shocking decline of a team that has lost the aura of invincibility that left many opponents beaten before kick-off in previous years. He has had stated City must \"survive\" this season - whatever qualifies as survival for a club of such rich ambition - but the quest for a record fifth successive Premier League title is surely over as they lie nine points behind leaders Liverpool having played a game more. Their Champions League aspirations are also in jeopardy after another loss, this time against Juventus in Turin. City's squad has been allowed to grow too old together. The insatiable thirst for success seems to have gone, the scales of superiority have fallen away and opponents now sense vulnerability right until the final whistle, as United did here. The manner in which United were able, and felt able, to snatch this victory drove right to the heart of how City, and Guardiola, are allowing opponents to prey on their downfall. Guardiola has every reason to cite injuries, most significantly to Rodri and also John Stones as well as others, but this cannot be used an excuse for such a dramatic decline in standards, allied to the appearance of a soft underbelly that is so easily exploited. And City's rebuild will not be a quick fix. With every performance, every defeat, the scale of what lies in front of Guardiola becomes more obvious - and daunting. Manchester City's fans did their best to reassure Guardiola of their faith in him with a giant Barcelona-inspired banner draped from the stands before kick-off emblazoned with his image reading \"Més que un entrenador\" - \"More Than A Coach\". And Guardiola will now need to be more than a coach than at any time in his career. He will have the finances but it will be done with City's challengers also strengthening. Kevin de Bruyne, 34 in June, lasted 68 minutes here before he was substituted. Age and injuries are catching up with one of the greatest players of the Premier League era and he is unlikely to be at City next season. Mateo Kovacic, who replaced De Bruyne, is also 31 in May. Kyle Walker, 34, is being increasingly exposed. His most notable contribution here was an embarrassing collapse to the ground after the mildest head-to-head collision with Rasmus Hojlund. Ilkay Gundogan, another 34-year-old and a previous pillar of Guardiola's great successes, no longer has the legs or energy to exert influence. This looks increasingly like a season too far following his return from Barcelona. Flaws are also being exposed elsewhere, with previously reliable performers failing to hit previous standards. Phil Foden scored 27 goals and had 12 assists when he was Premier League Player of the Season last term. This year he has just three goals and two assists in 18 appearances in all competitions. He has no goals and just one assist in 11 Premier League games. Jack Grealish, who came on after 77 minutes against United, has not scored in a year for Manchester City, his last goal coming in a 2-2 draw against Crystal Palace on 16 December last year. He has, in the meantime, scored twice for England. Erling Haaland is also struggling as City lack creativity and cutting edge. He has three goals in his past 11 Premier League games after scoring 10 in his first five. And in another indication of City's impotence, and their reliance on Haaland, defender Gvardiol's goal against United was his fourth this season, making him their second highest scorer in all competitions behind the Norwegian striker, who has 18. Goalkeeper Ederson, so reliable for so long, has already been dropped once this season and did not cover himself in glory for United's winner. Guardiola, with that freshly signed two-year contract, insists he \"wants it\" as he treads on this alien territory of failure. He will be under no illusions about the size of the job in front of him as he placed his head in his hands in anguish after yet another damaging and deeply revealing defeat. City and Guardiola are in new, unforgiving territory.\n", + "This video can not be played To play this video you need to enable JavaScript in ...\n", "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.3474, Text: 'Self-doubt, errors & big changes' - inside the crisis at Man City\n", - "\n", - "Pep Guardiola has not been through a moment like this in his managerial career. Manchester City have lost nine matches in their past 12 - as many defeats as they had suffered in their previous 106 fixtures. At the end of October, City were still unbeaten at the top of the Premier League and favourites to win a fifth successive title. Now they are seventh, 12 points behind leaders Liverpool having played a game more. It has been an incredible fall from grace and left people trying to work out what has happened - and whether Guardiola can make it right. After discussing the situation with those who know him best, I have taken a closer look at the future - both short and long term - and how the current crisis at Man City is going to be solved.\n", - "\n", - "Pep Guardiola's Man City have lost nine of their past 12 matches\n", - "\n", - "Guardiola has also been giving it a lot of thought. He has not been sleeping very well, as he has said, and has not been himself at times when talking to the media. He has been talking to a lot of people about what is going on as he tries to work out the reasons for City's demise. Some reasons he knows, others he still doesn't. What people perhaps do not realise is Guardiola hugely doubts himself and always has. He will be thinking \"I'm not going to be able to get us out of this\" and needs the support of people close to him to push away those insecurities - and he has that. He is protected by his people who are very aware, like he is, that there are a lot of people that want City to fail. It has been a turbulent time for Guardiola. Remember those marks he had on his head after the 3-3 draw with Feyenoord in the Champions League? He always scratches his head, it is a gesture of nervousness. Normally nothing happens but on that day one of his nails was far too sharp so, after talking to the players in the changing room where he scratched his head because of his usual agitated gesturing, he went to the news conference. His right-hand man Manel Estiarte sent him photos in a message saying \"what have you got on your head?\", but by the time Guardiola returned to the coaching room there was hardly anything there again. He started that day with a cover on his nose after the same thing happened at the training ground the day before. Guardiola was having a footballing debate with Kyle Walker about positional stuff and marked his nose with that same nail. There was also that remarkable news conference after the Manchester derby when he said \"I don't know what to do\". That is partly true and partly not true. Ignore the fact Guardiola suggested he was \"not good enough\". He actually meant he was not good enough to resolve the situation with the group of players he has available and with all the other current difficulties. There are obviously logical explanations for the crisis and the first one has been talked about many times - the absence of injured midfielder Rodri. You know the game Jenga? When you take the wrong piece out, the whole tower collapses. That is what has happened here. It is normal for teams to have an over-reliance on one player if he is the best in the world in his position. And you cannot calculate the consequences of an injury that rules someone like Rodri out for the season. City are a team, like many modern ones, in which the holding midfielder is a key element to the construction. So, when you take Rodri out, it is difficult to hold it together. There were Plan Bs - John Stones, Manuel Akanji, even Nathan Ake - but injuries struck. The big injury list has been out of the ordinary and the busy calendar has also played a part in compounding the issues. However, one factor even Guardiola cannot explain is the big uncharacteristic errors in almost every game from international players. Why did Matheus Nunes make that challenge to give away the penalty against Manchester United? Jack Grealish is sent on at the end to keep the ball and cannot do that. There are errors from Walker and other defenders. These are some of the best players in the world. Of course the players' mindset is important, and confidence is diminishing. Wrong decisions get taken so there is almost panic on the pitch instead of calm. There are also players badly out of form who are having to play because of injuries. Walker is now unable to hide behind his pace, I'm not sure Kevin de Bruyne is ever getting back to the level he used to be at, Bernardo Silva and Ilkay Gundogan do not have time to rest, Grealish is not playing at his best. Some of these players were only meant to be playing one game a week but, because of injuries, have played 12 games in 40 days. It all has a domino effect. One consequence is that Erling Haaland isn't getting the service to score. But the Norwegian still remains City's top-scorer with 13. Defender Josko Gvardiol is next on the list with just four. The way their form has been analysed inside the City camp is there have only been three games where they deserved to lose (Liverpool, Bournemouth and Aston Villa). But of course it is time to change the dynamic.\n", + "[Result 4] Vector Distance: 0.3474\n", + "Text: 'Self-doubt, errors & big changes' - inside the crisis at Man City\n", "\n", - "Guardiola has never protected his players so much. He has not criticised them and is not going to do so. They have won everything with him. Instead of doing more with them, he has tried doing less. He has sometimes given them more days off to clear their heads, so they can reset - two days this week for instance. Perhaps the time to change a team is when you are winning, but no-one was suggesting Man City were about to collapse when they were top and unbeaten after nine league games. Some people have asked how bad it has to get before City make a decision on Guardiola. The answer is that there is no decision to be made. Maybe if this was Real Madrid, Barcelona or Juventus, the pressure from outside would be massive and the argument would be made that Guardiola has to go. At City he has won the lot, so how can anyone say he is failing? Yes, this is a crisis. But given all their problems, City's renewed target is finishing in the top four. That is what is in all their heads now. The idea is to recover their essence by improving defensive concepts that are not there and re-establishing the intensity they are known for. Guardiola is planning to use the next two years of his contract, which is expected to be his last as a club manager, to prepare a new Manchester City. When he was at the end of his four years at Barcelona, he asked two managers what to do when you feel people are not responding to your instructions. Do you go or do the players go? Sir Alex Ferguson and Rafael Benitez both told him that the players need to go. Guardiola did not listen because of his emotional attachment to his players back then and he decided to leave the Camp Nou because he felt the cycle was over. He will still protect his players now but there is not the same emotional attachment - so it is the players who are going to leave this time. It is likely City will look to replace five or six regular starters. Guardiola knows it is the end of an era and the start of a new one. Changes will not be immediate and the majority of the work will be done in the summer. But they are open to any opportunities in January - and a holding midfielder is one thing they need. In the summer City might want to get Spain's Martin Zubimendi from Real Sociedad and they know 60m euros (£50m) will get him. He said no to Liverpool last summer even though everything was agreed, but he now wants to move on and the Premier League is the target. Even if they do not get Zubimendi, that is the calibre of footballer they are after. A new Manchester City is on its way - with changes driven by Guardiola, incoming sporting director Hugo Viana and the football department.\n", + "Pep Guardiola has not been through a moment like this in his managerial career. Ma...\n", "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.3665, Text: Man City's Dias ruled out for 'three or four weeks'\n", + "[Result 5] Vector Distance: 0.3665\n", + "Text: Man City's Dias ruled out for 'three or four weeks'\n", "\n", "Ruben Dias has won 10 major trophies during his time at Manchester City\n", "\n", - "Manchester City have suffered a fresh injury blow with manager Pep Guardiola confirming Portugal central defender Ruben Dias has been ruled out for \"three or four weeks\" with a muscle injury. Dias, who suffered the injury in Saturday's 2-1 defeat by Manchester United, will miss the entire festive programme and potentially the FA Cup third-round tie with Salford on 11 January. The 27-year-old also faces a battle to be fit for City's crucial Champions League trip to Paris St-Germain on 22 January. Dias has already missed seven games with a calf injury this season, adding to a defensive injury list that has seen John Stones, Nathan Ake, Manuel Akanji and Kyle Walker all ruled out at various points, while Ballon d'Or winner Rodri will miss the remainder of the domestic season after suffering a cruciate knee ligament injury. \"It's a muscular problem and he will be out for three to four weeks,\" said Guardiola. \"After 75 minutes against United he felt something. But he's so strong and wanted to stay on the pitch. Now he's injured.\" Guardiola confirmed Stones, Akanji and midfielder Mateo Kovacic have all trained this week and could feature at Aston Villa on Saturday (12:30 GMT), but said goalkeeper Ederson was \"a doubt\" with an unspecified problem. \"Ederson has been struggling with some niggles in his leg, he doesn't feel completely fine,\" said Guardiola. \"Ederson is so important for us.\" Amid City's current run of one win in 11 games, surprise has been expressed about Guardiola's use of youngsters James McAtee and Nico O'Reilly. City made a point of keeping both players despite numerous loan options. Yet McAtee has made just two substitute appearances - coming on in the last minute on both occasions - while O'Reilly is yet to make his league debut. But it seems they will stay at the club for the second half of the season, with Guardiola replying \"I don't think so\" when asked if players might leave during the January transfer window. The Spaniard said he is \"not a big fan\" of buying players in January but it is \"possible\" City will look to sign someone because \"the circumstances of this season have been special\".\n", - "\n", - "Guardiola's mood was so downbeat in the immediate aftermath of the United defeat it was easy to imagine he might conclude he was no longer capable of doing the job. He gave his players a couple of days off afterwards and was brighter when he spoke to journalists in his scheduled briefing before the Villa trip. \"We'd just finished a game that we lost in the circumstances and I was not happy,\" he said. \"I try to be honest about the feelings of my teams. We fell down six times [number of Premier League games without a win], we have to stand up seven. There is no alternative. \"I'm fine. I'm a normal person with feelings like all of us. When the situation is going well we are better but it's normal. I would not go to the press conference if we were 1-0 up and expressing something that I didn't feel.\" Former Villa forward Jack Grealish has not scored for City in over a year but Guardiola pointed out he is not the only attacking player struggling this season. He added: \"We are struggling to create a little bit up front, but always I am optimistic about my players that they are going to turn and perform well.\"\n", + "Manchester City have suf...\n", "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.3818, Text: 'We have to find a way' - Guardiola vows to end relegation form\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. 'Worrying' and 'staggering' - Why do Manchester City keep conceding?\n", - "\n", - "Manchester City are currently in relegation form and there is little sign of it ending. Saturday's 2-1 defeat at Aston Villa left them joint bottom of the form table over the past eight games with just Southampton for company. Saints, at the foot of the Premier League, have the same number of points, four, as City over their past eight matches having won one, drawn one and lost six - the same record as the floundering champions. And if Southampton - who appointed Ivan Juric as their new manager on Saturday - get at least a point at Fulham on Sunday, City will be on the worst run in the division. Even Wolves, who sacked boss Gary O'Neil last Sunday and replaced him with Vitor Pereira, have earned double the number of points during the same period having played a game fewer. They are damning statistics for Pep Guardiola, even if he does have some mitigating circumstances with injuries to Ederson, Nathan Ake and Ruben Dias - who all missed the loss at Villa Park - and the long-term loss of midfield powerhouse Rodri. Guardiola was happy with Saturday's performance, despite defeat in Birmingham, but there is little solace to take at slipping further out of the title race. He may have needed to field a half-fit Manuel Akanji and John Stones at Villa Park but that does not account for City looking a shadow of their former selves. That does not justify the error Josko Gvardiol made to gift Jhon Duran a golden chance inside the first 20 seconds, or £100m man Jack Grealish again failing to have an impact on a game. There may be legitimate reasons for City's drop off, whether that be injuries, mental fatigue or just simply a team coming to the end of its lifecycle, but their form, which has plunged off a cliff edge, would have been unthinkable as they strolled to a fourth straight title last season. \"The worrying thing is the number of goals conceded,\" said ex-England captain Alan Shearer on BBC Match of the Day. \"The number of times they were opened up because of the lack of protection and legs in midfield was staggering. There are so many things that are wrong at this moment in time.\"\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. Man City 'have to find a way' to return to form - Guardiola\n", - "\n", - "Afterwards Guardiola was calm, so much so it was difficult to hear him in the news conference, a contrast to the frustrated figure he cut on the touchline. He said: \"It depends on us. The solution is bring the players back. We have just one central defender fit, that is difficult. We are going to try next game - another opportunity and we don't think much further than that. \"Of course there are more reasons. We concede the goals we don't concede in the past, we [don't] score the goals we score in the past. Football is not just one reason. There are a lot of little factors. \"Last season we won the Premier League, but we came here and lost. We have to think positive and I have incredible trust in the guys. Some of them have incredible pride and desire to do it. We have to find a way, step by step, sooner or later to find a way back.\" Villa boss Unai Emery highlighted City's frailties, saying he felt Villa could seize on the visitors' lack of belief. \"Manchester City are a little bit under the confidence they have normally,\" he said. \"The second half was different, we dominated and we scored. Through those circumstances they were feeling worse than even in the first half.\"\n", + "[Result 6] Vector Distance: 0.3818\n", + "Text: 'We have to find a way' - Guardiola vows to end relegation form\n", "\n", - "Erling Haaland had one touch in the Villa box\n", - "\n", - "There are chinks in the armour never seen before at City under Guardiola and Erling Haaland conceded belief within the squad is low. He told TNT after the game: \"Of course, [confidence levels are] not the best. We know how important confidence is and you can see that it affects every human being. That is how it is, we have to continue and stay positive even though it is difficult.\" Haaland, with 76 goals in 83 Premier League appearances since joining City from Borussia Dortmund in 2022, had one shot and one touch in the Villa box. His 18 touches in the whole game were the lowest of all starting players and he has been self critical, despite scoring 13 goals in the top flight this season. Over City's last eight games he has netted just twice though, but Guardiola refused to criticise his star striker. He said: \"Without him we will be even worse but I like the players feeling that way. I don't agree with Erling. He needs to have the balls delivered in the right spots but he will fight for the next one.\"\n", + "This video can not be played To play this video you need to enable JavaScript in your...\n", "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.4157, Text: Man City might miss out on Champions League - Guardiola\n", - "\n", - "Erling Haaland was part of the Manchester City side that won the Champions League for the first time in 2023\n", - "\n", - "Manchester City boss Pep Guardiola says the club are in danger of missing out on a place in next season's Champions League. City are currently in their 14th consecutive season in European football's most prestigious club competition. Only Arsenal between 1998 and 2017, and Manchester United between 1996 and 2014, have a longer record of qualifying among English clubs. City are seventh in the Premier League after 17 matches, four points behind Nottingham Forest in fourth and a point behind fifth-placed Bournemouth. England are currently top of Uefa's European Performance Spot table and well placed to secure a fifth place in next season's Champions League, although City would still not qualify on current standings. \"When I said before, people laughed,\" said Guardiola. \"They said, 'qualifying for the Champions League is not a big success'. \"But I know it because it happens with clubs in this country. They were dominant for many years and after they were many years not qualifying for the Champions League.\"\n", + "[Result 7] Vector Distance: 0.4158\n", + "Text: Man City might miss out on Champions League - Guardiola\n", "\n", - "Guardiola's side host Everton on Boxing Day, before a trip to Leicester on 29 December and a home match against West Ham on 4 January. Given all three opponents are in the bottom seven, it offers City a chance to improve on an appalling recent record of four points from eight games, which Guardiola acknowledges has left their lofty European ambitions in doubt. \"The one team that has been in the Champions League for the past years has been Manchester City,\" he added. \"Now we are at risk, of course we are. Definitely.\" Arsenal, Chelsea, Liverpool and Manchester United finished in the Premier League's top four from the 2005-06 season to 2008-09. At least three of them also occupied the top four spots for 15 successive campaigns until 2012. But United have spent five out of the past 11 seasons outside the Champions League. Arsenal spent six seasons out of the competition before returning last term. Liverpool missed out all but one year in seven from 2010, while Chelsea are in their second successive campaign outside Europe's elite. This term the threat to City comes from unexpected sources. As well as Forest and Bournemouth, Aston Villa are ahead of City, while Newcastle, Fulham and Brighton are also within a couple of points. \"There are a lot of contenders,\" said Guardiola, whose side have lost nine of their last 12 games in all competitions. \"For every club it is so important and if we are not winning games, we will be out. \"If we don't qualify it is because we don't deserve it, because we were not prepared and because we had a lot of problems and didn't solve them.\"\n", + "Erling Haaland was part of the Manchester City side that won the Champions League for the fir...\n", "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.4705, Text: 'So happy he is back' - 'integral' De Bruyne 'one of best we've seen'\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. Match of the Day: How Kevin de Bruyne inspired Man City back to winning ways\n", - "\n", - "As Kevin de Bruyne made his way off the pitch after being replaced by Rico Lewis 16 minutes from the end of Manchester City's 3-0 win over Nottingham Forest, Pep Guardiola grabbed the Belgian and pulled him into an embrace. It was like a father offering affection to his son at a job well done. De Bruyne responded with a smile of satisfaction before continuing on his way to the top of the small terrace of City benches. A day earlier, Guardiola scoffed at the suggestions of Sky Sports duo Jamie Carragher and Gary Neville that there was some kind of rift between the boss and his star man. Why on earth, Guardiola countered, would he leave out someone who is capable of delivering moments like no-one else, even in his star-studded squad? Finally back on the pitch to start a Premier League game for the first time since August, De Bruyne proved exactly what Guardiola meant.\n", - "\n", - "Manchester City's Kevin de Bruyne has now scored two goals this season\n", + "[Result 8] Vector Distance: 0.4705\n", + "Text: 'So happy he is back' - 'integral' De Bruyne 'one of best we've seen'\n", "\n", - "On a night Guardiola conceded City had to win, it was the Belgian's firm header that created an eighth-minute opener for Bernardo Silva as they went on to finally end their seven-match winless run and close the gap to leaders Liverpool to nine points. The goal from De Bruyne that followed was a thing of beauty as he backed away from Jeremy Doku as his fellow countryman ran with the ball, arriving in enough space to take the short pass and send his shot into the corner. Afforded extra space by his manager's decision to play Jack Grealish alongside him in a central position, De Bruyne schemed in the way he usually does. As chances came and went, he was playing some significant role. De Bruyne lasted 74 minutes, his longest match time since completing the full 90 minutes against Brentford on 14 September. He was on the bench when he was announced as man of the match, a decision received with enthusiasm by the City support. \"I am so happy he is back,\" said Guardiola. \"He played 75 fantastic minutes. \"He deserves the best because he's a lovely guy and has been massively important for so many years since he arrived.\"\n", - "\n", - "City are now unbeaten in their past 31 Premier League games with De Bruyne starting. He has been involved in 25 goals (nine goals, 16 assists) in those games. De Bruyne said: \"There have never been issues between me and Pep. He knows I've been struggling. It's painful and uncomfortable. \"Hopefully I can get back to my body with not much pain and then I'll be fine.\" However, a bit like the victory itself - tarnished by an injury to Manuel Akanji that may rule the Switzerland defender out of the weekend trip to Crystal Palace, and a hamstring problem for Nathan Ake who has already missed five weeks with a similar injury this season that Guardiola said \"doesn’t look good\" and makes him feel \"sad\" for the Dutchman - there was a caveat as the City boss assessed De Bruyne's contribution. \"He fought a lot and he prepared himself,\" he said. \"He is back to his physicality. The minutes he played at Anfield were really good. \"Last season he was out for many months, this season as well. We will see how he recovers after a long time injured and how he feels in three days.\" De Bruyne recorded four or more shots and created four or more chances for the third time in a Premier League game this season. Despite only starting five Premier League games, only Arsenal's Bukayo Saka has done so more often this term. Guardiola's fear must be that if he pushes De Bruyne too far too quickly, his body will let him down. Former Man City defender Micah Richards told BBC Match of the Day: \"He is a top-quality player and one of the best we have seen. He always manages to find space on the pitch.\" \"He has been integral to Man City's success over a number of years,\" added former City boss Stuart Pearce on Amazon Prime. \"He is the go-to player that sets Erling Haaland alight with his passing. He creates goals, he scores goals. \"If you were to pick one player out over the last eight or nine years De Bruyne would be at the top of almost everyone's list.\" It is a delicate balance given his team are still nine points adrift of Premier League leaders Liverpool and are also outside the Champions League top eight before next week's trip to Italy and a meeting with Juventus, after which City will only have January first-phase games remaining to ensure they secure qualification for the last-16 without needing to be bothered by February's play-off round. However, as with his team, De Bruyne's recovery had to start somewhere.\n", + "This video can not be played To play this video you need to enable JavaScript i...\n", "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.4831, Text: 'Life is not easy' - Haaland penalty miss sums up Man City crisis\n", - "\n", - "Manchester City striker Erling Haaland has now missed two of his 17 penalties taken in the Premier League\n", - "\n", - "Nothing seems to be going Manchester City's way at the moment - and star striker Erling Haaland is not a happy man. If there was any player currently in the Premier League you would hand the ball to for a penalty to win a match, it would be the prolific Norwegian. Not on this occasion, though. Looking to land a knockout blow, Haaland saw his second-half strike saved by spot-kick expert Jordan Pickford, as Manchester City's crisis continued with a 1-1 draw against Everton at Etihad Stadium. \"How do you react mentally?\" asked boss Pep Guardiola about the penalty miss. \"Life is not easy. Sport is not easy. When it happens, it is OK. \"There are still a lot of minutes to play and we had the chances afterwards. We created, incredible how they ran and fight. In some games it was not good but today well played.\" At the full-time whistle, there were a smattering of boos from home supporters at Etihad Stadium and a despondent Haaland ripped off his hair bobble and shirt before heading straight down the tunnel. As Haaland trudged off out of sight, Guardiola stood motionless on the pitch with the look of a man unable to find answers to their current crisis. City's all-conquering side have suffered a remarkable drop-off and now won just once in their last 13 games in all competitions. In the post-match news conference, the stunned Spaniard was barely audible in the responses to the questions fired his way. \"My body language was positive,\" he uttered. \"The team played really good. We had I don't know how many shots. The first half was brilliant.\"\n", - "\n", - "City have a lengthy injury list, but their downturn has coincided with Haaland's struggles in front of goal. The striker made a blistering start to the campaign looked to be on his way to setting more goalscoring records when netting 10 goals in his first five games, but has only managed three in 13 since. This has largely been down to underperformance on his expected goal (xG) rate, with statistics showing he has scored five fewer goals than he should have done during this recent run. There is also a stark contrast from his shot conversion rate from the first five games to his last 13, dropping from 38.5% to a lowly 6.4%. The penalty miss was Haaland's only shot on target in the contest - but Everton did not set out to pay him extra attention. Asked if there had been a special plan to handle the frontman, Dyche said: \"No. I was asked the other day when playing Chelsea about Cole Palmer and what a great player he is turning out to be. \"We know Haaland is too but I believe in the team. I don't look at one aspect of it, I look at the team, what they are doing and they have done a good job today.\"\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser.\n", + "[Result 9] Vector Distance: 0.4831\n", + "Text: 'Life is not easy' - Haaland penalty miss sums up Man City crisis\n", "\n", - "Pickford saved his seventh penalty in the Premier League since joining Everton in 2017\n", - "\n", - "After Toffees forward Iliman Ndiaye had cancelled out Bernardo Silva's opener, the defining moment came in the second half when Vitalii Mykolenko tripped Savinho in the box. During the VAR check, it was a battle of the minds with Everton captain Seamus Coleman receiving a yellow card for trying to put off Haaland, who had the ball in his hand. The booking ultimately appeared to be a well-earned one for the team and Pickford seemed to revel in his big moment, jumping up and down on his line, sticking his tongue out before stooping low to his right to push the ball away. Pickford has made big penalty saves for England on the international stage and this was seventh spot-kick stop since joining Everton in 2017. Such is City's luck at the moment that they managed to recycle the ball and Haaland headed in, but it was ruled out for offside, much to the joy of the travelling supporters. Haaland had scored 15 of 16 of his previous penalties in the league, with the other hitting the woodwork, while this was the first he had seen saved. \"It is good from the analysts to give him as much benefit of as many penalties as they can,\" Dyche said of Pickford's save. \"Credit to him for making the right decision in the spur of the moment.\" Former England goalkeeper Paul Robinson said on BBC Radio 5 Live: \"Pickford is good at putting pressure on strikers. Haaland must have felt the pressure there, this place went silent. \"It was a decent save but a really poor penalty. He didn't whip it around like a left-footer should. A poor penalty, but a good save.\" Ex-England defender Matt Upson added on Amazon Prime: \"It was brilliant save from Pickford. He relishes situations like this. He doesn't make life any easier for the penalty taker. \"But for Haaland to miss that penalty speaks a bit about where the confidence is in this team at the moment.\"\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. Dyche 'very pleased' with Everton defence in Man City draw\n", + "Manchester City striker Erling Haaland has now missed two of his 17 penalties taken...\n", "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.5382, Text: Amorim knows job in 'danger' without victories\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. 'I know that every manager is in danger'\n", - "\n", - "Manchester United head coach Ruben Amorim says the vast expense of bringing him in will not shield him from the sack if he fails to produce a winning team. While United sources stress there is total support for the new boss inside Old Trafford, recent results and performances have made some fans nervous. Away supporters booed their team at the final whistle of the 2-0 defeat by Wolves at Molineux on Boxing Day, and with many exiting quickly it left the players to acknowledge hundreds of empty yellow seats before heading to the tunnel. \"The manager of Manchester United can never, no matter what, be comfortable,\" said Amorim. \"You can argue I have been here one month and I've had four training [sessions], but we are not winning. That is the reality.\" Amorim has collected seven points from seven Premier League games since taking charge last month - only one more point than fellow Portuguese Vitor Pereira, who has won both his games since becoming Wolves boss. Five defeats in Amorim's first 10 games is the worst record of any new United manager since Walter Crickmer, who stepped up from being club secretary in the 1930s. It is not what was anticipated when chief executive Omar Berrada flew to Lisbon to offer Amorim the job in the wake of Erik ten Hag's dismissal on 28 October. United were so convinced in Amorim, they paid Sporting £10.6m in compensation to get him out of his contract. But Amorim does not believe that will save him if results do not improve. \"I know that if we don't win, regardless if they pay the buyout or not, every manager is in danger,\" he said. \"I like that because that is the job.\"\n", - "\n", - "The five-day gap between the home game with Newcastle on 30 December and an immensely difficult visit to old rivals and title favourites Liverpool on 5 January is the longest spell Amorim will have had to work with his players since his appointment. He will have another spare week after that, then three more midweek games to work with a squad United sources say is not expected to change much in personnel during the January transfer window, because of the club's tight Profit and Sustainability position. Evidently, it would have been far easier for such a dramatic transition to take place during the summer. Amorim did ask if his switch could be delayed until the end of the season but that request was rejected by Berrada. \"There's no point talking or thinking about that,\" said Amorim. \"I'm here and have to focus on the job. \"It's part of football to have these difficult moments. I already knew it was going to be tough. You expect to win more games, to have players with more confidence to sell the idea and to work and improve things. \"At this moment it's really hard. We have to survive to have time and then to improve the team.\"\n", - "\n", - "Manchester United head coach Ruben Amorim has won four and lost five of his first 10 games in charge\n", + "[Result 10] Vector Distance: 0.5382\n", + "Text: Amorim knows job in 'danger' without victories\n", "\n", - "Former United star Cristiano Ronaldo has backed Portuguese compatriot Amorim to turn the club's fortunes around. Ronaldo's second spell at Old Trafford ended in November 2022 following an explosive TV interview with presenter and journalist Piers Morgan. \"He [Amorim] did a fantastic job in Portugal with my [club] Sporting,\" said Ronaldo. \"But the Premier League is a different beast, the most competitive league in the world. I knew that it would be tough and they will continue the storm. \"But the storm will finish and the sun will rise. Things crossed, it will be good with him and I hope the best for Manchester United because it is a club I still love.\" Al Nassr forward Ronaldo, 39, who was speaking at the Globe Soccer Awards in Dubai where he was named the Best Middle East player of 2024, added: \"I will continue to say, the problem is not the coaches. \"It's like the aquarium and you have the fish inside and it's sick, and you take him out and fix the problem. \"If you put it back in the aquarium it will be sick again. This is the problem of Manchester United. It is the same.\"\n", + "This video can not be played To play this video you need to enable JavaScript in your browser. 'I know...\n", "--------------------------------------------------------------------------------\n" ] } ], "source": [ + "# Phase 1: Baseline Performance (Without GSI Index)\n", + "print(\"=\"*80)\n", + "print(\"PHASE 1: BASELINE PERFORMANCE (NO GSI INDEX)\")\n", + "print(\"=\"*80)\n", + "\n", "query = \"What was manchester city manager pep guardiola's reaction to the team's current form?\"\n", "\n", "try:\n", " # Perform the semantic search\n", " start_time = time.time()\n", " search_results = vector_store.similarity_search_with_score(query, k=10)\n", - " search_elapsed_time = time.time() - start_time\n", + " baseline_time = time.time() - start_time\n", "\n", - " logging.info(f\"Semantic search completed in {search_elapsed_time:.2f} seconds\")\n", + " logging.info(f\"Baseline search completed in {baseline_time:.2f} seconds\")\n", "\n", " # Display search results\n", - " print(f\"\\nSemantic Search Results (completed in {search_elapsed_time:.2f} seconds):\")\n", - " print(\"-\" * 80) # Add separator line\n", - " for doc, distance in search_results:\n", - " print(f\"Vector Distance: {distance:.4f}, Text: {doc.page_content}\")\n", - " print(\"-\" * 80) # Add separator between results\n", + " print(f\"\\nBaseline Search Results (completed in {baseline_time:.4f} seconds):\")\n", + " print(\"-\" * 80)\n", + " for i, (doc, distance) in enumerate(search_results, 1):\n", + " print(f\"[Result {i}] Vector Distance: {distance:.4f}\")\n", + " # Truncate for readability\n", + " content_preview = doc.page_content[:150] + \"...\" if len(doc.page_content) > 150 else doc.page_content\n", + " print(f\"Text: {content_preview}\")\n", + " print(\"-\" * 80)\n", "\n", "except CouchbaseException as e:\n", " raise RuntimeError(f\"Error performing semantic search: {str(e)}\")\n", @@ -891,7 +873,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 21, "id": "19596edc", "metadata": {}, "outputs": [], @@ -912,15 +894,25 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 22, "id": "d1f4812c", "metadata": {}, "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "================================================================================\n", + "PHASE 2: GSI-OPTIMIZED PERFORMANCE (WITH BHIVE INDEX)\n", + "================================================================================\n" + ] + }, { "name": "stderr", "output_type": "stream", "text": [ - "2025-11-06 13:09:36,866 - INFO - Semantic search completed in 0.56 seconds\n" + "2025-11-07 13:16:48,306 - INFO - GSI-optimized search completed in 0.59 seconds\n" ] }, { @@ -928,150 +920,51 @@ "output_type": "stream", "text": [ "\n", - "Semantic Search Results (completed in 0.56 seconds):\n", - "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.2956, Text: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", - "\n", - "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affected by the worst run of results in his entire managerial career. In an interview with former Italy international Luca Toni for Amazon Prime Sport before Wednesday's Champions League defeat by Juventus, Guardiola touched on the personal impact City's sudden downturn in form has had. Guardiola said his state of mind was \"ugly\", that his sleep was \"worse\" and he was eating lighter as his digestion had suffered. City go into Sunday's derby against Manchester United at Etihad Stadium having won just one of their past 10 games. The Juventus loss means there is a chance they may not even secure a play-off spot in the Champions League. Asked to elaborate on his comments to Toni, Guardiola said: \"I'm fine. \"In our jobs we always want to do our best or the best as possible. When that doesn't happen you are more uncomfortable than when the situation is going well, always that happened. \"In good moments I am happier but when I get to the next game I am still concerned about what I have to do. There is no human being that makes an activity and it doesn't matter how they do.\" Guardiola said City have to defend better and \"avoid making mistakes at both ends\". To emphasise his point, Guardiola referred back to the third game of City's current run, against a Sporting side managed by Ruben Amorim, who will be in the United dugout at the weekend. City dominated the first half in Lisbon, led thanks to Phil Foden's early effort and looked to be cruising. Instead, they conceded three times in 11 minutes either side of half-time as Sporting eventually ran out 4-1 winners. \"I would like to play the game like we played in Lisbon on Sunday, believe me,\" said Guardiola, who is facing the prospect of only having three fit defenders for the derby as Nathan Ake and Manuel Akanji try to overcome injury concerns. If there is solace for City, it comes from the knowledge United are not exactly flying. Their comeback Europa League victory against Viktoria Plzen on Thursday was their third win of Amorim's short reign so far but only one of those successes has come in the Premier League, where United have lost their past two games against Arsenal and Nottingham Forest. Nevertheless, Guardiola can see improvements already on the red side of the city. \"It's already there,\" he said. \"You see all the patterns, the movements, the runners and the pace. He will do a good job at United, I'm pretty sure of that.\"\n", - "\n", - "Guardiola says skipper Kyle Walker has been offered support by the club after the City defender highlighted the racial abuse he had received on social media in the wake of the Juventus trip. \"It's unacceptable,\" he said. \"Not because it's Kyle - for any human being. \"Unfortunately it happens many times in the real world. It is not necessary to say he has the support of the entire club. It is completely unacceptable and we give our support to him.\"\n", - "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.3100, Text: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", - "\n", - "The former Barcelona and Bayern Munich boss has won 15 major trophies since taking charge of City in 2016.\n", - "\n", - "The 53-year-old Spaniard was approached in the summer about the possibility of becoming England manager, but last month signed a two-year contract extension with City until 2027.\n", - "\n", - "Speaking to celebrity chef Dani Garcia on YouTube, Guardiola did not indicate when he intends to step down at City but said he would not return to club football - in the Premier League or overseas.\n", - "\n", - "\"I'm not going to manage another team,\" he said.\n", - "\n", - "\"I'm not talking about the long-term future, but what I'm not going to do is leave Manchester City, go to another country, and do the same thing as now.\n", - "\n", - "\"I wouldn't have the energy. The thought of starting somewhere else, all the process of training and so on. No, no, no. Maybe a national team, but that's different.\n", - "\n", - "\"I want to leave it and go and play golf, but I can't [if he takes a club job]. I think stopping would do me good.\"\n", - "\n", - "City have won just once since Guardiola extended his contract - and once in nine games since beating Southampton on 26 October.\n", - "\n", - "That victory came at home to Nottingham Forest last Wednesday, but was followed by a 2-2 draw at Crystal Palace at the weekend.\n", - "\n", - "The Blues visit Juventus next in the Champions League on Wednesday (20:00 GMT), before hosting Manchester United in the Premier League on Sunday (16:30).\n", - "\n", - "\"Right now we are not in the position - when we have had the results of the last seven, eight games - to talk about winning games in plural,\" said Guardiola at his pre-match news conference.\n", - "\n", - "\"We have to win the game and not look at what happens in the next one yet.\"\n", + "GSI-Optimized Search Results (completed in 0.5927 seconds):\n", "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.3311, Text: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. 'I am not good enough' - Guardiola says he must find a 'solution' after derby loss\n", - "\n", - "Pep Guardiola says his sleep has suffered during Manchester City's deepening crisis, so he will not be helped by a nightmarish conclusion to one of the most stunning defeats of his long reign. Guardiola looked agitated, animated and on edge even after City led the Manchester derby through Josko Gvardiol's 36th-minute header, his reaction to the goal one of almost disdain that it came via a deflected cross as opposed to in his purist style. He sat alone with his eyes closed sipping from a water bottle before the resumption of the second half, then was denied even the respite of victory when Manchester United gave this largely dismal derby a dramatic conclusion it barely deserved with a remarkable late comeback. First, with 88 minutes on the clock, Matheus Nunes presented Amad Diallo with the ball before compounding his error by flattening the forward as he made an attempt to recover his mistake. Bruno Fernandes completed the formalities from the penalty spot. Worse was to come two minutes later when Lisandro Martinez's routine long ball caught City's defence inexplicably statuesque. Goalkeeper Ederson's positioning was awry, allowing the lively Diallo to pounce from an acute angle to leave Guardiola and his players stunned. It was the latest into any game, 88 minutes, that reigning Premier League champions had led then lost. It was also the first time City had lost a game they were leading so late on. And in a sign of City's previous excellence that is now being challenged, they have only lost four of 105 Premier League home games under Guardiola in which they have been ahead at half-time, winning 94 and drawing seven. Guardiola delivered a brutal self-analysis as he told Match of the Day: \"I am not good enough. I am the boss. I am the manager. I have to find solutions and so far I haven't. That's the reality. \"Not much else to say. No defence. Manchester United were incredibly persistent. We have not lost eight games in two seasons. We can't defend that.\"\n", - "\n", - "Manchester City manager Pep Guardiola in despair during the derby defeat to Manchester United\n", - "\n", - "Guardiola suggested the serious renewal will wait until the summer but the red flags have been appearing for weeks in the sudden and shocking decline of a team that has lost the aura of invincibility that left many opponents beaten before kick-off in previous years. He has had stated City must \"survive\" this season - whatever qualifies as survival for a club of such rich ambition - but the quest for a record fifth successive Premier League title is surely over as they lie nine points behind leaders Liverpool having played a game more. Their Champions League aspirations are also in jeopardy after another loss, this time against Juventus in Turin. City's squad has been allowed to grow too old together. The insatiable thirst for success seems to have gone, the scales of superiority have fallen away and opponents now sense vulnerability right until the final whistle, as United did here. The manner in which United were able, and felt able, to snatch this victory drove right to the heart of how City, and Guardiola, are allowing opponents to prey on their downfall. Guardiola has every reason to cite injuries, most significantly to Rodri and also John Stones as well as others, but this cannot be used an excuse for such a dramatic decline in standards, allied to the appearance of a soft underbelly that is so easily exploited. And City's rebuild will not be a quick fix. With every performance, every defeat, the scale of what lies in front of Guardiola becomes more obvious - and daunting. Manchester City's fans did their best to reassure Guardiola of their faith in him with a giant Barcelona-inspired banner draped from the stands before kick-off emblazoned with his image reading \"Més que un entrenador\" - \"More Than A Coach\". And Guardiola will now need to be more than a coach than at any time in his career. He will have the finances but it will be done with City's challengers also strengthening. Kevin de Bruyne, 34 in June, lasted 68 minutes here before he was substituted. Age and injuries are catching up with one of the greatest players of the Premier League era and he is unlikely to be at City next season. Mateo Kovacic, who replaced De Bruyne, is also 31 in May. Kyle Walker, 34, is being increasingly exposed. His most notable contribution here was an embarrassing collapse to the ground after the mildest head-to-head collision with Rasmus Hojlund. Ilkay Gundogan, another 34-year-old and a previous pillar of Guardiola's great successes, no longer has the legs or energy to exert influence. This looks increasingly like a season too far following his return from Barcelona. Flaws are also being exposed elsewhere, with previously reliable performers failing to hit previous standards. Phil Foden scored 27 goals and had 12 assists when he was Premier League Player of the Season last term. This year he has just three goals and two assists in 18 appearances in all competitions. He has no goals and just one assist in 11 Premier League games. Jack Grealish, who came on after 77 minutes against United, has not scored in a year for Manchester City, his last goal coming in a 2-2 draw against Crystal Palace on 16 December last year. He has, in the meantime, scored twice for England. Erling Haaland is also struggling as City lack creativity and cutting edge. He has three goals in his past 11 Premier League games after scoring 10 in his first five. And in another indication of City's impotence, and their reliance on Haaland, defender Gvardiol's goal against United was his fourth this season, making him their second highest scorer in all competitions behind the Norwegian striker, who has 18. Goalkeeper Ederson, so reliable for so long, has already been dropped once this season and did not cover himself in glory for United's winner. Guardiola, with that freshly signed two-year contract, insists he \"wants it\" as he treads on this alien territory of failure. He will be under no illusions about the size of the job in front of him as he placed his head in his hands in anguish after yet another damaging and deeply revealing defeat. City and Guardiola are in new, unforgiving territory.\n", - "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.3474, Text: 'Self-doubt, errors & big changes' - inside the crisis at Man City\n", - "\n", - "Pep Guardiola has not been through a moment like this in his managerial career. Manchester City have lost nine matches in their past 12 - as many defeats as they had suffered in their previous 106 fixtures. At the end of October, City were still unbeaten at the top of the Premier League and favourites to win a fifth successive title. Now they are seventh, 12 points behind leaders Liverpool having played a game more. It has been an incredible fall from grace and left people trying to work out what has happened - and whether Guardiola can make it right. After discussing the situation with those who know him best, I have taken a closer look at the future - both short and long term - and how the current crisis at Man City is going to be solved.\n", - "\n", - "Pep Guardiola's Man City have lost nine of their past 12 matches\n", - "\n", - "Guardiola has also been giving it a lot of thought. He has not been sleeping very well, as he has said, and has not been himself at times when talking to the media. He has been talking to a lot of people about what is going on as he tries to work out the reasons for City's demise. Some reasons he knows, others he still doesn't. What people perhaps do not realise is Guardiola hugely doubts himself and always has. He will be thinking \"I'm not going to be able to get us out of this\" and needs the support of people close to him to push away those insecurities - and he has that. He is protected by his people who are very aware, like he is, that there are a lot of people that want City to fail. It has been a turbulent time for Guardiola. Remember those marks he had on his head after the 3-3 draw with Feyenoord in the Champions League? He always scratches his head, it is a gesture of nervousness. Normally nothing happens but on that day one of his nails was far too sharp so, after talking to the players in the changing room where he scratched his head because of his usual agitated gesturing, he went to the news conference. His right-hand man Manel Estiarte sent him photos in a message saying \"what have you got on your head?\", but by the time Guardiola returned to the coaching room there was hardly anything there again. He started that day with a cover on his nose after the same thing happened at the training ground the day before. Guardiola was having a footballing debate with Kyle Walker about positional stuff and marked his nose with that same nail. There was also that remarkable news conference after the Manchester derby when he said \"I don't know what to do\". That is partly true and partly not true. Ignore the fact Guardiola suggested he was \"not good enough\". He actually meant he was not good enough to resolve the situation with the group of players he has available and with all the other current difficulties. There are obviously logical explanations for the crisis and the first one has been talked about many times - the absence of injured midfielder Rodri. You know the game Jenga? When you take the wrong piece out, the whole tower collapses. That is what has happened here. It is normal for teams to have an over-reliance on one player if he is the best in the world in his position. And you cannot calculate the consequences of an injury that rules someone like Rodri out for the season. City are a team, like many modern ones, in which the holding midfielder is a key element to the construction. So, when you take Rodri out, it is difficult to hold it together. There were Plan Bs - John Stones, Manuel Akanji, even Nathan Ake - but injuries struck. The big injury list has been out of the ordinary and the busy calendar has also played a part in compounding the issues. However, one factor even Guardiola cannot explain is the big uncharacteristic errors in almost every game from international players. Why did Matheus Nunes make that challenge to give away the penalty against Manchester United? Jack Grealish is sent on at the end to keep the ball and cannot do that. There are errors from Walker and other defenders. These are some of the best players in the world. Of course the players' mindset is important, and confidence is diminishing. Wrong decisions get taken so there is almost panic on the pitch instead of calm. There are also players badly out of form who are having to play because of injuries. Walker is now unable to hide behind his pace, I'm not sure Kevin de Bruyne is ever getting back to the level he used to be at, Bernardo Silva and Ilkay Gundogan do not have time to rest, Grealish is not playing at his best. Some of these players were only meant to be playing one game a week but, because of injuries, have played 12 games in 40 days. It all has a domino effect. One consequence is that Erling Haaland isn't getting the service to score. But the Norwegian still remains City's top-scorer with 13. Defender Josko Gvardiol is next on the list with just four. The way their form has been analysed inside the City camp is there have only been three games where they deserved to lose (Liverpool, Bournemouth and Aston Villa). But of course it is time to change the dynamic.\n", - "\n", - "Guardiola has never protected his players so much. He has not criticised them and is not going to do so. They have won everything with him. Instead of doing more with them, he has tried doing less. He has sometimes given them more days off to clear their heads, so they can reset - two days this week for instance. Perhaps the time to change a team is when you are winning, but no-one was suggesting Man City were about to collapse when they were top and unbeaten after nine league games. Some people have asked how bad it has to get before City make a decision on Guardiola. The answer is that there is no decision to be made. Maybe if this was Real Madrid, Barcelona or Juventus, the pressure from outside would be massive and the argument would be made that Guardiola has to go. At City he has won the lot, so how can anyone say he is failing? Yes, this is a crisis. But given all their problems, City's renewed target is finishing in the top four. That is what is in all their heads now. The idea is to recover their essence by improving defensive concepts that are not there and re-establishing the intensity they are known for. Guardiola is planning to use the next two years of his contract, which is expected to be his last as a club manager, to prepare a new Manchester City. When he was at the end of his four years at Barcelona, he asked two managers what to do when you feel people are not responding to your instructions. Do you go or do the players go? Sir Alex Ferguson and Rafael Benitez both told him that the players need to go. Guardiola did not listen because of his emotional attachment to his players back then and he decided to leave the Camp Nou because he felt the cycle was over. He will still protect his players now but there is not the same emotional attachment - so it is the players who are going to leave this time. It is likely City will look to replace five or six regular starters. Guardiola knows it is the end of an era and the start of a new one. Changes will not be immediate and the majority of the work will be done in the summer. But they are open to any opportunities in January - and a holding midfielder is one thing they need. In the summer City might want to get Spain's Martin Zubimendi from Real Sociedad and they know 60m euros (£50m) will get him. He said no to Liverpool last summer even though everything was agreed, but he now wants to move on and the Premier League is the target. Even if they do not get Zubimendi, that is the calibre of footballer they are after. A new Manchester City is on its way - with changes driven by Guardiola, incoming sporting director Hugo Viana and the football department.\n", - "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.3665, Text: Man City's Dias ruled out for 'three or four weeks'\n", - "\n", - "Ruben Dias has won 10 major trophies during his time at Manchester City\n", - "\n", - "Manchester City have suffered a fresh injury blow with manager Pep Guardiola confirming Portugal central defender Ruben Dias has been ruled out for \"three or four weeks\" with a muscle injury. Dias, who suffered the injury in Saturday's 2-1 defeat by Manchester United, will miss the entire festive programme and potentially the FA Cup third-round tie with Salford on 11 January. The 27-year-old also faces a battle to be fit for City's crucial Champions League trip to Paris St-Germain on 22 January. Dias has already missed seven games with a calf injury this season, adding to a defensive injury list that has seen John Stones, Nathan Ake, Manuel Akanji and Kyle Walker all ruled out at various points, while Ballon d'Or winner Rodri will miss the remainder of the domestic season after suffering a cruciate knee ligament injury. \"It's a muscular problem and he will be out for three to four weeks,\" said Guardiola. \"After 75 minutes against United he felt something. But he's so strong and wanted to stay on the pitch. Now he's injured.\" Guardiola confirmed Stones, Akanji and midfielder Mateo Kovacic have all trained this week and could feature at Aston Villa on Saturday (12:30 GMT), but said goalkeeper Ederson was \"a doubt\" with an unspecified problem. \"Ederson has been struggling with some niggles in his leg, he doesn't feel completely fine,\" said Guardiola. \"Ederson is so important for us.\" Amid City's current run of one win in 11 games, surprise has been expressed about Guardiola's use of youngsters James McAtee and Nico O'Reilly. City made a point of keeping both players despite numerous loan options. Yet McAtee has made just two substitute appearances - coming on in the last minute on both occasions - while O'Reilly is yet to make his league debut. But it seems they will stay at the club for the second half of the season, with Guardiola replying \"I don't think so\" when asked if players might leave during the January transfer window. The Spaniard said he is \"not a big fan\" of buying players in January but it is \"possible\" City will look to sign someone because \"the circumstances of this season have been special\".\n", - "\n", - "Guardiola's mood was so downbeat in the immediate aftermath of the United defeat it was easy to imagine he might conclude he was no longer capable of doing the job. He gave his players a couple of days off afterwards and was brighter when he spoke to journalists in his scheduled briefing before the Villa trip. \"We'd just finished a game that we lost in the circumstances and I was not happy,\" he said. \"I try to be honest about the feelings of my teams. We fell down six times [number of Premier League games without a win], we have to stand up seven. There is no alternative. \"I'm fine. I'm a normal person with feelings like all of us. When the situation is going well we are better but it's normal. I would not go to the press conference if we were 1-0 up and expressing something that I didn't feel.\" Former Villa forward Jack Grealish has not scored for City in over a year but Guardiola pointed out he is not the only attacking player struggling this season. He added: \"We are struggling to create a little bit up front, but always I am optimistic about my players that they are going to turn and perform well.\"\n", - "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.3818, Text: 'We have to find a way' - Guardiola vows to end relegation form\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. 'Worrying' and 'staggering' - Why do Manchester City keep conceding?\n", - "\n", - "Manchester City are currently in relegation form and there is little sign of it ending. Saturday's 2-1 defeat at Aston Villa left them joint bottom of the form table over the past eight games with just Southampton for company. Saints, at the foot of the Premier League, have the same number of points, four, as City over their past eight matches having won one, drawn one and lost six - the same record as the floundering champions. And if Southampton - who appointed Ivan Juric as their new manager on Saturday - get at least a point at Fulham on Sunday, City will be on the worst run in the division. Even Wolves, who sacked boss Gary O'Neil last Sunday and replaced him with Vitor Pereira, have earned double the number of points during the same period having played a game fewer. They are damning statistics for Pep Guardiola, even if he does have some mitigating circumstances with injuries to Ederson, Nathan Ake and Ruben Dias - who all missed the loss at Villa Park - and the long-term loss of midfield powerhouse Rodri. Guardiola was happy with Saturday's performance, despite defeat in Birmingham, but there is little solace to take at slipping further out of the title race. He may have needed to field a half-fit Manuel Akanji and John Stones at Villa Park but that does not account for City looking a shadow of their former selves. That does not justify the error Josko Gvardiol made to gift Jhon Duran a golden chance inside the first 20 seconds, or £100m man Jack Grealish again failing to have an impact on a game. There may be legitimate reasons for City's drop off, whether that be injuries, mental fatigue or just simply a team coming to the end of its lifecycle, but their form, which has plunged off a cliff edge, would have been unthinkable as they strolled to a fourth straight title last season. \"The worrying thing is the number of goals conceded,\" said ex-England captain Alan Shearer on BBC Match of the Day. \"The number of times they were opened up because of the lack of protection and legs in midfield was staggering. There are so many things that are wrong at this moment in time.\"\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. Man City 'have to find a way' to return to form - Guardiola\n", - "\n", - "Afterwards Guardiola was calm, so much so it was difficult to hear him in the news conference, a contrast to the frustrated figure he cut on the touchline. He said: \"It depends on us. The solution is bring the players back. We have just one central defender fit, that is difficult. We are going to try next game - another opportunity and we don't think much further than that. \"Of course there are more reasons. We concede the goals we don't concede in the past, we [don't] score the goals we score in the past. Football is not just one reason. There are a lot of little factors. \"Last season we won the Premier League, but we came here and lost. We have to think positive and I have incredible trust in the guys. Some of them have incredible pride and desire to do it. We have to find a way, step by step, sooner or later to find a way back.\" Villa boss Unai Emery highlighted City's frailties, saying he felt Villa could seize on the visitors' lack of belief. \"Manchester City are a little bit under the confidence they have normally,\" he said. \"The second half was different, we dominated and we scored. Through those circumstances they were feeling worse than even in the first half.\"\n", - "\n", - "Erling Haaland had one touch in the Villa box\n", - "\n", - "There are chinks in the armour never seen before at City under Guardiola and Erling Haaland conceded belief within the squad is low. He told TNT after the game: \"Of course, [confidence levels are] not the best. We know how important confidence is and you can see that it affects every human being. That is how it is, we have to continue and stay positive even though it is difficult.\" Haaland, with 76 goals in 83 Premier League appearances since joining City from Borussia Dortmund in 2022, had one shot and one touch in the Villa box. His 18 touches in the whole game were the lowest of all starting players and he has been self critical, despite scoring 13 goals in the top flight this season. Over City's last eight games he has netted just twice though, but Guardiola refused to criticise his star striker. He said: \"Without him we will be even worse but I like the players feeling that way. I don't agree with Erling. He needs to have the balls delivered in the right spots but he will fight for the next one.\"\n", - "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.4157, Text: Man City might miss out on Champions League - Guardiola\n", - "\n", - "Erling Haaland was part of the Manchester City side that won the Champions League for the first time in 2023\n", - "\n", - "Manchester City boss Pep Guardiola says the club are in danger of missing out on a place in next season's Champions League. City are currently in their 14th consecutive season in European football's most prestigious club competition. Only Arsenal between 1998 and 2017, and Manchester United between 1996 and 2014, have a longer record of qualifying among English clubs. City are seventh in the Premier League after 17 matches, four points behind Nottingham Forest in fourth and a point behind fifth-placed Bournemouth. England are currently top of Uefa's European Performance Spot table and well placed to secure a fifth place in next season's Champions League, although City would still not qualify on current standings. \"When I said before, people laughed,\" said Guardiola. \"They said, 'qualifying for the Champions League is not a big success'. \"But I know it because it happens with clubs in this country. They were dominant for many years and after they were many years not qualifying for the Champions League.\"\n", - "\n", - "Guardiola's side host Everton on Boxing Day, before a trip to Leicester on 29 December and a home match against West Ham on 4 January. Given all three opponents are in the bottom seven, it offers City a chance to improve on an appalling recent record of four points from eight games, which Guardiola acknowledges has left their lofty European ambitions in doubt. \"The one team that has been in the Champions League for the past years has been Manchester City,\" he added. \"Now we are at risk, of course we are. Definitely.\" Arsenal, Chelsea, Liverpool and Manchester United finished in the Premier League's top four from the 2005-06 season to 2008-09. At least three of them also occupied the top four spots for 15 successive campaigns until 2012. But United have spent five out of the past 11 seasons outside the Champions League. Arsenal spent six seasons out of the competition before returning last term. Liverpool missed out all but one year in seven from 2010, while Chelsea are in their second successive campaign outside Europe's elite. This term the threat to City comes from unexpected sources. As well as Forest and Bournemouth, Aston Villa are ahead of City, while Newcastle, Fulham and Brighton are also within a couple of points. \"There are a lot of contenders,\" said Guardiola, whose side have lost nine of their last 12 games in all competitions. \"For every club it is so important and if we are not winning games, we will be out. \"If we don't qualify it is because we don't deserve it, because we were not prepared and because we had a lot of problems and didn't solve them.\"\n", - "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.4705, Text: 'So happy he is back' - 'integral' De Bruyne 'one of best we've seen'\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. Match of the Day: How Kevin de Bruyne inspired Man City back to winning ways\n", - "\n", - "As Kevin de Bruyne made his way off the pitch after being replaced by Rico Lewis 16 minutes from the end of Manchester City's 3-0 win over Nottingham Forest, Pep Guardiola grabbed the Belgian and pulled him into an embrace. It was like a father offering affection to his son at a job well done. De Bruyne responded with a smile of satisfaction before continuing on his way to the top of the small terrace of City benches. A day earlier, Guardiola scoffed at the suggestions of Sky Sports duo Jamie Carragher and Gary Neville that there was some kind of rift between the boss and his star man. Why on earth, Guardiola countered, would he leave out someone who is capable of delivering moments like no-one else, even in his star-studded squad? Finally back on the pitch to start a Premier League game for the first time since August, De Bruyne proved exactly what Guardiola meant.\n", - "\n", - "Manchester City's Kevin de Bruyne has now scored two goals this season\n", + "[Result 1] Vector Distance: 0.2956\n", + "Text: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", "\n", - "On a night Guardiola conceded City had to win, it was the Belgian's firm header that created an eighth-minute opener for Bernardo Silva as they went on to finally end their seven-match winless run and close the gap to leaders Liverpool to nine points. The goal from De Bruyne that followed was a thing of beauty as he backed away from Jeremy Doku as his fellow countryman ran with the ball, arriving in enough space to take the short pass and send his shot into the corner. Afforded extra space by his manager's decision to play Jack Grealish alongside him in a central position, De Bruyne schemed in the way he usually does. As chances came and went, he was playing some significant role. De Bruyne lasted 74 minutes, his longest match time since completing the full 90 minutes against Brentford on 14 September. He was on the bench when he was announced as man of the match, a decision received with enthusiasm by the City support. \"I am so happy he is back,\" said Guardiola. \"He played 75 fantastic minutes. \"He deserves the best because he's a lovely guy and has been massively important for so many years since he arrived.\"\n", - "\n", - "City are now unbeaten in their past 31 Premier League games with De Bruyne starting. He has been involved in 25 goals (nine goals, 16 assists) in those games. De Bruyne said: \"There have never been issues between me and Pep. He knows I've been struggling. It's painful and uncomfortable. \"Hopefully I can get back to my body with not much pain and then I'll be fine.\" However, a bit like the victory itself - tarnished by an injury to Manuel Akanji that may rule the Switzerland defender out of the weekend trip to Crystal Palace, and a hamstring problem for Nathan Ake who has already missed five weeks with a similar injury this season that Guardiola said \"doesn’t look good\" and makes him feel \"sad\" for the Dutchman - there was a caveat as the City boss assessed De Bruyne's contribution. \"He fought a lot and he prepared himself,\" he said. \"He is back to his physicality. The minutes he played at Anfield were really good. \"Last season he was out for many months, this season as well. We will see how he recovers after a long time injured and how he feels in three days.\" De Bruyne recorded four or more shots and created four or more chances for the third time in a Premier League game this season. Despite only starting five Premier League games, only Arsenal's Bukayo Saka has done so more often this term. Guardiola's fear must be that if he pushes De Bruyne too far too quickly, his body will let him down. Former Man City defender Micah Richards told BBC Match of the Day: \"He is a top-quality player and one of the best we have seen. He always manages to find space on the pitch.\" \"He has been integral to Man City's success over a number of years,\" added former City boss Stuart Pearce on Amazon Prime. \"He is the go-to player that sets Erling Haaland alight with his passing. He creates goals, he scores goals. \"If you were to pick one player out over the last eight or nine years De Bruyne would be at the top of almost everyone's list.\" It is a delicate balance given his team are still nine points adrift of Premier League leaders Liverpool and are also outside the Champions League top eight before next week's trip to Italy and a meeting with Juventus, after which City will only have January first-phase games remaining to ensure they secure qualification for the last-16 without needing to be bothered by February's play-off round. However, as with his team, De Bruyne's recovery had to start somewhere.\n", + "Manchester City boss Pep Guardiola says he is \"fine\" desp...\n", "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.4831, Text: 'Life is not easy' - Haaland penalty miss sums up Man City crisis\n", - "\n", - "Manchester City striker Erling Haaland has now missed two of his 17 penalties taken in the Premier League\n", - "\n", - "Nothing seems to be going Manchester City's way at the moment - and star striker Erling Haaland is not a happy man. If there was any player currently in the Premier League you would hand the ball to for a penalty to win a match, it would be the prolific Norwegian. Not on this occasion, though. Looking to land a knockout blow, Haaland saw his second-half strike saved by spot-kick expert Jordan Pickford, as Manchester City's crisis continued with a 1-1 draw against Everton at Etihad Stadium. \"How do you react mentally?\" asked boss Pep Guardiola about the penalty miss. \"Life is not easy. Sport is not easy. When it happens, it is OK. \"There are still a lot of minutes to play and we had the chances afterwards. We created, incredible how they ran and fight. In some games it was not good but today well played.\" At the full-time whistle, there were a smattering of boos from home supporters at Etihad Stadium and a despondent Haaland ripped off his hair bobble and shirt before heading straight down the tunnel. As Haaland trudged off out of sight, Guardiola stood motionless on the pitch with the look of a man unable to find answers to their current crisis. City's all-conquering side have suffered a remarkable drop-off and now won just once in their last 13 games in all competitions. In the post-match news conference, the stunned Spaniard was barely audible in the responses to the questions fired his way. \"My body language was positive,\" he uttered. \"The team played really good. We had I don't know how many shots. The first half was brilliant.\"\n", - "\n", - "City have a lengthy injury list, but their downturn has coincided with Haaland's struggles in front of goal. The striker made a blistering start to the campaign looked to be on his way to setting more goalscoring records when netting 10 goals in his first five games, but has only managed three in 13 since. This has largely been down to underperformance on his expected goal (xG) rate, with statistics showing he has scored five fewer goals than he should have done during this recent run. There is also a stark contrast from his shot conversion rate from the first five games to his last 13, dropping from 38.5% to a lowly 6.4%. The penalty miss was Haaland's only shot on target in the contest - but Everton did not set out to pay him extra attention. Asked if there had been a special plan to handle the frontman, Dyche said: \"No. I was asked the other day when playing Chelsea about Cole Palmer and what a great player he is turning out to be. \"We know Haaland is too but I believe in the team. I don't look at one aspect of it, I look at the team, what they are doing and they have done a good job today.\"\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser.\n", - "\n", - "Pickford saved his seventh penalty in the Premier League since joining Everton in 2017\n", + "[Result 2] Vector Distance: 0.3100\n", + "Text: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", "\n", - "After Toffees forward Iliman Ndiaye had cancelled out Bernardo Silva's opener, the defining moment came in the second half when Vitalii Mykolenko tripped Savinho in the box. During the VAR check, it was a battle of the minds with Everton captain Seamus Coleman receiving a yellow card for trying to put off Haaland, who had the ball in his hand. The booking ultimately appeared to be a well-earned one for the team and Pickford seemed to revel in his big moment, jumping up and down on his line, sticking his tongue out before stooping low to his right to push the ball away. Pickford has made big penalty saves for England on the international stage and this was seventh spot-kick stop since joining Everton in 2017. Such is City's luck at the moment that they managed to recycle the ball and Haaland headed in, but it was ruled out for offside, much to the joy of the travelling supporters. Haaland had scored 15 of 16 of his previous penalties in the league, with the other hitting the woodwork, while this was the first he had seen saved. \"It is good from the analysts to give him as much benefit of as many penalties as they can,\" Dyche said of Pickford's save. \"Credit to him for making the right decision in the spur of the moment.\" Former England goalkeeper Paul Robinson said on BBC Radio 5 Live: \"Pickford is good at putting pressure on strikers. Haaland must have felt the pressure there, this place went silent. \"It was a decent save but a really poor penalty. He didn't whip it around like a left-footer should. A poor penalty, but a good save.\" Ex-England defender Matt Upson added on Amazon Prime: \"It was brilliant save from Pickford. He relishes situations like this. He doesn't make life any easier for the penalty taker. \"But for Haaland to miss that penalty speaks a bit about where the confidence is in this team at the moment.\"\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. Dyche 'very pleased' with Everton defence in Man City draw\n", + "The former Barcel...\n", "--------------------------------------------------------------------------------\n", - "Vector Distance: 0.5382, Text: Amorim knows job in 'danger' without victories\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. 'I know that every manager is in danger'\n", - "\n", - "Manchester United head coach Ruben Amorim says the vast expense of bringing him in will not shield him from the sack if he fails to produce a winning team. While United sources stress there is total support for the new boss inside Old Trafford, recent results and performances have made some fans nervous. Away supporters booed their team at the final whistle of the 2-0 defeat by Wolves at Molineux on Boxing Day, and with many exiting quickly it left the players to acknowledge hundreds of empty yellow seats before heading to the tunnel. \"The manager of Manchester United can never, no matter what, be comfortable,\" said Amorim. \"You can argue I have been here one month and I've had four training [sessions], but we are not winning. That is the reality.\" Amorim has collected seven points from seven Premier League games since taking charge last month - only one more point than fellow Portuguese Vitor Pereira, who has won both his games since becoming Wolves boss. Five defeats in Amorim's first 10 games is the worst record of any new United manager since Walter Crickmer, who stepped up from being club secretary in the 1930s. It is not what was anticipated when chief executive Omar Berrada flew to Lisbon to offer Amorim the job in the wake of Erik ten Hag's dismissal on 28 October. United were so convinced in Amorim, they paid Sporting £10.6m in compensation to get him out of his contract. But Amorim does not believe that will save him if results do not improve. \"I know that if we don't win, regardless if they pay the buyout or not, every manager is in danger,\" he said. \"I like that because that is the job.\"\n", - "\n", - "The five-day gap between the home game with Newcastle on 30 December and an immensely difficult visit to old rivals and title favourites Liverpool on 5 January is the longest spell Amorim will have had to work with his players since his appointment. He will have another spare week after that, then three more midweek games to work with a squad United sources say is not expected to change much in personnel during the January transfer window, because of the club's tight Profit and Sustainability position. Evidently, it would have been far easier for such a dramatic transition to take place during the summer. Amorim did ask if his switch could be delayed until the end of the season but that request was rejected by Berrada. \"There's no point talking or thinking about that,\" said Amorim. \"I'm here and have to focus on the job. \"It's part of football to have these difficult moments. I already knew it was going to be tough. You expect to win more games, to have players with more confidence to sell the idea and to work and improve things. \"At this moment it's really hard. We have to survive to have time and then to improve the team.\"\n", - "\n", - "Manchester United head coach Ruben Amorim has won four and lost five of his first 10 games in charge\n", + "[Result 3] Vector Distance: 0.3311\n", + "Text: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", "\n", - "Former United star Cristiano Ronaldo has backed Portuguese compatriot Amorim to turn the club's fortunes around. Ronaldo's second spell at Old Trafford ended in November 2022 following an explosive TV interview with presenter and journalist Piers Morgan. \"He [Amorim] did a fantastic job in Portugal with my [club] Sporting,\" said Ronaldo. \"But the Premier League is a different beast, the most competitive league in the world. I knew that it would be tough and they will continue the storm. \"But the storm will finish and the sun will rise. Things crossed, it will be good with him and I hope the best for Manchester United because it is a club I still love.\" Al Nassr forward Ronaldo, 39, who was speaking at the Globe Soccer Awards in Dubai where he was named the Best Middle East player of 2024, added: \"I will continue to say, the problem is not the coaches. \"It's like the aquarium and you have the fish inside and it's sick, and you take him out and fix the problem. \"If you put it back in the aquarium it will be sick again. This is the problem of Manchester United. It is the same.\"\n", + "This video can not be played To play this video you need to enable JavaScript in ...\n", "--------------------------------------------------------------------------------\n" ] } ], "source": [ + "# Phase 2: GSI-Optimized Performance (With BHIVE Index)\n", + "print(\"\\n\" + \"=\"*80)\n", + "print(\"PHASE 2: GSI-OPTIMIZED PERFORMANCE (WITH BHIVE INDEX)\")\n", + "print(\"=\"*80)\n", + "\n", "query = \"What was manchester city manager pep guardiola's reaction to the team's current form?\"\n", "\n", "try:\n", - " # Perform the semantic search\n", + " # Perform the semantic search with GSI\n", " start_time = time.time()\n", - " search_results = vector_store.similarity_search_with_score(query, k=10)\n", - " search_elapsed_time = time.time() - start_time\n", + " search_results = vector_store.similarity_search_with_score(query, k=3)\n", + " gsi_time = time.time() - start_time\n", "\n", - " logging.info(f\"Semantic search completed in {search_elapsed_time:.2f} seconds\")\n", + " logging.info(f\"GSI-optimized search completed in {gsi_time:.2f} seconds\")\n", "\n", " # Display search results\n", - " print(f\"\\nSemantic Search Results (completed in {search_elapsed_time:.2f} seconds):\")\n", - " print(\"-\" * 80) # Add separator line\n", - " for doc, distance in search_results:\n", - " print(f\"Vector Distance: {distance:.4f}, Text: {doc.page_content}\")\n", - " print(\"-\" * 80) # Add separator between results\n", + " print(f\"\\nGSI-Optimized Search Results (completed in {gsi_time:.4f} seconds):\")\n", + " print(\"-\" * 80)\n", + " for i, (doc, distance) in enumerate(search_results, 1):\n", + " print(f\"[Result {i}] Vector Distance: {distance:.4f}\")\n", + " # Truncate for readability\n", + " content_preview = doc.page_content[:150] + \"...\" if len(doc.page_content) > 150 else doc.page_content\n", + " print(f\"Text: {content_preview}\")\n", + " print(\"-\" * 80)\n", "\n", "except CouchbaseException as e:\n", " raise RuntimeError(f\"Error performing semantic search: {str(e)}\")\n", @@ -1079,248 +972,19 @@ " raise RuntimeError(f\"Unexpected error: {str(e)}\")" ] }, - { - "cell_type": "markdown", - "id": "bd651112", - "metadata": {}, - "source": [ - "# Vector Search Performance Optimization\n", - "\n", - "Now let's demonstrate the performance benefits of different optimization approaches available in Couchbase. We'll compare three optimization levels to show how each contributes to building a production-ready semantic search system:\n", - "\n", - "1. **Baseline (Raw Search)**: Basic vector similarity search without GSI optimization\n", - "2. **GSI-Optimized Search**: High-performance search using BHIVE GSI index\n", - "3. **Cache Benefits**: Show how caching can be applied on top of any search approach\n", - "\n", - "**Important**: Caching is orthogonal to index types - you can apply caching benefits to both raw searches and GSI-optimized searches to improve repeated query performance.\n" - ] - }, - { - "cell_type": "markdown", - "id": "6eef087c", - "metadata": {}, - "source": [ - "## Understanding Vector Search Results\n", - "\n", - "Before we start our performance comparisons, let's understand what the search results mean:\n", - "\n", - "When you perform a search query with vector search:\n", - "\n", - "1. **Query Embedding**: Your search text is converted into a vector embedding using the OpenAI model\n", - "2. **Vector Similarity Calculation**: The system compares your query vector against all stored document vectors\n", - "3. **Distance Computation**: Using the COSINE distance metric, the system calculates similarity distances\n", - "4. **Result Ranking**: Documents are ranked by their distance values (lower = more similar)\n", - "5. **Post-processing**: Results include both the document content and metadata\n", - "\n", - "**Note**: The returned value represents the vector distance between query and document embeddings. Lower distance values indicate higher similarity.\n" - ] - }, - { - "cell_type": "markdown", - "id": "9f0b75c4", - "metadata": {}, - "source": [ - "## Performance Testing Search Function\n", - "\n", - "Let's create a comprehensive search function for our performance comparison:\n" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "b901a559", - "metadata": {}, - "outputs": [], - "source": [ - "def search_with_performance_metrics(query_text, stage_name, k=3):\n", - " \"\"\"Perform optimized semantic search with detailed performance metrics\"\"\"\n", - " print(f\"\\n=== {stage_name.upper()} ===\")\n", - " print(f\"Query: \\\"{query_text}\\\"\")\n", - " \n", - " start_time = time.time()\n", - " results = vector_store.similarity_search_with_score(query_text, k=k)\n", - " end_time = time.time()\n", - " \n", - " search_time = end_time - start_time\n", - " print(f\"Search Time: {search_time:.4f} seconds\")\n", - " print(f\"Results Found: {len(results)} documents\")\n", - " \n", - " for i, (doc, distance) in enumerate(results, 1):\n", - " print(f\"\\n[Result {i}]\")\n", - " print(f\"Vector Distance: {distance:.6f} (lower = more similar)\")\n", - " # Truncate long documents for readability\n", - " content_preview = doc.page_content[:200] + \"...\" if len(doc.page_content) > 200 else doc.page_content\n", - " print(f\"Document Content: {content_preview}\")\n", - " if hasattr(doc, 'metadata') and doc.metadata:\n", - " print(f\"Metadata: {doc.metadata}\")\n", - " \n", - " return search_time, results\n" - ] - }, - { - "cell_type": "markdown", - "id": "206089f3", - "metadata": {}, - "source": [ - "## Phase 1: Baseline Performance (Raw Vector Search)\n", - "\n", - "First, let's establish baseline performance with raw vector search - no GSI optimization yet.\n", - "\n", - "**Note**: For this demonstration, we'll temporarily remove any existing GSI indexes to show the baseline performance without optimization.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "5d1a1d19", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "2025-11-06 13:09:54,294 - INFO - No existing index to drop or error: ParsingFailedException()\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Testing baseline performance without GSI optimization...\n", - "\n", - "=== PHASE 1: BASELINE VECTOR SEARCH ===\n", - "Query: \"What was manchester city manager pep guardiola's reaction to the team's current form?\"\n", - "Search Time: 0.4738 seconds\n", - "Results Found: 3 documents\n", - "\n", - "[Result 1]\n", - "Vector Distance: 0.295601 (lower = more similar)\n", - "Document Content: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", - "\n", - "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affecte...\n", - "\n", - "[Result 2]\n", - "Vector Distance: 0.309984 (lower = more similar)\n", - "Document Content: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", - "\n", - "The former Barcelona and Bayern Munich boss has won 15 major trophi...\n", - "\n", - "[Result 3]\n", - "Vector Distance: 0.331093 (lower = more similar)\n", - "Document Content: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. 'I am not good enough' - Guardiola s...\n" - ] - } - ], - "source": [ - "# Drop any existing GSI index to demonstrate baseline performance\n", - "try:\n", - " cluster.query(f\"DROP INDEX `{CB_BUCKET_NAME}`.`{SCOPE_NAME}`.`{COLLECTION_NAME}`.pydantic_ai_bhive_index\").execute()\n", - " logging.info(\"Dropped existing BHIVE index for baseline testing\")\n", - " time.sleep(2) # Wait for index drop to complete\n", - "except Exception as e:\n", - " logging.info(f\"No existing index to drop or error: {str(e)}\")\n", - "\n", - "test_query = \"What was manchester city manager pep guardiola's reaction to the team's current form?\"\n", - "print(\"Testing baseline performance without GSI optimization...\")\n", - "baseline_time, baseline_results = search_with_performance_metrics(\n", - " test_query, \"Phase 1: Baseline Vector Search\"\n", - ")\n" - ] - }, - { - "cell_type": "markdown", - "id": "5d528ee6", - "metadata": {}, - "source": [ - "## Phase 2: Create BHIVE GSI Index and Test Performance\n", - "\n", - "Now let's create the BHIVE GSI index and measure the performance improvement:\n" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "id": "73e9a28a", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Creating BHIVE GSI vector index...\n", - "✓ BHIVE GSI vector index already exists, proceeding...\n", - "\n", - "Testing performance with BHIVE GSI optimization...\n", - "\n", - "=== PHASE 2: GSI-OPTIMIZED SEARCH ===\n", - "Query: \"What was manchester city manager pep guardiola's reaction to the team's current form?\"\n", - "Search Time: 0.5026 seconds\n", - "Results Found: 3 documents\n", - "\n", - "[Result 1]\n", - "Vector Distance: 0.295612 (lower = more similar)\n", - "Document Content: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", - "\n", - "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affecte...\n", - "\n", - "[Result 2]\n", - "Vector Distance: 0.310004 (lower = more similar)\n", - "Document Content: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", - "\n", - "The former Barcelona and Bayern Munich boss has won 15 major trophi...\n", - "\n", - "[Result 3]\n", - "Vector Distance: 0.331096 (lower = more similar)\n", - "Document Content: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. 'I am not good enough' - Guardiola s...\n" - ] - } - ], - "source": [ - "# Create BHIVE index for optimized vector search\n", - "print(\"Creating BHIVE GSI vector index...\")\n", - "try:\n", - " vector_store.create_index(\n", - " index_type=IndexType.BHIVE,\n", - " index_description=\"IVF,SQ8\",\n", - " distance_metric=DistanceStrategy.COSINE,\n", - " index_name=\"pydantic_ai_bhive_index\",\n", - " )\n", - " print(\"✓ BHIVE GSI vector index created successfully!\")\n", - " \n", - " # Wait for index to become available\n", - " print(\"Waiting for index to become available...\")\n", - " time.sleep(3)\n", - " \n", - "except Exception as e:\n", - " if \"already exists\" in str(e).lower():\n", - " print(\"✓ BHIVE GSI vector index already exists, proceeding...\")\n", - " else:\n", - " print(f\"Error creating GSI index: {str(e)}\")\n", - "\n", - "# Test the same query with GSI optimization\n", - "print(\"\\nTesting performance with BHIVE GSI optimization...\")\n", - "gsi_time, gsi_results = search_with_performance_metrics(\n", - " test_query, \"Phase 2: GSI-Optimized Search\"\n", - ")\n" - ] - }, { "cell_type": "markdown", "id": "01132754", "metadata": {}, "source": [ - "## Phase 3: Demonstrate Cache Benefits\n", + "## Phase 3: LLM Cache Setup for Agent\n", "\n", - "Now let's show how caching can improve performance for repeated queries. **Note**: Caching benefits apply to both raw searches and GSI-optimized searches.\n" + "Now let's set up LLM caching which will improve performance for repeated LLM queries in the PydanticAI agent section. This caching is separate from vector search optimization and helps reduce costs by caching LLM responses.\n" ] }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 24, "id": "c8b93f92", "metadata": {}, "outputs": [ @@ -1328,47 +992,16 @@ "name": "stdout", "output_type": "stream", "text": [ - "Setting up Couchbase cache for improved performance on repeated queries...\n", - "✓ Couchbase cache enabled!\n", - "\n", - "Testing cache benefits with a different query...\n", - "First execution (cache miss):\n", - "\n", - "=== PHASE 3A: FIRST QUERY (CACHE MISS) ===\n", - "Query: \"How does Manchester United's performance compare to Manchester City?\"\n", - "Search Time: 0.7791 seconds\n", - "Results Found: 2 documents\n", - "\n", - "[Result 1]\n", - "Vector Distance: 0.461303 (lower = more similar)\n", - "Document Content: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. 'I am not good enough' - Guardiola s...\n", - "\n", - "[Result 2]\n", - "Vector Distance: 0.472090 (lower = more similar)\n", - "Document Content: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", - "\n", - "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affecte...\n", - "\n", - "Second execution (cache hit):\n", - "\n", - "=== PHASE 3B: REPEATED QUERY (CACHE HIT) ===\n", - "Query: \"How does Manchester United's performance compare to Manchester City?\"\n", - "Search Time: 0.7957 seconds\n", - "Results Found: 2 documents\n", "\n", - "[Result 1]\n", - "Vector Distance: 0.461303 (lower = more similar)\n", - "Document Content: 'I am not good enough' - Guardiola faces daunting and major rebuild\n", - "\n", - "This video can not be played To play this video you need to enable JavaScript in your browser. 'I am not good enough' - Guardiola s...\n", + "================================================================================\n", + "PHASE 3: LLM CACHE SETUP\n", + "================================================================================\n", "\n", - "[Result 2]\n", - "Vector Distance: 0.472090 (lower = more similar)\n", - "Document Content: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", + "Setting up Couchbase LLM cache...\n", + "✓ Couchbase LLM cache enabled successfully!\n", "\n", - "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affecte...\n" + "Note: This cache will store and reuse LLM responses for repeated queries,\n", + "reducing API costs and latency for the PydanticAI agent in the next section.\n" ] } ], @@ -1377,8 +1010,12 @@ "from langchain_core.globals import set_llm_cache\n", "from langchain_couchbase.cache import CouchbaseCache\n", "\n", - "# Set up Couchbase cache (can be applied to any search approach)\n", - "print(\"Setting up Couchbase cache for improved performance on repeated queries...\")\n", + "print(\"\\n\" + \"=\"*80)\n", + "print(\"PHASE 3: LLM CACHE SETUP\")\n", + "print(\"=\"*80)\n", + "\n", + "# Set up Couchbase LLM cache for the PydanticAI agent\n", + "print(\"\\nSetting up Couchbase LLM cache...\")\n", "cache = CouchbaseCache(\n", " cluster=cluster,\n", " bucket_name=CB_BUCKET_NAME,\n", @@ -1386,37 +1023,25 @@ " collection_name=COLLECTION_NAME,\n", ")\n", "set_llm_cache(cache)\n", - "print(\"✓ Couchbase cache enabled!\")\n", - "\n", - "# Test cache benefits with the same query (should show improvement on second run)\n", - "cache_query = \"How does Manchester United's performance compare to Manchester City?\"\n", - "\n", - "print(\"\\nTesting cache benefits with a different query...\")\n", - "print(\"First execution (cache miss):\")\n", - "cache_time_1, _ = search_with_performance_metrics(\n", - " cache_query, \"Phase 3a: First Query (Cache Miss)\", k=2\n", - ")\n", - "\n", - "print(\"\\nSecond execution (cache hit):\")\n", - "cache_time_2, _ = search_with_performance_metrics(\n", - " cache_query, \"Phase 3b: Repeated Query (Cache Hit)\", k=2\n", - ")\n" + "print(\"✓ Couchbase LLM cache enabled successfully!\")\n", + "print(\"\\nNote: This cache will store and reuse LLM responses for repeated queries,\")\n", + "print(\"reducing API costs and latency for the PydanticAI agent in the next section.\")\n" ] }, { "cell_type": "markdown", - "id": "f800b0c0", + "id": "898161ed", "metadata": {}, "source": [ - "## Complete Performance Analysis\n", + "## Performance Analysis Summary\n", "\n", - "Let's analyze the complete performance improvements across all optimization levels:\n" + "Let's analyze the performance improvements we've achieved through different optimization strategies:\n" ] }, { "cell_type": "code", - "execution_count": 19, - "id": "6a6e909f", + "execution_count": 25, + "id": "00a3146f", "metadata": {}, "outputs": [ { @@ -1427,23 +1052,40 @@ "================================================================================\n", "VECTOR SEARCH PERFORMANCE OPTIMIZATION SUMMARY\n", "================================================================================\n", - "Phase 1 - Baseline (Raw Search): 0.4738 seconds\n", - "Phase 2 - GSI-Optimized Search: 0.5026 seconds\n", - "Phase 3 - Cache Benefits:\n", - " First execution (cache miss): 0.7791 seconds\n", - " Second execution (cache hit): 0.7957 seconds\n", "\n", + "📊 Performance Comparison:\n", + "Optimization Level Time (seconds) Status\n", "--------------------------------------------------------------------------------\n", - "OPTIMIZATION IMPACT ANALYSIS:\n", - "--------------------------------------------------------------------------------\n", - "GSI Index Benefit: Performance similar to baseline (may vary with dataset size)\n", - "Cache Benefit: No significant improvement (results may be cached already)\n", + "Phase 1 - Baseline (No Index) 1.0870 ⚪ Baseline\n", + "Phase 2 - GSI-Optimized (BHIVE) 0.5927 ✅ Optimized\n", + "\n", + "✨ GSI Performance Gain: 1.83x faster (45.5% improvement)\n", "\n", - "Key Insights:\n", - "• GSI optimization provides consistent performance benefits, especially with larger datasets\n", - "• Caching benefits apply to both raw and GSI-optimized searches\n", - "• Combined GSI + Cache provides the best performance for production applications\n", - "• BHIVE indexes scale to billions of vectors with optimized concurrent operations\n" + "--------------------------------------------------------------------------------\n", + "KEY INSIGHTS:\n", + "--------------------------------------------------------------------------------\n", + "1. 🚀 GSI Optimization:\n", + " • BHIVE indexes excel with large-scale datasets (millions+ vectors)\n", + " • Performance gains increase with dataset size and concurrent queries\n", + " • Optimal for production workloads with sustained traffic patterns\n", + "\n", + "2. 📦 Dataset Size Impact:\n", + " • Current dataset: ~1,700 articles\n", + " • At this scale, performance differences may be minimal or variable\n", + " • Significant gains typically seen with 10M+ vectors\n", + "\n", + "3. 🎯 When to Use GSI:\n", + " • Large-scale vector search applications\n", + " • High query-per-second (QPS) requirements\n", + " • Multi-user concurrent access scenarios\n", + " • Production environments requiring scalability\n", + "\n", + "4. 💾 LLM Caching:\n", + " • Separate optimization layer for LLM responses\n", + " • Reduces API costs and latency for repeated queries\n", + " • Complements GSI optimization for complete RAG performance\n", + "\n", + "================================================================================\n" ] } ], @@ -1452,104 +1094,49 @@ "print(\"VECTOR SEARCH PERFORMANCE OPTIMIZATION SUMMARY\")\n", "print(\"=\"*80)\n", "\n", - "print(f\"Phase 1 - Baseline (Raw Search): {baseline_time:.4f} seconds\")\n", - "print(f\"Phase 2 - GSI-Optimized Search: {gsi_time:.4f} seconds\")\n", - "print(f\"Phase 3 - Cache Benefits:\")\n", - "print(f\" First execution (cache miss): {cache_time_1:.4f} seconds\")\n", - "print(f\" Second execution (cache hit): {cache_time_2:.4f} seconds\")\n", + "print(f\"\\n📊 Performance Comparison:\")\n", + "print(f\"{'Optimization Level':<35} {'Time (seconds)':<20} {'Status'}\")\n", + "print(\"-\" * 80)\n", + "print(f\"{'Phase 1 - Baseline (No Index)':<35} {baseline_time:.4f}{'':16} ⚪ Baseline\")\n", + "print(f\"{'Phase 2 - GSI-Optimized (BHIVE)':<35} {gsi_time:.4f}{'':16} ✅ Optimized\")\n", + "\n", + "# Calculate improvement\n", + "if baseline_time > gsi_time:\n", + " speedup = baseline_time / gsi_time\n", + " improvement = ((baseline_time - gsi_time) / baseline_time) * 100\n", + " print(f\"\\n✨ GSI Performance Gain: {speedup:.2f}x faster ({improvement:.1f}% improvement)\")\n", + "elif gsi_time > baseline_time:\n", + " slowdown_pct = ((gsi_time - baseline_time) / baseline_time) * 100\n", + " print(f\"\\n⚠️ Note: GSI was {slowdown_pct:.1f}% slower than baseline in this run\")\n", + " print(f\" This can happen with small datasets. GSI benefits emerge with scale.\")\n", + "else:\n", + " print(f\"\\n⚖️ Performance: Comparable to baseline\")\n", "\n", "print(\"\\n\" + \"-\"*80)\n", - "print(\"OPTIMIZATION IMPACT ANALYSIS:\")\n", + "print(\"KEY INSIGHTS:\")\n", "print(\"-\"*80)\n", - "\n", - "# GSI improvement analysis\n", - "if gsi_time and baseline_time and gsi_time < baseline_time:\n", - " gsi_speedup = baseline_time / gsi_time\n", - " gsi_improvement = ((baseline_time - gsi_time) / baseline_time) * 100\n", - " print(f\"GSI Index Benefit: {gsi_speedup:.2f}x faster ({gsi_improvement:.1f}% improvement)\")\n", - "else:\n", - " print(f\"GSI Index Benefit: Performance similar to baseline (may vary with dataset size)\")\n", - "\n", - "# Cache improvement analysis\n", - "if cache_time_2 and cache_time_1 and cache_time_2 < cache_time_1:\n", - " cache_speedup = cache_time_1 / cache_time_2\n", - " cache_improvement = ((cache_time_1 - cache_time_2) / cache_time_1) * 100\n", - " print(f\"Cache Benefit: {cache_speedup:.2f}x faster ({cache_improvement:.1f}% improvement)\")\n", - "else:\n", - " print(f\"Cache Benefit: No significant improvement (results may be cached already)\")\n", - "\n", - "print(f\"\\nKey Insights:\")\n", - "print(f\"• GSI optimization provides consistent performance benefits, especially with larger datasets\")\n", - "print(f\"• Caching benefits apply to both raw and GSI-optimized searches\")\n", - "print(f\"• Combined GSI + Cache provides the best performance for production applications\")\n", - "print(f\"• BHIVE indexes scale to billions of vectors with optimized concurrent operations\")\n" - ] - }, - { - "cell_type": "markdown", - "id": "898161ed", - "metadata": {}, - "source": [ - "## Interactive Testing\n", - "\n", - "Try your own queries with the optimized search system:\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "id": "00a3146f", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "=== INTERACTIVE GSI-OPTIMIZED SEARCH ===\n", - "Query: \"Tell me about Manchester City Uniform\"\n", - "Search Time: 0.5068 seconds\n", - "Results Found: 3 documents\n", - "\n", - "[Result 1]\n", - "Vector Distance: 0.469613 (lower = more similar)\n", - "Document Content: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", - "\n", - "Manchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affecte...\n", - "\n", - "[Result 2]\n", - "Vector Distance: 0.491903 (lower = more similar)\n", - "Document Content: 'Self-doubt, errors & big changes' - inside the crisis at Man City\n", - "\n", - "Pep Guardiola has not been through a moment like this in his managerial career. Manchester City have lost nine matches in their past...\n", - "\n", - "[Result 3]\n", - "Vector Distance: 0.515922 (lower = more similar)\n", - "Document Content: Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\n", - "\n", - "The former Barcelona and Bayern Munich boss has won 15 major trophi...\n" - ] - }, - { - "data": { - "text/plain": [ - "(0.5067887306213379,\n", - " [(Document(id='33e94f118f7946ab8b8c3a41bf886184', metadata={}, page_content='Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\\n\\nManchester City boss Pep Guardiola says he is \"fine\" despite admitting his sleep and diet are being affected by the worst run of results in his entire managerial career. In an interview with former Italy international Luca Toni for Amazon Prime Sport before Wednesday\\'s Champions League defeat by Juventus, Guardiola touched on the personal impact City\\'s sudden downturn in form has had. Guardiola said his state of mind was \"ugly\", that his sleep was \"worse\" and he was eating lighter as his digestion had suffered. City go into Sunday\\'s derby against Manchester United at Etihad Stadium having won just one of their past 10 games. The Juventus loss means there is a chance they may not even secure a play-off spot in the Champions League. Asked to elaborate on his comments to Toni, Guardiola said: \"I\\'m fine. \"In our jobs we always want to do our best or the best as possible. When that doesn\\'t happen you are more uncomfortable than when the situation is going well, always that happened. \"In good moments I am happier but when I get to the next game I am still concerned about what I have to do. There is no human being that makes an activity and it doesn\\'t matter how they do.\" Guardiola said City have to defend better and \"avoid making mistakes at both ends\". To emphasise his point, Guardiola referred back to the third game of City\\'s current run, against a Sporting side managed by Ruben Amorim, who will be in the United dugout at the weekend. City dominated the first half in Lisbon, led thanks to Phil Foden\\'s early effort and looked to be cruising. Instead, they conceded three times in 11 minutes either side of half-time as Sporting eventually ran out 4-1 winners. \"I would like to play the game like we played in Lisbon on Sunday, believe me,\" said Guardiola, who is facing the prospect of only having three fit defenders for the derby as Nathan Ake and Manuel Akanji try to overcome injury concerns. If there is solace for City, it comes from the knowledge United are not exactly flying. Their comeback Europa League victory against Viktoria Plzen on Thursday was their third win of Amorim\\'s short reign so far but only one of those successes has come in the Premier League, where United have lost their past two games against Arsenal and Nottingham Forest. Nevertheless, Guardiola can see improvements already on the red side of the city. \"It\\'s already there,\" he said. \"You see all the patterns, the movements, the runners and the pace. He will do a good job at United, I\\'m pretty sure of that.\"\\n\\nGuardiola says skipper Kyle Walker has been offered support by the club after the City defender highlighted the racial abuse he had received on social media in the wake of the Juventus trip. \"It\\'s unacceptable,\" he said. \"Not because it\\'s Kyle - for any human being. \"Unfortunately it happens many times in the real world. It is not necessary to say he has the support of the entire club. It is completely unacceptable and we give our support to him.\"'),\n", - " 0.46961301382250065),\n", - " (Document(id='20b142f6e1f54a568942aac9b4bcdd81', metadata={}, page_content='\\'Self-doubt, errors & big changes\\' - inside the crisis at Man City\\n\\nPep Guardiola has not been through a moment like this in his managerial career. Manchester City have lost nine matches in their past 12 - as many defeats as they had suffered in their previous 106 fixtures. At the end of October, City were still unbeaten at the top of the Premier League and favourites to win a fifth successive title. Now they are seventh, 12 points behind leaders Liverpool having played a game more. It has been an incredible fall from grace and left people trying to work out what has happened - and whether Guardiola can make it right. After discussing the situation with those who know him best, I have taken a closer look at the future - both short and long term - and how the current crisis at Man City is going to be solved.\\n\\nPep Guardiola\\'s Man City have lost nine of their past 12 matches\\n\\nGuardiola has also been giving it a lot of thought. He has not been sleeping very well, as he has said, and has not been himself at times when talking to the media. He has been talking to a lot of people about what is going on as he tries to work out the reasons for City\\'s demise. Some reasons he knows, others he still doesn\\'t. What people perhaps do not realise is Guardiola hugely doubts himself and always has. He will be thinking \"I\\'m not going to be able to get us out of this\" and needs the support of people close to him to push away those insecurities - and he has that. He is protected by his people who are very aware, like he is, that there are a lot of people that want City to fail. It has been a turbulent time for Guardiola. Remember those marks he had on his head after the 3-3 draw with Feyenoord in the Champions League? He always scratches his head, it is a gesture of nervousness. Normally nothing happens but on that day one of his nails was far too sharp so, after talking to the players in the changing room where he scratched his head because of his usual agitated gesturing, he went to the news conference. His right-hand man Manel Estiarte sent him photos in a message saying \"what have you got on your head?\", but by the time Guardiola returned to the coaching room there was hardly anything there again. He started that day with a cover on his nose after the same thing happened at the training ground the day before. Guardiola was having a footballing debate with Kyle Walker about positional stuff and marked his nose with that same nail. There was also that remarkable news conference after the Manchester derby when he said \"I don\\'t know what to do\". That is partly true and partly not true. Ignore the fact Guardiola suggested he was \"not good enough\". He actually meant he was not good enough to resolve the situation with the group of players he has available and with all the other current difficulties. There are obviously logical explanations for the crisis and the first one has been talked about many times - the absence of injured midfielder Rodri. You know the game Jenga? When you take the wrong piece out, the whole tower collapses. That is what has happened here. It is normal for teams to have an over-reliance on one player if he is the best in the world in his position. And you cannot calculate the consequences of an injury that rules someone like Rodri out for the season. City are a team, like many modern ones, in which the holding midfielder is a key element to the construction. So, when you take Rodri out, it is difficult to hold it together. There were Plan Bs - John Stones, Manuel Akanji, even Nathan Ake - but injuries struck. The big injury list has been out of the ordinary and the busy calendar has also played a part in compounding the issues. However, one factor even Guardiola cannot explain is the big uncharacteristic errors in almost every game from international players. Why did Matheus Nunes make that challenge to give away the penalty against Manchester United? Jack Grealish is sent on at the end to keep the ball and cannot do that. There are errors from Walker and other defenders. These are some of the best players in the world. Of course the players\\' mindset is important, and confidence is diminishing. Wrong decisions get taken so there is almost panic on the pitch instead of calm. There are also players badly out of form who are having to play because of injuries. Walker is now unable to hide behind his pace, I\\'m not sure Kevin de Bruyne is ever getting back to the level he used to be at, Bernardo Silva and Ilkay Gundogan do not have time to rest, Grealish is not playing at his best. Some of these players were only meant to be playing one game a week but, because of injuries, have played 12 games in 40 days. It all has a domino effect. One consequence is that Erling Haaland isn\\'t getting the service to score. But the Norwegian still remains City\\'s top-scorer with 13. Defender Josko Gvardiol is next on the list with just four. The way their form has been analysed inside the City camp is there have only been three games where they deserved to lose (Liverpool, Bournemouth and Aston Villa). But of course it is time to change the dynamic.\\n\\nGuardiola has never protected his players so much. He has not criticised them and is not going to do so. They have won everything with him. Instead of doing more with them, he has tried doing less. He has sometimes given them more days off to clear their heads, so they can reset - two days this week for instance. Perhaps the time to change a team is when you are winning, but no-one was suggesting Man City were about to collapse when they were top and unbeaten after nine league games. Some people have asked how bad it has to get before City make a decision on Guardiola. The answer is that there is no decision to be made. Maybe if this was Real Madrid, Barcelona or Juventus, the pressure from outside would be massive and the argument would be made that Guardiola has to go. At City he has won the lot, so how can anyone say he is failing? Yes, this is a crisis. But given all their problems, City\\'s renewed target is finishing in the top four. That is what is in all their heads now. The idea is to recover their essence by improving defensive concepts that are not there and re-establishing the intensity they are known for. Guardiola is planning to use the next two years of his contract, which is expected to be his last as a club manager, to prepare a new Manchester City. When he was at the end of his four years at Barcelona, he asked two managers what to do when you feel people are not responding to your instructions. Do you go or do the players go? Sir Alex Ferguson and Rafael Benitez both told him that the players need to go. Guardiola did not listen because of his emotional attachment to his players back then and he decided to leave the Camp Nou because he felt the cycle was over. He will still protect his players now but there is not the same emotional attachment - so it is the players who are going to leave this time. It is likely City will look to replace five or six regular starters. Guardiola knows it is the end of an era and the start of a new one. Changes will not be immediate and the majority of the work will be done in the summer. But they are open to any opportunities in January - and a holding midfielder is one thing they need. In the summer City might want to get Spain\\'s Martin Zubimendi from Real Sociedad and they know 60m euros (£50m) will get him. He said no to Liverpool last summer even though everything was agreed, but he now wants to move on and the Premier League is the target. Even if they do not get Zubimendi, that is the calibre of footballer they are after. A new Manchester City is on its way - with changes driven by Guardiola, incoming sporting director Hugo Viana and the football department.'),\n", - " 0.4919030863142537),\n", - " (Document(id='77b8b9fea1984454b49a9b566c0d893d', metadata={}, page_content='Pep Guardiola has said Manchester City will be his final managerial job in club football before he \"maybe\" coaches a national team.\\n\\nThe former Barcelona and Bayern Munich boss has won 15 major trophies since taking charge of City in 2016.\\n\\nThe 53-year-old Spaniard was approached in the summer about the possibility of becoming England manager, but last month signed a two-year contract extension with City until 2027.\\n\\nSpeaking to celebrity chef Dani Garcia on YouTube, Guardiola did not indicate when he intends to step down at City but said he would not return to club football - in the Premier League or overseas.\\n\\n\"I\\'m not going to manage another team,\" he said.\\n\\n\"I\\'m not talking about the long-term future, but what I\\'m not going to do is leave Manchester City, go to another country, and do the same thing as now.\\n\\n\"I wouldn\\'t have the energy. The thought of starting somewhere else, all the process of training and so on. No, no, no. Maybe a national team, but that\\'s different.\\n\\n\"I want to leave it and go and play golf, but I can\\'t [if he takes a club job]. I think stopping would do me good.\"\\n\\nCity have won just once since Guardiola extended his contract - and once in nine games since beating Southampton on 26 October.\\n\\nThat victory came at home to Nottingham Forest last Wednesday, but was followed by a 2-2 draw at Crystal Palace at the weekend.\\n\\nThe Blues visit Juventus next in the Champions League on Wednesday (20:00 GMT), before hosting Manchester United in the Premier League on Sunday (16:30).\\n\\n\"Right now we are not in the position - when we have had the results of the last seven, eight games - to talk about winning games in plural,\" said Guardiola at his pre-match news conference.\\n\\n\"We have to win the game and not look at what happens in the next one yet.\"'),\n", - " 0.5159221949345671)])" - ] - }, - "execution_count": 20, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "custom_query = input(\"Enter your search query: \")\n", - "search_with_performance_metrics(custom_query, \"Interactive GSI-Optimized Search\")\n" + "print(\"1. 🚀 GSI Optimization:\")\n", + "print(\" • BHIVE indexes excel with large-scale datasets (millions+ vectors)\")\n", + "print(\" • Performance gains increase with dataset size and concurrent queries\")\n", + "print(\" • Optimal for production workloads with sustained traffic patterns\")\n", + "\n", + "print(\"\\n2. 📦 Dataset Size Impact:\")\n", + "print(f\" • Current dataset: ~1,700 articles\")\n", + "print(\" • At this scale, performance differences may be minimal or variable\")\n", + "print(\" • Significant gains typically seen with 10M+ vectors\")\n", + "\n", + "print(\"\\n3. 🎯 When to Use GSI:\")\n", + "print(\" • Large-scale vector search applications\")\n", + "print(\" • High query-per-second (QPS) requirements\")\n", + "print(\" • Multi-user concurrent access scenarios\")\n", + "print(\" • Production environments requiring scalability\")\n", + "\n", + "print(\"\\n4. 💾 LLM Caching:\")\n", + "print(\" • Separate optimization layer for LLM responses\")\n", + "print(\" • Reduces API costs and latency for repeated queries\")\n", + "print(\" • Complements GSI optimization for complete RAG performance\")\n", + "\n", + "print(\"\\n\" + \"=\"*80)\n" ] }, { @@ -1583,14 +1170,14 @@ "\n", "PydanticAI makes heavy use of dependency injection to provide data and services to your agent's system prompts and tools. We define dependencies using a `dataclass`, which serves as a container for our dependencies.\n", "\n", - "In our case, the only dependency for our agent to work in the `CouchbaseQueryVectorStore` instance. However, we will still use a `dataclass` as it is good practice. In the future, in case we wish to add more dependencies, we can just add more fields to the `dataclass` `Deps`.\n", + "In our case, the only dependency for our agent to work is the `CouchbaseQueryVectorStore` instance. However, we will still use a `dataclass` as it is good practice. In the future, in case we wish to add more dependencies, we can just add more fields to the `dataclass` `Deps`.\n", "\n", "We also initialize an agent as a GPT-4o model. PydanticAI supports many different LLM providers, including Anthropic, Google, Cohere, etc. which can also be used. While initializing the agent, we also pass the type of the dependencies. This is mainly used for type checking, and not actually used at runtime.\n" ] }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 26, "id": "ec663fb6", "metadata": {}, "outputs": [], @@ -1617,7 +1204,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 27, "id": "fab753b9", "metadata": {}, "outputs": [], @@ -1647,7 +1234,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 28, "id": "e9a034ab", "metadata": {}, "outputs": [], @@ -1671,7 +1258,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 29, "id": "799e9fb5", "metadata": {}, "outputs": [ @@ -1680,21 +1267,15 @@ "output_type": "stream", "text": [ "==================== Agent Output ====================\n", - "Manchester City manager Pep Guardiola has been candid about the team's current form, admitting to personal impacts such as affected sleep and diet due to stress over what he describes as the \"worst run of results\" in his managerial career. He expressed that his state of mind was \"ugly\" and acknowledged that the team's downturn has made him uncomfortable. Guardiola emphasized the team's need to improve defensively and reduce mistakes.\n", + "Manchester City manager Pep Guardiola has expressed significant concern over the team's current form, noting that the situation is affecting him personally and professionally. Guardiola admitted that recent poor performances have impacted his sleep and diet, referring to his mental state as \"ugly\" due to the team's downturn. He emphasized the need for better defense and fewer mistakes as crucial areas for improvement.\n", "\n", - "In addition, he openly analyzed his own role in the situation, stating, \"I am not good enough. I am the boss. I am the manager. I have to find solutions and so far I haven't.\" Despite this brutal self-criticism, Guardiola remains committed to finding a way to turn the team's fortunes around.\n", + "Guardiola described this period as possibly the most difficult in his managerial career, acknowledging that he has not been able to find solutions to their struggles. His analysis was self-critical, admitting, \"I am not good enough. I am the boss. I am the manager. I have to find solutions and so far I haven't.\"\n", "\n", - "Recently, Guardiola noted the need for serious rebuilding, while acknowledging the challenges of an aging squad and injuries to key players. However, he continues to express optimism and trust in his players, believing in their desire and pride to overcome current difficulties.\n", + "The team is currently experiencing one of its worst runs, having only won one out of their last ten games. This has placed them in an unfamiliar position, both in the Premier League and in European competitions. Guardiola faces additional challenges due to injuries and a need for what he termed a \"major rebuild.\"\n", "\n", - "In summary, Guardiola's reaction to Manchester City's current form reflects a mix of personal distress, professional determination to find solutions, and a realistic acknowledgment of the challenges the team faces.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/var/folders/zd/hmksn4rj5kd_pj7w7bw48y9m0000gn/T/ipykernel_55688/1716427794.py:5: DeprecationWarning: `result.data` is deprecated, use `result.output` instead.\n", - " print(output.data)\n" + "Despite these issues, Guardiola remains hopeful, stating that the team needs to \"survive\" this season, with aspirations for a record fifth successive Premier League title seemingly distant as they trail league leaders Liverpool significantly. He has insisted that City must \"find a way, step by step\" to return to form.\n", + "\n", + "Guardiola also faced a dramatic Manchester derby loss, resulting in a further blow to the team and an introspective moment for him as he commits to resolving the situation moving forward.\n" ] } ], @@ -1703,7 +1284,7 @@ "output = await run_agent(query)\n", "\n", "print(\"=\" * 20, \"Agent Output\", \"=\" * 20)\n", - "print(output.data)\n" + "print(output.output)\n" ] }, { @@ -1717,13 +1298,13 @@ "In the cell below, we see an extremely detailed list of all the model's messages and tool calls, which happens step by step:\n", "1. The `UserPromptPart`, which consists of the query the user sends to the agent.\n", "2. The agent calls the `retrieve` tool in the `ToolCallPart` message. This includes the `search_query` argument. Couchbase uses this `search_query` to perform semantic search over all the ingested news articles.\n", - "3. The `retrieve` tool returns a `ToolReturnPart` object with all the context required for the model to answer the user's query. The retrieve documents were truncated, because a large amount of context was retrieved. \n", + "3. The `retrieve` tool returns a `ToolReturnPart` object with all the context required for the model to answer the user's query. The retrieved documents were truncated, because a large amount of context was retrieved. \n", "4. The final message is the LLM generated response with the added context, which is sent back to the user.\n" ] }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 30, "id": "3ea69905", "metadata": {}, "outputs": [ @@ -1734,16 +1315,16 @@ "Step 1:\n", "('ModelRequest(parts=[UserPromptPart(content=\"What was manchester city manager '\n", " 'pep guardiola\\'s reaction to the team\\'s current form?\", '\n", - " 'timestamp=datetime.datetime(2025, 11, 6, 7, 41, 32, 361643, '\n", + " 'timestamp=datetime.datetime(2025, 11, 7, 7, 48, 34, 167630, '\n", " \"tzinfo=datetime.timezone.utc), part_kind='user-prompt')], instructions=None, \"\n", " \"kind='request')\")\n", "==================================================\n", "Step 2:\n", "(\"ModelResponse(parts=[ToolCallPart(tool_name='retrieve', \"\n", - " 'args=\\'{\"search_query\":\"Manchester City Pep Guardiola reaction to team\\\\\\'s '\n", - " 'current form\"}\\', tool_call_id=\\'call_kLchsPmUPdNovuMcE2StuBxU\\', '\n", + " 'args=\\'{\"search_query\":\"Man City manager Pep Guardiola reaction to team\\\\\\'s '\n", + " 'current form\"}\\', tool_call_id=\\'call_JiFf7HC9upNSFvLznTB0VI1w\\', '\n", " \"part_kind='tool-call')], model_name='gpt-4o-2024-08-06', \"\n", - " 'timestamp=datetime.datetime(2025, 11, 6, 7, 41, 33, '\n", + " 'timestamp=datetime.datetime(2025, 11, 7, 7, 48, 35, '\n", " \"tzinfo=datetime.timezone.utc), kind='response')\")\n", "==================================================\n", "Step 3:\n", @@ -1904,72 +1485,7 @@ " 'results of the last seven, eight games - to talk about winning games in '\n", " 'plural,\" said Guardiola at his pre-match news conference.\\\\n\\\\n\"We have to '\n", " 'win the game and not look at what happens in the next one yet.\"\\\\n\\\\n# '\n", - " \"Documents:\\\\n\\\\'We have to find a way\\\\' - Guardiola vows to end relegation \"\n", - " 'form\\\\n\\\\nThis video can not be played To play this video you need to enable '\n", - " \"JavaScript in your browser. \\\\'Worrying\\\\' and \\\\'staggering\\\\' - Why do \"\n", - " 'Manchester City keep conceding?\\\\n\\\\nManchester City are currently in '\n", - " \"relegation form and there is little sign of it ending. Saturday\\\\'s 2-1 \"\n", - " 'defeat at Aston Villa left them joint bottom of the form table over the past '\n", - " 'eight games with just Southampton for company. Saints, at the foot of the '\n", - " 'Premier League, have the same number of points, four, as City over their '\n", - " 'past eight matches having won one, drawn one and lost six - the same record '\n", - " 'as the floundering champions. And if Southampton - who appointed Ivan Juric '\n", - " 'as their new manager on Saturday - get at least a point at Fulham on Sunday, '\n", - " 'City will be on the worst run in the division. Even Wolves, who sacked boss '\n", - " \"Gary O\\\\'Neil last Sunday and replaced him with Vitor Pereira, have earned \"\n", - " 'double the number of points during the same period having played a game '\n", - " 'fewer. They are damning statistics for Pep Guardiola, even if he does have '\n", - " 'some mitigating circumstances with injuries to Ederson, Nathan Ake and Ruben '\n", - " 'Dias - who all missed the loss at Villa Park - and the long-term loss of '\n", - " \"midfield powerhouse Rodri. Guardiola was happy with Saturday\\\\'s \"\n", - " 'performance, despite defeat in Birmingham, but there is little solace to '\n", - " 'take at slipping further out of the title race. He may have needed to field '\n", - " 'a half-fit Manuel Akanji and John Stones at Villa Park but that does not '\n", - " 'account for City looking a shadow of their former selves. That does not '\n", - " 'justify the error Josko Gvardiol made to gift Jhon Duran a golden chance '\n", - " 'inside the first 20 seconds, or £100m man Jack Grealish again failing to '\n", - " \"have an impact on a game. There may be legitimate reasons for City\\\\'s drop \"\n", - " 'off, whether that be injuries, mental fatigue or just simply a team coming '\n", - " 'to the end of its lifecycle, but their form, which has plunged off a cliff '\n", - " 'edge, would have been unthinkable as they strolled to a fourth straight '\n", - " 'title last season. \"The worrying thing is the number of goals conceded,\" '\n", - " 'said ex-England captain Alan Shearer on BBC Match of the Day. \"The number of '\n", - " 'times they were opened up because of the lack of protection and legs in '\n", - " 'midfield was staggering. There are so many things that are wrong at this '\n", - " 'moment in time.\"\\\\n\\\\nThis video can not be played To play this video you '\n", - " \"need to enable JavaScript in your browser. Man City \\\\'have to find a way\\\\' \"\n", - " 'to return to form - Guardiola\\\\n\\\\nAfterwards Guardiola was calm, so much so '\n", - " 'it was difficult to hear him in the news conference, a contrast to the '\n", - " 'frustrated figure he cut on the touchline. He said: \"It depends on us. The '\n", - " 'solution is bring the players back. We have just one central defender fit, '\n", - " 'that is difficult. We are going to try next game - another opportunity and '\n", - " 'we don\\\\\\'t think much further than that. \"Of course there are more reasons. '\n", - " \"We concede the goals we don\\\\'t concede in the past, we [don\\\\'t] score the \"\n", - " 'goals we score in the past. Football is not just one reason. There are a lot '\n", - " 'of little factors. \"Last season we won the Premier League, but we came here '\n", - " 'and lost. We have to think positive and I have incredible trust in the guys. '\n", - " 'Some of them have incredible pride and desire to do it. We have to find a '\n", - " 'way, step by step, sooner or later to find a way back.\" Villa boss Unai '\n", - " \"Emery highlighted City\\\\'s frailties, saying he felt Villa could seize on \"\n", - " 'the visitors\\\\\\' lack of belief. \"Manchester City are a little bit under the '\n", - " 'confidence they have normally,\" he said. \"The second half was different, we '\n", - " 'dominated and we scored. Through those circumstances they were feeling worse '\n", - " 'than even in the first half.\"\\\\n\\\\nErling Haaland had one touch in the Villa '\n", - " 'box\\\\n\\\\nThere are chinks in the armour never seen before at City under '\n", - " 'Guardiola and Erling Haaland conceded belief within the squad is low. He '\n", - " 'told TNT after the game: \"Of course, [confidence levels are] not the best. '\n", - " 'We know how important confidence is and you can see that it affects every '\n", - " 'human being. That is how it is, we have to continue and stay positive even '\n", - " 'though it is difficult.\" Haaland, with 76 goals in 83 Premier League '\n", - " 'appearances since joining City from Borussia Dortmund in 2022, had one shot '\n", - " 'and one touch in the Villa box. His 18 touches in the whole game were the '\n", - " 'lowest of all starting players and he has been self critical, despite '\n", - " \"scoring 13 goals in the top flight this season. Over City\\\\'s last eight \"\n", - " 'games he has netted just twice though, but Guardiola refused to criticise '\n", - " 'his star striker. He said: \"Without him we will be even worse but I like the '\n", - " \"players feeling that way. I don\\\\'t agree with Erling. He needs to have the \"\n", - " 'balls delivered in the right spots but he will fight for the next '\n", - " 'one.\"\\\\n\\\\n# Documents:\\\\nMan City\\\\\\'s Dias ruled out for \\\\\\'three or four '\n", + " \"Documents:\\\\nMan City\\\\'s Dias ruled out for \\\\'three or four \"\n", " \"weeks\\\\'\\\\n\\\\nRuben Dias has won 10 major trophies during his time at \"\n", " 'Manchester City\\\\n\\\\nManchester City have suffered a fresh injury blow with '\n", " 'manager Pep Guardiola confirming Portugal central defender Ruben Dias has '\n", @@ -2016,32 +1532,102 @@ " 'out he is not the only attacking player struggling this season. He added: '\n", " '\"We are struggling to create a little bit up front, but always I am '\n", " 'optimistic about my players that they are going to turn and perform '\n", - " 'well.\"\\', tool_call_id=\\'call_kLchsPmUPdNovuMcE2StuBxU\\', '\n", - " 'timestamp=datetime.datetime(2025, 11, 6, 7, 41, 35, 220049, '\n", + " 'well.\"\\\\n\\\\n# Documents:\\\\n\\\\\\'We have to find a way\\\\\\' - Guardiola vows to '\n", + " 'end relegation form\\\\n\\\\nThis video can not be played To play this video you '\n", + " \"need to enable JavaScript in your browser. \\\\'Worrying\\\\' and \"\n", + " \"\\\\'staggering\\\\' - Why do Manchester City keep conceding?\\\\n\\\\nManchester \"\n", + " 'City are currently in relegation form and there is little sign of it ending. '\n", + " \"Saturday\\\\'s 2-1 defeat at Aston Villa left them joint bottom of the form \"\n", + " 'table over the past eight games with just Southampton for company. Saints, '\n", + " 'at the foot of the Premier League, have the same number of points, four, as '\n", + " 'City over their past eight matches having won one, drawn one and lost six - '\n", + " 'the same record as the floundering champions. And if Southampton - who '\n", + " 'appointed Ivan Juric as their new manager on Saturday - get at least a point '\n", + " 'at Fulham on Sunday, City will be on the worst run in the division. Even '\n", + " \"Wolves, who sacked boss Gary O\\\\'Neil last Sunday and replaced him with \"\n", + " 'Vitor Pereira, have earned double the number of points during the same '\n", + " 'period having played a game fewer. They are damning statistics for Pep '\n", + " 'Guardiola, even if he does have some mitigating circumstances with injuries '\n", + " 'to Ederson, Nathan Ake and Ruben Dias - who all missed the loss at Villa '\n", + " 'Park - and the long-term loss of midfield powerhouse Rodri. Guardiola was '\n", + " \"happy with Saturday\\\\'s performance, despite defeat in Birmingham, but there \"\n", + " 'is little solace to take at slipping further out of the title race. He may '\n", + " 'have needed to field a half-fit Manuel Akanji and John Stones at Villa Park '\n", + " 'but that does not account for City looking a shadow of their former selves. '\n", + " 'That does not justify the error Josko Gvardiol made to gift Jhon Duran a '\n", + " 'golden chance inside the first 20 seconds, or £100m man Jack Grealish again '\n", + " 'failing to have an impact on a game. There may be legitimate reasons for '\n", + " \"City\\\\'s drop off, whether that be injuries, mental fatigue or just simply a \"\n", + " 'team coming to the end of its lifecycle, but their form, which has plunged '\n", + " 'off a cliff edge, would have been unthinkable as they strolled to a fourth '\n", + " 'straight title last season. \"The worrying thing is the number of goals '\n", + " 'conceded,\" said ex-England captain Alan Shearer on BBC Match of the Day. '\n", + " '\"The number of times they were opened up because of the lack of protection '\n", + " 'and legs in midfield was staggering. There are so many things that are wrong '\n", + " 'at this moment in time.\"\\\\n\\\\nThis video can not be played To play this '\n", + " \"video you need to enable JavaScript in your browser. Man City \\\\'have to \"\n", + " \"find a way\\\\' to return to form - Guardiola\\\\n\\\\nAfterwards Guardiola was \"\n", + " 'calm, so much so it was difficult to hear him in the news conference, a '\n", + " 'contrast to the frustrated figure he cut on the touchline. He said: \"It '\n", + " 'depends on us. The solution is bring the players back. We have just one '\n", + " 'central defender fit, that is difficult. We are going to try next game - '\n", + " 'another opportunity and we don\\\\\\'t think much further than that. \"Of course '\n", + " \"there are more reasons. We concede the goals we don\\\\'t concede in the past, \"\n", + " \"we [don\\\\'t] score the goals we score in the past. Football is not just one \"\n", + " 'reason. There are a lot of little factors. \"Last season we won the Premier '\n", + " 'League, but we came here and lost. We have to think positive and I have '\n", + " 'incredible trust in the guys. Some of them have incredible pride and desire '\n", + " 'to do it. We have to find a way, step by step, sooner or later to find a way '\n", + " 'back.\" Villa boss Unai Emery highlighted City\\\\\\'s frailties, saying he felt '\n", + " 'Villa could seize on the visitors\\\\\\' lack of belief. \"Manchester City are a '\n", + " 'little bit under the confidence they have normally,\" he said. \"The second '\n", + " 'half was different, we dominated and we scored. Through those circumstances '\n", + " 'they were feeling worse than even in the first half.\"\\\\n\\\\nErling Haaland '\n", + " 'had one touch in the Villa box\\\\n\\\\nThere are chinks in the armour never '\n", + " 'seen before at City under Guardiola and Erling Haaland conceded belief '\n", + " 'within the squad is low. He told TNT after the game: \"Of course, [confidence '\n", + " 'levels are] not the best. We know how important confidence is and you can '\n", + " 'see that it affects every human being. That is how it is, we have to '\n", + " 'continue and stay positive even though it is difficult.\" Haaland, with 76 '\n", + " 'goals in 83 Premier League appearances since joining City from Borussia '\n", + " 'Dortmund in 2022, had one shot and one touch in the Villa box. His 18 '\n", + " 'touches in the whole game were the lowest of all starting players and he has '\n", + " 'been self critical, despite scoring 13 goals in the top flight this season. '\n", + " \"Over City\\\\'s last eight games he has netted just twice though, but \"\n", + " 'Guardiola refused to criticise his star striker. He said: \"Without him we '\n", + " \"will be even worse but I like the players feeling that way. I don\\\\'t agree \"\n", + " 'with Erling. He needs to have the balls delivered in the right spots but he '\n", + " 'will fight for the next one.\"\\', '\n", + " \"tool_call_id='call_JiFf7HC9upNSFvLznTB0VI1w', \"\n", + " 'timestamp=datetime.datetime(2025, 11, 7, 7, 48, 36, 507812, '\n", " \"tzinfo=datetime.timezone.utc), part_kind='tool-return')], instructions=None, \"\n", " \"kind='request')\")\n", "==================================================\n", "Step 4:\n", "(\"ModelResponse(parts=[TextPart(content='Manchester City manager Pep Guardiola \"\n", - " \"has been candid about the team\\\\'s current form, admitting to personal \"\n", - " 'impacts such as affected sleep and diet due to stress over what he describes '\n", - " 'as the \"worst run of results\" in his managerial career. He expressed that '\n", - " 'his state of mind was \"ugly\" and acknowledged that the team\\\\\\'s downturn '\n", - " \"has made him uncomfortable. Guardiola emphasized the team\\\\'s need to \"\n", - " 'improve defensively and reduce mistakes.\\\\n\\\\nIn addition, he openly '\n", - " 'analyzed his own role in the situation, stating, \"I am not good enough. I am '\n", - " 'the boss. I am the manager. I have to find solutions and so far I '\n", - " 'haven\\\\\\'t.\" Despite this brutal self-criticism, Guardiola remains committed '\n", - " \"to finding a way to turn the team\\\\'s fortunes around.\\\\n\\\\nRecently, \"\n", - " 'Guardiola noted the need for serious rebuilding, while acknowledging the '\n", - " 'challenges of an aging squad and injuries to key players. However, he '\n", - " 'continues to express optimism and trust in his players, believing in their '\n", - " 'desire and pride to overcome current difficulties.\\\\n\\\\nIn summary, '\n", - " \"Guardiola\\\\'s reaction to Manchester City\\\\'s current form reflects a mix of \"\n", - " 'personal distress, professional determination to find solutions, and a '\n", - " \"realistic acknowledgment of the challenges the team faces.', \"\n", - " \"part_kind='text')], model_name='gpt-4o-2024-08-06', \"\n", - " 'timestamp=datetime.datetime(2025, 11, 6, 7, 41, 36, '\n", + " \"has expressed significant concern over the team\\\\'s current form, noting \"\n", + " 'that the situation is affecting him personally and professionally. Guardiola '\n", + " 'admitted that recent poor performances have impacted his sleep and diet, '\n", + " 'referring to his mental state as \"ugly\" due to the team\\\\\\'s downturn. He '\n", + " 'emphasized the need for better defense and fewer mistakes as crucial areas '\n", + " 'for improvement.\\\\n\\\\nGuardiola described this period as possibly the most '\n", + " 'difficult in his managerial career, acknowledging that he has not been able '\n", + " 'to find solutions to their struggles. His analysis was self-critical, '\n", + " 'admitting, \"I am not good enough. I am the boss. I am the manager. I have to '\n", + " 'find solutions and so far I haven\\\\\\'t.\"\\\\n\\\\nThe team is currently '\n", + " 'experiencing one of its worst runs, having only won one out of their last '\n", + " 'ten games. This has placed them in an unfamiliar position, both in the '\n", + " 'Premier League and in European competitions. Guardiola faces additional '\n", + " 'challenges due to injuries and a need for what he termed a \"major '\n", + " 'rebuild.\"\\\\n\\\\nDespite these issues, Guardiola remains hopeful, stating that '\n", + " 'the team needs to \"survive\" this season, with aspirations for a record fifth '\n", + " 'successive Premier League title seemingly distant as they trail league '\n", + " 'leaders Liverpool significantly. He has insisted that City must \"find a way, '\n", + " 'step by step\" to return to form.\\\\n\\\\nGuardiola also faced a dramatic '\n", + " 'Manchester derby loss, resulting in a further blow to the team and an '\n", + " 'introspective moment for him as he commits to resolving the situation moving '\n", + " \"forward.', part_kind='text')], model_name='gpt-4o-2024-08-06', \"\n", + " 'timestamp=datetime.datetime(2025, 11, 7, 7, 48, 37, '\n", " \"tzinfo=datetime.timezone.utc), kind='response')\")\n", "==================================================\n" ] From 19186726aa8e40cb89e5ff2869209836ce58b345 Mon Sep 17 00:00:00 2001 From: AayushTyagi1 Date: Fri, 7 Nov 2025 14:16:10 +0530 Subject: [PATCH 9/9] DA-1075: Modified: Removed Caching --- .../RAG_with_Couchbase_and_PydanticAI.ipynb | 545 +++++++++--------- 1 file changed, 266 insertions(+), 279 deletions(-) diff --git a/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb b/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb index 4bc2fcb0..6a7b2847 100644 --- a/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb +++ b/pydantic_ai/gsi/RAG_with_Couchbase_and_PydanticAI.ipynb @@ -60,10 +60,21 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "id": "b885b1a8", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m25.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.3\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n", + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], "source": [ "%pip install --quiet datasets==3.5.0 langchain-couchbase==0.5.0rc1 langchain-openai==0.3.32 python-dotenv==1.1.1 pydantic-ai==0.1.1" ] @@ -188,7 +199,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-11-07 13:08:54,718 - INFO - Successfully connected to Couchbase\n" + "2025-11-07 14:02:31,041 - INFO - Successfully connected to Couchbase\n" ] } ], @@ -233,7 +244,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 13, "id": "d9494a6f", "metadata": {}, "outputs": [ @@ -241,21 +252,21 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-11-07 13:13:43,503 - INFO - Bucket 'travel-sample' exists.\n", - "2025-11-07 13:13:43,505 - INFO - Scope 'shared' does not exist. Creating it...\n", - "2025-11-07 13:13:43,540 - INFO - Scope 'shared' created successfully.\n", - "2025-11-07 13:13:43,543 - INFO - Collection 'demo' does not exist. Creating it...\n", - "2025-11-07 13:13:43,594 - INFO - Collection 'demo' created successfully.\n", - "2025-11-07 13:13:45,644 - INFO - All documents cleared from the collection.\n" + "2025-11-07 14:08:09,363 - INFO - Bucket 'travel-sample' exists.\n", + "2025-11-07 14:08:09,368 - INFO - Scope 'shared' does not exist. Creating it...\n", + "2025-11-07 14:08:09,417 - INFO - Scope 'shared' created successfully.\n", + "2025-11-07 14:08:09,420 - INFO - Collection 'demo' does not exist. Creating it...\n", + "2025-11-07 14:08:09,480 - INFO - Collection 'demo' created successfully.\n", + "2025-11-07 14:08:11,534 - INFO - All documents cleared from the collection.\n" ] }, { "data": { "text/plain": [ - "" + "" ] }, - "execution_count": 16, + "execution_count": 13, "metadata": {}, "output_type": "execute_result" } @@ -336,7 +347,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 14, "id": "162e5be5", "metadata": {}, "outputs": [ @@ -344,7 +355,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-11-07 13:09:11,598 - INFO - Successfully created OpenAIEmbeddings\n" + "2025-11-07 14:08:15,469 - INFO - Successfully created OpenAIEmbeddings\n" ] } ], @@ -489,7 +500,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 15, "id": "2603a98a", "metadata": {}, "outputs": [ @@ -497,7 +508,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-11-07 13:16:27,214 - INFO - Successfully created vector store\n" + "2025-11-07 14:08:23,662 - INFO - Successfully created vector store\n" ] } ], @@ -529,7 +540,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 16, "id": "2bfb848c", "metadata": {}, "outputs": [ @@ -537,7 +548,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-11-07 13:14:26,898 - INFO - Successfully loaded the BBC News dataset with 2687 rows.\n" + "2025-11-07 14:08:28,000 - INFO - Successfully loaded the BBC News dataset with 2687 rows.\n" ] }, { @@ -572,7 +583,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 17, "id": "81457c3a", "metadata": {}, "outputs": [ @@ -621,7 +632,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 18, "id": "7d85286e", "metadata": {}, "outputs": [ @@ -629,7 +640,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-11-07 13:15:32,204 - INFO - Document ingestion completed successfully.\n" + "2025-11-07 14:09:34,499 - INFO - Document ingestion completed successfully.\n" ] } ], @@ -685,8 +696,7 @@ "## Performance Testing Phases\n", "\n", "1. **Phase 1 - Baseline Performance**: Test vector search without GSI indexes to establish baseline metrics\n", - "2. **Phase 2 - GSI-Optimized Search**: Create BHIVE index and measure performance improvements \n", - "3. **Phase 3 - LLM Cache Setup**: Configure caching for the PydanticAI agent to optimize LLM responses\n", + "2. **Phase 2 - GSI-Optimized Search**: Create BHIVE index and measure performance improvements\n", "\n", "**Important Context:**\n", "- GSI performance benefits scale with dataset size and concurrent load\n", @@ -697,7 +707,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 19, "id": "be49ce6f", "metadata": {}, "outputs": [ @@ -714,7 +724,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-11-07 13:12:25,470 - INFO - Baseline search completed in 1.09 seconds\n" + "2025-11-07 14:09:41,604 - INFO - Baseline search completed in 1.49 seconds\n" ] }, { @@ -722,7 +732,7 @@ "output_type": "stream", "text": [ "\n", - "Baseline Search Results (completed in 1.0870 seconds):\n", + "Baseline Search Results (completed in 1.4906 seconds):\n", "--------------------------------------------------------------------------------\n", "[Result 1] Vector Distance: 0.2956\n", "Text: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", @@ -873,7 +883,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 20, "id": "19596edc", "metadata": {}, "outputs": [], @@ -894,7 +904,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 21, "id": "d1f4812c", "metadata": {}, "outputs": [ @@ -912,7 +922,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "2025-11-07 13:16:48,306 - INFO - GSI-optimized search completed in 0.59 seconds\n" + "2025-11-07 14:09:58,611 - INFO - GSI-optimized search completed in 0.47 seconds\n" ] }, { @@ -920,7 +930,7 @@ "output_type": "stream", "text": [ "\n", - "GSI-Optimized Search Results (completed in 0.5927 seconds):\n", + "GSI-Optimized Search Results (completed in 0.4728 seconds):\n", "--------------------------------------------------------------------------------\n", "[Result 1] Vector Distance: 0.2956\n", "Text: Manchester City boss Pep Guardiola has won 18 trophies since he arrived at the club in 2016\n", @@ -972,62 +982,6 @@ " raise RuntimeError(f\"Unexpected error: {str(e)}\")" ] }, - { - "cell_type": "markdown", - "id": "01132754", - "metadata": {}, - "source": [ - "## Phase 3: LLM Cache Setup for Agent\n", - "\n", - "Now let's set up LLM caching which will improve performance for repeated LLM queries in the PydanticAI agent section. This caching is separate from vector search optimization and helps reduce costs by caching LLM responses.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "id": "c8b93f92", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\n", - "================================================================================\n", - "PHASE 3: LLM CACHE SETUP\n", - "================================================================================\n", - "\n", - "Setting up Couchbase LLM cache...\n", - "✓ Couchbase LLM cache enabled successfully!\n", - "\n", - "Note: This cache will store and reuse LLM responses for repeated queries,\n", - "reducing API costs and latency for the PydanticAI agent in the next section.\n" - ] - } - ], - "source": [ - "# Import cache utilities\n", - "from langchain_core.globals import set_llm_cache\n", - "from langchain_couchbase.cache import CouchbaseCache\n", - "\n", - "print(\"\\n\" + \"=\"*80)\n", - "print(\"PHASE 3: LLM CACHE SETUP\")\n", - "print(\"=\"*80)\n", - "\n", - "# Set up Couchbase LLM cache for the PydanticAI agent\n", - "print(\"\\nSetting up Couchbase LLM cache...\")\n", - "cache = CouchbaseCache(\n", - " cluster=cluster,\n", - " bucket_name=CB_BUCKET_NAME,\n", - " scope_name=SCOPE_NAME,\n", - " collection_name=COLLECTION_NAME,\n", - ")\n", - "set_llm_cache(cache)\n", - "print(\"✓ Couchbase LLM cache enabled successfully!\")\n", - "print(\"\\nNote: This cache will store and reuse LLM responses for repeated queries,\")\n", - "print(\"reducing API costs and latency for the PydanticAI agent in the next section.\")\n" - ] - }, { "cell_type": "markdown", "id": "898161ed", @@ -1040,7 +994,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 26, "id": "00a3146f", "metadata": {}, "outputs": [ @@ -1056,10 +1010,10 @@ "📊 Performance Comparison:\n", "Optimization Level Time (seconds) Status\n", "--------------------------------------------------------------------------------\n", - "Phase 1 - Baseline (No Index) 1.0870 ⚪ Baseline\n", - "Phase 2 - GSI-Optimized (BHIVE) 0.5927 ✅ Optimized\n", + "Phase 1 - Baseline (No Index) 1.4906 ⚪ Baseline\n", + "Phase 2 - GSI-Optimized (BHIVE) 0.4728 ✅ Optimized\n", "\n", - "✨ GSI Performance Gain: 1.83x faster (45.5% improvement)\n", + "✨ GSI Performance Gain: 3.15x faster (68.3% improvement)\n", "\n", "--------------------------------------------------------------------------------\n", "KEY INSIGHTS:\n", @@ -1080,11 +1034,6 @@ " • Multi-user concurrent access scenarios\n", " • Production environments requiring scalability\n", "\n", - "4. 💾 LLM Caching:\n", - " • Separate optimization layer for LLM responses\n", - " • Reduces API costs and latency for repeated queries\n", - " • Complements GSI optimization for complete RAG performance\n", - "\n", "================================================================================\n" ] } @@ -1131,11 +1080,6 @@ "print(\" • Multi-user concurrent access scenarios\")\n", "print(\" • Production environments requiring scalability\")\n", "\n", - "print(\"\\n4. 💾 LLM Caching:\")\n", - "print(\" • Separate optimization layer for LLM responses\")\n", - "print(\" • Reduces API costs and latency for repeated queries\")\n", - "print(\" • Complements GSI optimization for complete RAG performance\")\n", - "\n", "print(\"\\n\" + \"=\"*80)\n" ] }, @@ -1177,7 +1121,7 @@ }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 27, "id": "ec663fb6", "metadata": {}, "outputs": [], @@ -1204,7 +1148,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 28, "id": "fab753b9", "metadata": {}, "outputs": [], @@ -1234,7 +1178,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 29, "id": "e9a034ab", "metadata": {}, "outputs": [], @@ -1258,7 +1202,7 @@ }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 30, "id": "799e9fb5", "metadata": {}, "outputs": [ @@ -1267,15 +1211,15 @@ "output_type": "stream", "text": [ "==================== Agent Output ====================\n", - "Manchester City manager Pep Guardiola has expressed significant concern over the team's current form, noting that the situation is affecting him personally and professionally. Guardiola admitted that recent poor performances have impacted his sleep and diet, referring to his mental state as \"ugly\" due to the team's downturn. He emphasized the need for better defense and fewer mistakes as crucial areas for improvement.\n", + "Pep Guardiola, the manager of Manchester City, has been experiencing an unprecedented period of poor form with his team. This downturn has significantly impacted him personally and professionally. Guardiola admitted that this challenging period has affected his sleep and diet, stating his mental state as \"ugly\" while enduring the worst sequence of results in his managerial career. Despite these challenges, he expressed that he remains \"fine\" and committed to addressing the issues facing the team.\n", "\n", - "Guardiola described this period as possibly the most difficult in his managerial career, acknowledging that he has not been able to find solutions to their struggles. His analysis was self-critical, admitting, \"I am not good enough. I am the boss. I am the manager. I have to find solutions and so far I haven't.\"\n", + "Guardiola criticized himself publicly, stating, \"I am not good enough. I am the boss. I am the manager. I have to find solutions and so far I haven't. That's the reality.\" These statements highlight his struggles in finding solutions for the team's current problems.\n", "\n", - "The team is currently experiencing one of its worst runs, having only won one out of their last ten games. This has placed them in an unfamiliar position, both in the Premier League and in European competitions. Guardiola faces additional challenges due to injuries and a need for what he termed a \"major rebuild.\"\n", + "Concerns about the team's defensive performance and mistakes at both ends of the pitch have been highlighted as key areas for improvement. The absence of key players due to injuries, including midfielder Rodri, has compounded the situation, leading to a lack of consistency and stability in the team.\n", "\n", - "Despite these issues, Guardiola remains hopeful, stating that the team needs to \"survive\" this season, with aspirations for a record fifth successive Premier League title seemingly distant as they trail league leaders Liverpool significantly. He has insisted that City must \"find a way, step by step\" to return to form.\n", + "Internally, Guardiola has been working on a possible long-term rebuild of the team, aiming to replace several regular starters to address the core issues. He plans to use the time remaining on his contract to prepare a new Manchester City, potentially integrating new talents and addressing strategic weaknesses.\n", "\n", - "Guardiola also faced a dramatic Manchester derby loss, resulting in a further blow to the team and an introspective moment for him as he commits to resolving the situation moving forward.\n" + "Though he has not criticized his players personally, Guardiola has given them time off to refresh and reset mentally, recognizing the intense schedule and pressures they have been under. Despite the downturn, Guardiola's commitment to the team remains strong, with plans for future changes reflecting a realistic assessment of the challenges faced by the club.\n" ] } ], @@ -1304,7 +1248,7 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 31, "id": "3ea69905", "metadata": {}, "outputs": [ @@ -1315,16 +1259,16 @@ "Step 1:\n", "('ModelRequest(parts=[UserPromptPart(content=\"What was manchester city manager '\n", " 'pep guardiola\\'s reaction to the team\\'s current form?\", '\n", - " 'timestamp=datetime.datetime(2025, 11, 7, 7, 48, 34, 167630, '\n", + " 'timestamp=datetime.datetime(2025, 11, 7, 8, 45, 7, 384589, '\n", " \"tzinfo=datetime.timezone.utc), part_kind='user-prompt')], instructions=None, \"\n", " \"kind='request')\")\n", "==================================================\n", "Step 2:\n", "(\"ModelResponse(parts=[ToolCallPart(tool_name='retrieve', \"\n", - " 'args=\\'{\"search_query\":\"Man City manager Pep Guardiola reaction to team\\\\\\'s '\n", - " 'current form\"}\\', tool_call_id=\\'call_JiFf7HC9upNSFvLznTB0VI1w\\', '\n", + " 'args=\\'{\"search_query\":\"Manchester City Pep Guardiola reaction team '\n", + " 'form\"}\\', tool_call_id=\\'call_W3weSbPGeZzkCmQh0p3nYVb6\\', '\n", " \"part_kind='tool-call')], model_name='gpt-4o-2024-08-06', \"\n", - " 'timestamp=datetime.datetime(2025, 11, 7, 7, 48, 35, '\n", + " 'timestamp=datetime.datetime(2025, 11, 7, 8, 45, 7, '\n", " \"tzinfo=datetime.timezone.utc), kind='response')\")\n", "==================================================\n", "Step 3:\n", @@ -1371,6 +1315,137 @@ " 'Kyle - for any human being. \"Unfortunately it happens many times in the real '\n", " 'world. It is not necessary to say he has the support of the entire club. It '\n", " 'is completely unacceptable and we give our support to him.\"\\\\n\\\\n# '\n", + " \"Documents:\\\\n\\\\'Self-doubt, errors & big changes\\\\' - inside the crisis at \"\n", + " 'Man City\\\\n\\\\nPep Guardiola has not been through a moment like this in his '\n", + " 'managerial career. Manchester City have lost nine matches in their past 12 - '\n", + " 'as many defeats as they had suffered in their previous 106 fixtures. At the '\n", + " 'end of October, City were still unbeaten at the top of the Premier League '\n", + " 'and favourites to win a fifth successive title. Now they are seventh, 12 '\n", + " 'points behind leaders Liverpool having played a game more. It has been an '\n", + " 'incredible fall from grace and left people trying to work out what has '\n", + " 'happened - and whether Guardiola can make it right. After discussing the '\n", + " 'situation with those who know him best, I have taken a closer look at the '\n", + " 'future - both short and long term - and how the current crisis at Man City '\n", + " \"is going to be solved.\\\\n\\\\nPep Guardiola\\\\'s Man City have lost nine of \"\n", + " 'their past 12 matches\\\\n\\\\nGuardiola has also been giving it a lot of '\n", + " 'thought. He has not been sleeping very well, as he has said, and has not '\n", + " 'been himself at times when talking to the media. He has been talking to a '\n", + " 'lot of people about what is going on as he tries to work out the reasons for '\n", + " \"City\\\\'s demise. Some reasons he knows, others he still doesn\\\\'t. What \"\n", + " 'people perhaps do not realise is Guardiola hugely doubts himself and always '\n", + " 'has. He will be thinking \"I\\\\\\'m not going to be able to get us out of this\" '\n", + " 'and needs the support of people close to him to push away those insecurities '\n", + " '- and he has that. He is protected by his people who are very aware, like he '\n", + " 'is, that there are a lot of people that want City to fail. It has been a '\n", + " 'turbulent time for Guardiola. Remember those marks he had on his head after '\n", + " 'the 3-3 draw with Feyenoord in the Champions League? He always scratches his '\n", + " 'head, it is a gesture of nervousness. Normally nothing happens but on that '\n", + " 'day one of his nails was far too sharp so, after talking to the players in '\n", + " 'the changing room where he scratched his head because of his usual agitated '\n", + " 'gesturing, he went to the news conference. His right-hand man Manel Estiarte '\n", + " 'sent him photos in a message saying \"what have you got on your head?\", but '\n", + " 'by the time Guardiola returned to the coaching room there was hardly '\n", + " 'anything there again. He started that day with a cover on his nose after the '\n", + " 'same thing happened at the training ground the day before. Guardiola was '\n", + " 'having a footballing debate with Kyle Walker about positional stuff and '\n", + " 'marked his nose with that same nail. There was also that remarkable news '\n", + " 'conference after the Manchester derby when he said \"I don\\\\\\'t know what to '\n", + " 'do\". That is partly true and partly not true. Ignore the fact Guardiola '\n", + " 'suggested he was \"not good enough\". He actually meant he was not good enough '\n", + " 'to resolve the situation with the group of players he has available and with '\n", + " 'all the other current difficulties. There are obviously logical explanations '\n", + " 'for the crisis and the first one has been talked about many times - the '\n", + " 'absence of injured midfielder Rodri. You know the game Jenga? When you take '\n", + " 'the wrong piece out, the whole tower collapses. That is what has happened '\n", + " 'here. It is normal for teams to have an over-reliance on one player if he is '\n", + " 'the best in the world in his position. And you cannot calculate the '\n", + " 'consequences of an injury that rules someone like Rodri out for the season. '\n", + " 'City are a team, like many modern ones, in which the holding midfielder is a '\n", + " 'key element to the construction. So, when you take Rodri out, it is '\n", + " 'difficult to hold it together. There were Plan Bs - John Stones, Manuel '\n", + " 'Akanji, even Nathan Ake - but injuries struck. The big injury list has been '\n", + " 'out of the ordinary and the busy calendar has also played a part in '\n", + " 'compounding the issues. However, one factor even Guardiola cannot explain is '\n", + " 'the big uncharacteristic errors in almost every game from international '\n", + " 'players. Why did Matheus Nunes make that challenge to give away the penalty '\n", + " 'against Manchester United? Jack Grealish is sent on at the end to keep the '\n", + " 'ball and cannot do that. There are errors from Walker and other defenders. '\n", + " \"These are some of the best players in the world. Of course the players\\\\' \"\n", + " 'mindset is important, and confidence is diminishing. Wrong decisions get '\n", + " 'taken so there is almost panic on the pitch instead of calm. There are also '\n", + " 'players badly out of form who are having to play because of injuries. Walker '\n", + " \"is now unable to hide behind his pace, I\\\\'m not sure Kevin de Bruyne is \"\n", + " 'ever getting back to the level he used to be at, Bernardo Silva and Ilkay '\n", + " 'Gundogan do not have time to rest, Grealish is not playing at his best. Some '\n", + " 'of these players were only meant to be playing one game a week but, because '\n", + " 'of injuries, have played 12 games in 40 days. It all has a domino effect. '\n", + " \"One consequence is that Erling Haaland isn\\\\'t getting the service to score. \"\n", + " \"But the Norwegian still remains City\\\\'s top-scorer with 13. Defender Josko \"\n", + " 'Gvardiol is next on the list with just four. The way their form has been '\n", + " 'analysed inside the City camp is there have only been three games where they '\n", + " 'deserved to lose (Liverpool, Bournemouth and Aston Villa). But of course it '\n", + " 'is time to change the dynamic.\\\\n\\\\nGuardiola has never protected his '\n", + " 'players so much. He has not criticised them and is not going to do so. They '\n", + " 'have won everything with him. Instead of doing more with them, he has tried '\n", + " 'doing less. He has sometimes given them more days off to clear their heads, '\n", + " 'so they can reset - two days this week for instance. Perhaps the time to '\n", + " 'change a team is when you are winning, but no-one was suggesting Man City '\n", + " 'were about to collapse when they were top and unbeaten after nine league '\n", + " 'games. Some people have asked how bad it has to get before City make a '\n", + " 'decision on Guardiola. The answer is that there is no decision to be made. '\n", + " 'Maybe if this was Real Madrid, Barcelona or Juventus, the pressure from '\n", + " 'outside would be massive and the argument would be made that Guardiola has '\n", + " 'to go. At City he has won the lot, so how can anyone say he is failing? Yes, '\n", + " \"this is a crisis. But given all their problems, City\\\\'s renewed target is \"\n", + " 'finishing in the top four. That is what is in all their heads now. The idea '\n", + " 'is to recover their essence by improving defensive concepts that are not '\n", + " 'there and re-establishing the intensity they are known for. Guardiola is '\n", + " 'planning to use the next two years of his contract, which is expected to be '\n", + " 'his last as a club manager, to prepare a new Manchester City. When he was at '\n", + " 'the end of his four years at Barcelona, he asked two managers what to do '\n", + " 'when you feel people are not responding to your instructions. Do you go or '\n", + " 'do the players go? Sir Alex Ferguson and Rafael Benitez both told him that '\n", + " 'the players need to go. Guardiola did not listen because of his emotional '\n", + " 'attachment to his players back then and he decided to leave the Camp Nou '\n", + " 'because he felt the cycle was over. He will still protect his players now '\n", + " 'but there is not the same emotional attachment - so it is the players who '\n", + " 'are going to leave this time. It is likely City will look to replace five or '\n", + " 'six regular starters. Guardiola knows it is the end of an era and the start '\n", + " 'of a new one. Changes will not be immediate and the majority of the work '\n", + " 'will be done in the summer. But they are open to any opportunities in '\n", + " 'January - and a holding midfielder is one thing they need. In the summer '\n", + " \"City might want to get Spain\\\\'s Martin Zubimendi from Real Sociedad and \"\n", + " 'they know 60m euros (£50m) will get him. He said no to Liverpool last summer '\n", + " 'even though everything was agreed, but he now wants to move on and the '\n", + " 'Premier League is the target. Even if they do not get Zubimendi, that is the '\n", + " 'calibre of footballer they are after. A new Manchester City is on its way - '\n", + " 'with changes driven by Guardiola, incoming sporting director Hugo Viana and '\n", + " 'the football department.\\\\n\\\\n# Documents:\\\\nPep Guardiola has said '\n", + " 'Manchester City will be his final managerial job in club football before he '\n", + " '\"maybe\" coaches a national team.\\\\n\\\\nThe former Barcelona and Bayern Munich '\n", + " 'boss has won 15 major trophies since taking charge of City in 2016.\\\\n\\\\nThe '\n", + " '53-year-old Spaniard was approached in the summer about the possibility of '\n", + " 'becoming England manager, but last month signed a two-year contract '\n", + " 'extension with City until 2027.\\\\n\\\\nSpeaking to celebrity chef Dani Garcia '\n", + " 'on YouTube, Guardiola did not indicate when he intends to step down at City '\n", + " 'but said he would not return to club football - in the Premier League or '\n", + " 'overseas.\\\\n\\\\n\"I\\\\\\'m not going to manage another team,\" he '\n", + " 'said.\\\\n\\\\n\"I\\\\\\'m not talking about the long-term future, but what I\\\\\\'m '\n", + " 'not going to do is leave Manchester City, go to another country, and do the '\n", + " 'same thing as now.\\\\n\\\\n\"I wouldn\\\\\\'t have the energy. The thought of '\n", + " 'starting somewhere else, all the process of training and so on. No, no, no. '\n", + " 'Maybe a national team, but that\\\\\\'s different.\\\\n\\\\n\"I want to leave it and '\n", + " \"go and play golf, but I can\\\\'t [if he takes a club job]. I think stopping \"\n", + " 'would do me good.\"\\\\n\\\\nCity have won just once since Guardiola extended his '\n", + " 'contract - and once in nine games since beating Southampton on 26 '\n", + " 'October.\\\\n\\\\nThat victory came at home to Nottingham Forest last Wednesday, '\n", + " 'but was followed by a 2-2 draw at Crystal Palace at the weekend.\\\\n\\\\nThe '\n", + " 'Blues visit Juventus next in the Champions League on Wednesday (20:00 GMT), '\n", + " 'before hosting Manchester United in the Premier League on Sunday '\n", + " '(16:30).\\\\n\\\\n\"Right now we are not in the position - when we have had the '\n", + " 'results of the last seven, eight games - to talk about winning games in '\n", + " 'plural,\" said Guardiola at his pre-match news conference.\\\\n\\\\n\"We have to '\n", + " 'win the game and not look at what happens in the next one yet.\"\\\\n\\\\n# '\n", " \"Documents:\\\\n\\\\'I am not good enough\\\\' - Guardiola faces daunting and major \"\n", " 'rebuild\\\\n\\\\nThis video can not be played To play this video you need to '\n", " \"enable JavaScript in your browser. \\\\'I am not good enough\\\\' - Guardiola \"\n", @@ -1459,175 +1534,87 @@ " 'alien territory of failure. He will be under no illusions about the size of '\n", " 'the job in front of him as he placed his head in his hands in anguish after '\n", " 'yet another damaging and deeply revealing defeat. City and Guardiola are in '\n", - " 'new, unforgiving territory.\\\\n\\\\n# Documents:\\\\nPep Guardiola has said '\n", - " 'Manchester City will be his final managerial job in club football before he '\n", - " '\"maybe\" coaches a national team.\\\\n\\\\nThe former Barcelona and Bayern Munich '\n", - " 'boss has won 15 major trophies since taking charge of City in 2016.\\\\n\\\\nThe '\n", - " '53-year-old Spaniard was approached in the summer about the possibility of '\n", - " 'becoming England manager, but last month signed a two-year contract '\n", - " 'extension with City until 2027.\\\\n\\\\nSpeaking to celebrity chef Dani Garcia '\n", - " 'on YouTube, Guardiola did not indicate when he intends to step down at City '\n", - " 'but said he would not return to club football - in the Premier League or '\n", - " 'overseas.\\\\n\\\\n\"I\\\\\\'m not going to manage another team,\" he '\n", - " 'said.\\\\n\\\\n\"I\\\\\\'m not talking about the long-term future, but what I\\\\\\'m '\n", - " 'not going to do is leave Manchester City, go to another country, and do the '\n", - " 'same thing as now.\\\\n\\\\n\"I wouldn\\\\\\'t have the energy. The thought of '\n", - " 'starting somewhere else, all the process of training and so on. No, no, no. '\n", - " 'Maybe a national team, but that\\\\\\'s different.\\\\n\\\\n\"I want to leave it and '\n", - " \"go and play golf, but I can\\\\'t [if he takes a club job]. I think stopping \"\n", - " 'would do me good.\"\\\\n\\\\nCity have won just once since Guardiola extended his '\n", - " 'contract - and once in nine games since beating Southampton on 26 '\n", - " 'October.\\\\n\\\\nThat victory came at home to Nottingham Forest last Wednesday, '\n", - " 'but was followed by a 2-2 draw at Crystal Palace at the weekend.\\\\n\\\\nThe '\n", - " 'Blues visit Juventus next in the Champions League on Wednesday (20:00 GMT), '\n", - " 'before hosting Manchester United in the Premier League on Sunday '\n", - " '(16:30).\\\\n\\\\n\"Right now we are not in the position - when we have had the '\n", - " 'results of the last seven, eight games - to talk about winning games in '\n", - " 'plural,\" said Guardiola at his pre-match news conference.\\\\n\\\\n\"We have to '\n", - " 'win the game and not look at what happens in the next one yet.\"\\\\n\\\\n# '\n", - " \"Documents:\\\\nMan City\\\\'s Dias ruled out for \\\\'three or four \"\n", - " \"weeks\\\\'\\\\n\\\\nRuben Dias has won 10 major trophies during his time at \"\n", - " 'Manchester City\\\\n\\\\nManchester City have suffered a fresh injury blow with '\n", - " 'manager Pep Guardiola confirming Portugal central defender Ruben Dias has '\n", - " 'been ruled out for \"three or four weeks\" with a muscle injury. Dias, who '\n", - " \"suffered the injury in Saturday\\\\'s 2-1 defeat by Manchester United, will \"\n", - " 'miss the entire festive programme and potentially the FA Cup third-round tie '\n", - " 'with Salford on 11 January. The 27-year-old also faces a battle to be fit '\n", - " \"for City\\\\'s crucial Champions League trip to Paris St-Germain on 22 \"\n", - " 'January. Dias has already missed seven games with a calf injury this season, '\n", - " 'adding to a defensive injury list that has seen John Stones, Nathan Ake, '\n", - " 'Manuel Akanji and Kyle Walker all ruled out at various points, while Ballon '\n", - " \"d\\\\'Or winner Rodri will miss the remainder of the domestic season after \"\n", - " 'suffering a cruciate knee ligament injury. \"It\\\\\\'s a muscular problem and '\n", - " 'he will be out for three to four weeks,\" said Guardiola. \"After 75 minutes '\n", - " \"against United he felt something. But he\\\\'s so strong and wanted to stay on \"\n", - " 'the pitch. Now he\\\\\\'s injured.\" Guardiola confirmed Stones, Akanji and '\n", - " 'midfielder Mateo Kovacic have all trained this week and could feature at '\n", - " 'Aston Villa on Saturday (12:30 GMT), but said goalkeeper Ederson was \"a '\n", - " 'doubt\" with an unspecified problem. \"Ederson has been struggling with some '\n", - " 'niggles in his leg, he doesn\\\\\\'t feel completely fine,\" said Guardiola. '\n", - " '\"Ederson is so important for us.\" Amid City\\\\\\'s current run of one win in '\n", - " \"11 games, surprise has been expressed about Guardiola\\\\'s use of youngsters \"\n", - " \"James McAtee and Nico O\\\\'Reilly. City made a point of keeping both players \"\n", - " 'despite numerous loan options. Yet McAtee has made just two substitute '\n", - " 'appearances - coming on in the last minute on both occasions - while '\n", - " \"O\\\\'Reilly is yet to make his league debut. But it seems they will stay at \"\n", - " 'the club for the second half of the season, with Guardiola replying \"I '\n", - " 'don\\\\\\'t think so\" when asked if players might leave during the January '\n", - " 'transfer window. The Spaniard said he is \"not a big fan\" of buying players '\n", - " 'in January but it is \"possible\" City will look to sign someone because \"the '\n", - " 'circumstances of this season have been special\".\\\\n\\\\nGuardiola\\\\\\'s mood '\n", - " 'was so downbeat in the immediate aftermath of the United defeat it was easy '\n", - " 'to imagine he might conclude he was no longer capable of doing the job. He '\n", - " 'gave his players a couple of days off afterwards and was brighter when he '\n", - " 'spoke to journalists in his scheduled briefing before the Villa trip. '\n", - " '\"We\\\\\\'d just finished a game that we lost in the circumstances and I was '\n", - " 'not happy,\" he said. \"I try to be honest about the feelings of my teams. We '\n", - " 'fell down six times [number of Premier League games without a win], we have '\n", - " 'to stand up seven. There is no alternative. \"I\\\\\\'m fine. I\\\\\\'m a normal '\n", - " 'person with feelings like all of us. When the situation is going well we are '\n", - " \"better but it\\\\'s normal. I would not go to the press conference if we were \"\n", - " '1-0 up and expressing something that I didn\\\\\\'t feel.\" Former Villa forward '\n", - " 'Jack Grealish has not scored for City in over a year but Guardiola pointed '\n", - " 'out he is not the only attacking player struggling this season. He added: '\n", - " '\"We are struggling to create a little bit up front, but always I am '\n", - " 'optimistic about my players that they are going to turn and perform '\n", - " 'well.\"\\\\n\\\\n# Documents:\\\\n\\\\\\'We have to find a way\\\\\\' - Guardiola vows to '\n", - " 'end relegation form\\\\n\\\\nThis video can not be played To play this video you '\n", - " \"need to enable JavaScript in your browser. \\\\'Worrying\\\\' and \"\n", - " \"\\\\'staggering\\\\' - Why do Manchester City keep conceding?\\\\n\\\\nManchester \"\n", - " 'City are currently in relegation form and there is little sign of it ending. '\n", - " \"Saturday\\\\'s 2-1 defeat at Aston Villa left them joint bottom of the form \"\n", - " 'table over the past eight games with just Southampton for company. Saints, '\n", - " 'at the foot of the Premier League, have the same number of points, four, as '\n", - " 'City over their past eight matches having won one, drawn one and lost six - '\n", - " 'the same record as the floundering champions. And if Southampton - who '\n", - " 'appointed Ivan Juric as their new manager on Saturday - get at least a point '\n", - " 'at Fulham on Sunday, City will be on the worst run in the division. Even '\n", - " \"Wolves, who sacked boss Gary O\\\\'Neil last Sunday and replaced him with \"\n", - " 'Vitor Pereira, have earned double the number of points during the same '\n", - " 'period having played a game fewer. They are damning statistics for Pep '\n", - " 'Guardiola, even if he does have some mitigating circumstances with injuries '\n", - " 'to Ederson, Nathan Ake and Ruben Dias - who all missed the loss at Villa '\n", - " 'Park - and the long-term loss of midfield powerhouse Rodri. Guardiola was '\n", - " \"happy with Saturday\\\\'s performance, despite defeat in Birmingham, but there \"\n", - " 'is little solace to take at slipping further out of the title race. He may '\n", - " 'have needed to field a half-fit Manuel Akanji and John Stones at Villa Park '\n", - " 'but that does not account for City looking a shadow of their former selves. '\n", - " 'That does not justify the error Josko Gvardiol made to gift Jhon Duran a '\n", - " 'golden chance inside the first 20 seconds, or £100m man Jack Grealish again '\n", - " 'failing to have an impact on a game. There may be legitimate reasons for '\n", - " \"City\\\\'s drop off, whether that be injuries, mental fatigue or just simply a \"\n", - " 'team coming to the end of its lifecycle, but their form, which has plunged '\n", - " 'off a cliff edge, would have been unthinkable as they strolled to a fourth '\n", - " 'straight title last season. \"The worrying thing is the number of goals '\n", - " 'conceded,\" said ex-England captain Alan Shearer on BBC Match of the Day. '\n", - " '\"The number of times they were opened up because of the lack of protection '\n", - " 'and legs in midfield was staggering. There are so many things that are wrong '\n", - " 'at this moment in time.\"\\\\n\\\\nThis video can not be played To play this '\n", - " \"video you need to enable JavaScript in your browser. Man City \\\\'have to \"\n", - " \"find a way\\\\' to return to form - Guardiola\\\\n\\\\nAfterwards Guardiola was \"\n", - " 'calm, so much so it was difficult to hear him in the news conference, a '\n", - " 'contrast to the frustrated figure he cut on the touchline. He said: \"It '\n", - " 'depends on us. The solution is bring the players back. We have just one '\n", - " 'central defender fit, that is difficult. We are going to try next game - '\n", - " 'another opportunity and we don\\\\\\'t think much further than that. \"Of course '\n", - " \"there are more reasons. We concede the goals we don\\\\'t concede in the past, \"\n", - " \"we [don\\\\'t] score the goals we score in the past. Football is not just one \"\n", - " 'reason. There are a lot of little factors. \"Last season we won the Premier '\n", - " 'League, but we came here and lost. We have to think positive and I have '\n", - " 'incredible trust in the guys. Some of them have incredible pride and desire '\n", - " 'to do it. We have to find a way, step by step, sooner or later to find a way '\n", - " 'back.\" Villa boss Unai Emery highlighted City\\\\\\'s frailties, saying he felt '\n", - " 'Villa could seize on the visitors\\\\\\' lack of belief. \"Manchester City are a '\n", - " 'little bit under the confidence they have normally,\" he said. \"The second '\n", - " 'half was different, we dominated and we scored. Through those circumstances '\n", - " 'they were feeling worse than even in the first half.\"\\\\n\\\\nErling Haaland '\n", - " 'had one touch in the Villa box\\\\n\\\\nThere are chinks in the armour never '\n", - " 'seen before at City under Guardiola and Erling Haaland conceded belief '\n", - " 'within the squad is low. He told TNT after the game: \"Of course, [confidence '\n", - " 'levels are] not the best. We know how important confidence is and you can '\n", - " 'see that it affects every human being. That is how it is, we have to '\n", - " 'continue and stay positive even though it is difficult.\" Haaland, with 76 '\n", - " 'goals in 83 Premier League appearances since joining City from Borussia '\n", - " 'Dortmund in 2022, had one shot and one touch in the Villa box. His 18 '\n", - " 'touches in the whole game were the lowest of all starting players and he has '\n", - " 'been self critical, despite scoring 13 goals in the top flight this season. '\n", - " \"Over City\\\\'s last eight games he has netted just twice though, but \"\n", - " 'Guardiola refused to criticise his star striker. He said: \"Without him we '\n", - " \"will be even worse but I like the players feeling that way. I don\\\\'t agree \"\n", - " 'with Erling. He needs to have the balls delivered in the right spots but he '\n", - " 'will fight for the next one.\"\\', '\n", - " \"tool_call_id='call_JiFf7HC9upNSFvLznTB0VI1w', \"\n", - " 'timestamp=datetime.datetime(2025, 11, 7, 7, 48, 36, 507812, '\n", + " \"new, unforgiving territory.\\\\n\\\\n# Documents:\\\\nMan City\\\\'s Dias ruled out \"\n", + " \"for \\\\'three or four weeks\\\\'\\\\n\\\\nRuben Dias has won 10 major trophies \"\n", + " 'during his time at Manchester City\\\\n\\\\nManchester City have suffered a '\n", + " 'fresh injury blow with manager Pep Guardiola confirming Portugal central '\n", + " 'defender Ruben Dias has been ruled out for \"three or four weeks\" with a '\n", + " \"muscle injury. Dias, who suffered the injury in Saturday\\\\'s 2-1 defeat by \"\n", + " 'Manchester United, will miss the entire festive programme and potentially '\n", + " 'the FA Cup third-round tie with Salford on 11 January. The 27-year-old also '\n", + " \"faces a battle to be fit for City\\\\'s crucial Champions League trip to Paris \"\n", + " 'St-Germain on 22 January. Dias has already missed seven games with a calf '\n", + " 'injury this season, adding to a defensive injury list that has seen John '\n", + " 'Stones, Nathan Ake, Manuel Akanji and Kyle Walker all ruled out at various '\n", + " \"points, while Ballon d\\\\'Or winner Rodri will miss the remainder of the \"\n", + " 'domestic season after suffering a cruciate knee ligament injury. \"It\\\\\\'s a '\n", + " 'muscular problem and he will be out for three to four weeks,\" said '\n", + " 'Guardiola. \"After 75 minutes against United he felt something. But he\\\\\\'s '\n", + " 'so strong and wanted to stay on the pitch. Now he\\\\\\'s injured.\" Guardiola '\n", + " 'confirmed Stones, Akanji and midfielder Mateo Kovacic have all trained this '\n", + " 'week and could feature at Aston Villa on Saturday (12:30 GMT), but said '\n", + " 'goalkeeper Ederson was \"a doubt\" with an unspecified problem. \"Ederson has '\n", + " \"been struggling with some niggles in his leg, he doesn\\\\'t feel completely \"\n", + " 'fine,\" said Guardiola. \"Ederson is so important for us.\" Amid City\\\\\\'s '\n", + " 'current run of one win in 11 games, surprise has been expressed about '\n", + " \"Guardiola\\\\'s use of youngsters James McAtee and Nico O\\\\'Reilly. City made \"\n", + " 'a point of keeping both players despite numerous loan options. Yet McAtee '\n", + " 'has made just two substitute appearances - coming on in the last minute on '\n", + " \"both occasions - while O\\\\'Reilly is yet to make his league debut. But it \"\n", + " 'seems they will stay at the club for the second half of the season, with '\n", + " 'Guardiola replying \"I don\\\\\\'t think so\" when asked if players might leave '\n", + " 'during the January transfer window. The Spaniard said he is \"not a big fan\" '\n", + " 'of buying players in January but it is \"possible\" City will look to sign '\n", + " 'someone because \"the circumstances of this season have been '\n", + " 'special\".\\\\n\\\\nGuardiola\\\\\\'s mood was so downbeat in the immediate '\n", + " 'aftermath of the United defeat it was easy to imagine he might conclude he '\n", + " 'was no longer capable of doing the job. He gave his players a couple of days '\n", + " 'off afterwards and was brighter when he spoke to journalists in his '\n", + " 'scheduled briefing before the Villa trip. \"We\\\\\\'d just finished a game that '\n", + " 'we lost in the circumstances and I was not happy,\" he said. \"I try to be '\n", + " 'honest about the feelings of my teams. We fell down six times [number of '\n", + " 'Premier League games without a win], we have to stand up seven. There is no '\n", + " 'alternative. \"I\\\\\\'m fine. I\\\\\\'m a normal person with feelings like all of '\n", + " \"us. When the situation is going well we are better but it\\\\'s normal. I \"\n", + " 'would not go to the press conference if we were 1-0 up and expressing '\n", + " 'something that I didn\\\\\\'t feel.\" Former Villa forward Jack Grealish has not '\n", + " 'scored for City in over a year but Guardiola pointed out he is not the only '\n", + " 'attacking player struggling this season. He added: \"We are struggling to '\n", + " 'create a little bit up front, but always I am optimistic about my players '\n", + " 'that they are going to turn and perform well.\"\\', '\n", + " \"tool_call_id='call_W3weSbPGeZzkCmQh0p3nYVb6', \"\n", + " 'timestamp=datetime.datetime(2025, 11, 7, 8, 45, 9, 341493, '\n", " \"tzinfo=datetime.timezone.utc), part_kind='tool-return')], instructions=None, \"\n", " \"kind='request')\")\n", "==================================================\n", "Step 4:\n", - "(\"ModelResponse(parts=[TextPart(content='Manchester City manager Pep Guardiola \"\n", - " \"has expressed significant concern over the team\\\\'s current form, noting \"\n", - " 'that the situation is affecting him personally and professionally. Guardiola '\n", - " 'admitted that recent poor performances have impacted his sleep and diet, '\n", - " 'referring to his mental state as \"ugly\" due to the team\\\\\\'s downturn. He '\n", - " 'emphasized the need for better defense and fewer mistakes as crucial areas '\n", - " 'for improvement.\\\\n\\\\nGuardiola described this period as possibly the most '\n", - " 'difficult in his managerial career, acknowledging that he has not been able '\n", - " 'to find solutions to their struggles. His analysis was self-critical, '\n", - " 'admitting, \"I am not good enough. I am the boss. I am the manager. I have to '\n", - " 'find solutions and so far I haven\\\\\\'t.\"\\\\n\\\\nThe team is currently '\n", - " 'experiencing one of its worst runs, having only won one out of their last '\n", - " 'ten games. This has placed them in an unfamiliar position, both in the '\n", - " 'Premier League and in European competitions. Guardiola faces additional '\n", - " 'challenges due to injuries and a need for what he termed a \"major '\n", - " 'rebuild.\"\\\\n\\\\nDespite these issues, Guardiola remains hopeful, stating that '\n", - " 'the team needs to \"survive\" this season, with aspirations for a record fifth '\n", - " 'successive Premier League title seemingly distant as they trail league '\n", - " 'leaders Liverpool significantly. He has insisted that City must \"find a way, '\n", - " 'step by step\" to return to form.\\\\n\\\\nGuardiola also faced a dramatic '\n", - " 'Manchester derby loss, resulting in a further blow to the team and an '\n", - " 'introspective moment for him as he commits to resolving the situation moving '\n", - " \"forward.', part_kind='text')], model_name='gpt-4o-2024-08-06', \"\n", - " 'timestamp=datetime.datetime(2025, 11, 7, 7, 48, 37, '\n", + "(\"ModelResponse(parts=[TextPart(content='Pep Guardiola, the manager of \"\n", + " 'Manchester City, has been experiencing an unprecedented period of poor form '\n", + " 'with his team. This downturn has significantly impacted him personally and '\n", + " 'professionally. Guardiola admitted that this challenging period has affected '\n", + " 'his sleep and diet, stating his mental state as \"ugly\" while enduring the '\n", + " 'worst sequence of results in his managerial career. Despite these '\n", + " 'challenges, he expressed that he remains \"fine\" and committed to addressing '\n", + " 'the issues facing the team.\\\\n\\\\nGuardiola criticized himself publicly, '\n", + " 'stating, \"I am not good enough. I am the boss. I am the manager. I have to '\n", + " 'find solutions and so far I haven\\\\\\'t. That\\\\\\'s the reality.\" These '\n", + " \"statements highlight his struggles in finding solutions for the team\\\\'s \"\n", + " \"current problems.\\\\n\\\\nConcerns about the team\\\\'s defensive performance and \"\n", + " 'mistakes at both ends of the pitch have been highlighted as key areas for '\n", + " 'improvement. The absence of key players due to injuries, including '\n", + " 'midfielder Rodri, has compounded the situation, leading to a lack of '\n", + " 'consistency and stability in the team.\\\\n\\\\nInternally, Guardiola has been '\n", + " 'working on a possible long-term rebuild of the team, aiming to replace '\n", + " 'several regular starters to address the core issues. He plans to use the '\n", + " 'time remaining on his contract to prepare a new Manchester City, potentially '\n", + " 'integrating new talents and addressing strategic weaknesses.\\\\n\\\\nThough he '\n", + " 'has not criticized his players personally, Guardiola has given them time off '\n", + " 'to refresh and reset mentally, recognizing the intense schedule and '\n", + " \"pressures they have been under. Despite the downturn, Guardiola\\\\'s \"\n", + " 'commitment to the team remains strong, with plans for future changes '\n", + " \"reflecting a realistic assessment of the challenges faced by the club.', \"\n", + " \"part_kind='text')], model_name='gpt-4o-2024-08-06', \"\n", + " 'timestamp=datetime.datetime(2025, 11, 7, 8, 45, 9, '\n", " \"tzinfo=datetime.timezone.utc), kind='response')\")\n", "==================================================\n" ]