From 2e648f2ca2c39bc58ebe06d3224081a79525ab0c Mon Sep 17 00:00:00 2001 From: Amna Mubashar Date: Tue, 8 Jul 2025 14:42:24 +0200 Subject: [PATCH 1/4] Migrate to GoogleGenAI --- notebooks/vertexai-gemini-examples.ipynb | 143 ++++------------------- 1 file changed, 21 insertions(+), 122 deletions(-) diff --git a/notebooks/vertexai-gemini-examples.ipynb b/notebooks/vertexai-gemini-examples.ipynb index ab44854..35b7c3b 100644 --- a/notebooks/vertexai-gemini-examples.ipynb +++ b/notebooks/vertexai-gemini-examples.ipynb @@ -28,9 +28,7 @@ "\n", "As a prerequisite, you need to have a Google Cloud Project set up that has access to Gemini. Following that, you'll only need to authenticate yourself in this Colab.\n", "\n", - "First thing first we need to install our dependencies.\n", - "\n", - "(You can ignore the pip dependency error for `cohere` and `tiktoken`, that's irrelevant for our purposes.)" + "First thing first we need to install our dependencies.\n" ] }, { @@ -41,7 +39,7 @@ }, "outputs": [], "source": [ - "!pip install --upgrade haystack-ai google-vertex-haystack trafilatura" + "!pip install --upgrade haystack-ai google-genai-haystack trafilatura" ] }, { @@ -108,7 +106,7 @@ "source": [ "### Answer Questions\n", "\n", - "Now that we setup everything we can create an instance of our Gemini component." + "Now that we setup everything we can create an instance of our `GoogleGenAIChatGenerator`. This component supports both Gemini and Vertex AI. For this demo, we will set `api=\"vertex\"`, and pass our project_id as vertex_ai_project. " ] }, { @@ -119,9 +117,9 @@ }, "outputs": [], "source": [ - "from haystack_integrations.components.generators.google_vertex import VertexAIGeminiGenerator\n", + "from haystack_integrations.components.generators.googlegenai import GoogleGenAIChatGenerator\n", "\n", - "gemini = VertexAIGeminiGenerator(model=\"gemini-2.0-flash\", project_id=project_id)" + "gemini = GoogleGenAIChatGenerator(model=\"gemini-2.0-flash\", api=\"vertex\", vertex_ai_project=project_id, vertex_ai_location=\"europe-west1\")" ] }, { @@ -132,7 +130,7 @@ "source": [ "Let's start by asking something simple.\n", "\n", - "This component expects a list of `Parts` as input to the `run()` method. Parts can be anything from a message, to images, or even function calls. Here are the docstrings from the source code for the most up-to-date reference we could find [here.](https://github.com/googleapis/python-aiplatform/blob/5f6ad8df5a08e78a121a72a21e21d95abb072e58/vertexai/generative_models/_generative_models.py#L1427-L1446)" + "This component expects a list of `ChatMessage` as input to the `run()` method. You can pass text or function calls through the messages." ] }, { @@ -143,92 +141,20 @@ }, "outputs": [], "source": [ - "result = gemini.run(parts = [\"What is the most interesting thing you know?\"])\n", - "for answer in result[\"replies\"]:\n", - " print(answer)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "H3pU-t_7mPyH" - }, - "source": [ - "### Answer Questions about Images" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "U-wi810nOGZW" - }, - "source": [ - "Let's try something a bit different! `gemini-2.0-flash` can also work with images, let's see if we can have it answer questions about some robots 👇\n", - "\n", - "We're going to download some images for this example. 🤖" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "sGqQ8hkhOGZW" - }, - "outputs": [], - "source": [ - "import requests\n", - "from haystack.dataclasses.byte_stream import ByteStream\n", - "\n", - "URLS = [\n", - " \"https://raw.githubusercontent.com/silvanocerza/robots/main/robot1.jpg\",\n", - " \"https://raw.githubusercontent.com/silvanocerza/robots/main/robot2.jpg\",\n", - " \"https://raw.githubusercontent.com/silvanocerza/robots/main/robot3.jpg\",\n", - " \"https://raw.githubusercontent.com/silvanocerza/robots/main/robot4.jpg\"\n", - "]\n", - "images = [\n", - " ByteStream(data=requests.get(url).content, mime_type=\"image/jpeg\")\n", - " for url in URLS\n", - "]" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "tQsdovv2pWzh" - }, - "source": [ - "Next, let's run the `VertexAIGeminiGenerator` component on it's own." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "6jSBGEkmpPsN" - }, - "outputs": [], - "source": [ - "result = gemini.run(parts = [\"What can you tell me about this robots?\", *images])\n", + "from haystack.dataclasses import ChatMessage\n", + "messages = [ChatMessage.from_user(\"What is the most interesting thing you know?\")]\n", + "result = gemini.run(messages = messages)\n", "for answer in result[\"replies\"]:\n", " print(answer)" ] }, - { - "cell_type": "markdown", - "metadata": { - "id": "_Qp9tGaNOGZW" - }, - "source": [ - "Did Gemini recognize all its friends? 👀" - ] - }, { "cell_type": "markdown", "metadata": { "id": "gLIIQ4PZmX-H" }, "source": [ - "## Function Calling with `gemini-1.5-pro`" + "## Function Calling with `gemini-2.0-flash`" ] }, { @@ -237,8 +163,8 @@ "id": "iCiSz840mfME" }, "source": [ - "Note that [Google recommends upgrading](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/model-versions) from gemini-1.5-pro to gemini-2.0-flash.\n", - "With `gemini-1.5-pro`, we can also start introducing function calling!\n", + "\n", + "With `gemini-2.0-flash`, we can also use function calling!\n", "So let's see how we can do that 👇\n", "\n", "Let's see if we can build a system that can run a `get_current_weather` function, based on a question asked in natural language.\n", @@ -269,32 +195,6 @@ "weather_tool = create_tool_from_function(get_current_weather)" ] }, - { - "cell_type": "markdown", - "metadata": { - "id": "TlbWFohMOGZW" - }, - "source": [ - "We're also going to chat with Gemini this time, we're going to use another class for this.\n", - "\n", - "We also need the Gemini Pro model to use functions, Gemini Pro Vision doesn't support functions.\n", - "\n", - "Let's create a `VertexAIGeminiChatGenerator`" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "uPIPj6HsOGZW" - }, - "outputs": [], - "source": [ - "from haystack_integrations.components.generators.google_vertex import VertexAIGeminiChatGenerator\n", - "\n", - "gemini_chat = VertexAIGeminiChatGenerator(model=\"gemini-1.5-pro\", project_id=project_id, tools=[weather_tool])" - ] - }, { "cell_type": "code", "execution_count": null, @@ -303,10 +203,9 @@ }, "outputs": [], "source": [ - "from haystack.dataclasses import ChatMessage\n", "\n", "user_message = [ChatMessage.from_user(\"What is the temperature in celsius in Berlin?\")]\n", - "replies = gemini_chat.run(messages=user_message)[\"replies\"]\n", + "replies = gemini.run(messages=user_message)[\"replies\"]\n", "print(replies)" ] }, @@ -364,7 +263,7 @@ "id": "fQz9_N46hniU" }, "source": [ - "As a final exercise, let's add the `VertexAIGeminiGenerator` to a full RAG pipeline. In the example below, we are building a RAG pipeline that does question answering on the web, using `gemini-2.0-flash`" + "As a final exercise, let's add the `GoogleGenAIChatGenerator` to a full RAG pipeline. In the example below, we are building a RAG pipeline that does question answering on the web, using `gemini-2.0-flash`" ] }, { @@ -379,16 +278,16 @@ "from haystack.components.converters import HTMLToDocument\n", "from haystack.components.preprocessors import DocumentSplitter\n", "from haystack.components.rankers import TransformersSimilarityRanker\n", - "from haystack.components.builders.prompt_builder import PromptBuilder\n", + "from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder\n", "from haystack import Pipeline\n", "\n", "fetcher = LinkContentFetcher()\n", "converter = HTMLToDocument()\n", "document_splitter = DocumentSplitter(split_by=\"word\", split_length=50)\n", "similarity_ranker = TransformersSimilarityRanker(top_k=3)\n", - "gemini = VertexAIGeminiGenerator(model=\"gemini-2.0-flash\", project_id=project_id)\n", + "gemini = GoogleGenAIChatGenerator(model=\"gemini-2.0-flash\", api=\"vertex\", vertex_ai_project=project_id, vertex_ai_location=\"europe-west1\")\n", "\n", - "prompt_template = \"\"\"\n", + "prompt_template = [ChatMessage.from_user(\"\"\"\n", "According to these documents:\n", "\n", "{% for doc in documents %}\n", @@ -397,8 +296,8 @@ "\n", "Answer the given question: {{question}}\n", "Answer:\n", - "\"\"\"\n", - "prompt_builder = PromptBuilder(template=prompt_template)\n", + "\"\"\")]\n", + "prompt_builder = ChatPromptBuilder(template=prompt_template)\n", "\n", "pipeline = Pipeline()\n", "pipeline.add_component(\"fetcher\", fetcher)\n", @@ -460,7 +359,7 @@ "provenance": [] }, "kernelspec": { - "display_name": "google-vertex-haystack", + "display_name": ".venv", "language": "python", "name": "python3" }, @@ -474,7 +373,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.9" + "version": "3.13.5" } }, "nbformat": 4, From 668221c762c2e49221fda2f3607d3e34ac1f5d28 Mon Sep 17 00:00:00 2001 From: Amna Mubashar Date: Tue, 8 Jul 2025 14:59:32 +0200 Subject: [PATCH 2/4] small fix --- notebooks/vertexai-gemini-examples.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/vertexai-gemini-examples.ipynb b/notebooks/vertexai-gemini-examples.ipynb index 35b7c3b..a8bebeb 100644 --- a/notebooks/vertexai-gemini-examples.ipynb +++ b/notebooks/vertexai-gemini-examples.ipynb @@ -106,7 +106,7 @@ "source": [ "### Answer Questions\n", "\n", - "Now that we setup everything we can create an instance of our `GoogleGenAIChatGenerator`. This component supports both Gemini and Vertex AI. For this demo, we will set `api=\"vertex\"`, and pass our project_id as vertex_ai_project. " + "Now that we setup everything we can create an instance of our `GoogleGenAIChatGenerator`. This component supports both Gemini and Vertex AI. For this demo, we will set `api=\"vertex\"`, and pass our project_id as vertex_ai_project." ] }, { From dbbd7d9148179a3183e7d1c33ff770e367620d1c Mon Sep 17 00:00:00 2001 From: bilgeyucel Date: Tue, 8 Jul 2025 16:03:39 +0200 Subject: [PATCH 3/4] add links --- notebooks/vertexai-gemini-examples.ipynb | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/notebooks/vertexai-gemini-examples.ipynb b/notebooks/vertexai-gemini-examples.ipynb index a8bebeb..58c3035 100644 --- a/notebooks/vertexai-gemini-examples.ipynb +++ b/notebooks/vertexai-gemini-examples.ipynb @@ -28,7 +28,7 @@ "\n", "As a prerequisite, you need to have a Google Cloud Project set up that has access to Gemini. Following that, you'll only need to authenticate yourself in this Colab.\n", "\n", - "First thing first we need to install our dependencies.\n" + "First thing first we need to install our dependencies including [Google Gen AI](https://haystack.deepset.ai/integrations/google-genai) integration:\n" ] }, { @@ -106,7 +106,7 @@ "source": [ "### Answer Questions\n", "\n", - "Now that we setup everything we can create an instance of our `GoogleGenAIChatGenerator`. This component supports both Gemini and Vertex AI. For this demo, we will set `api=\"vertex\"`, and pass our project_id as vertex_ai_project." + "Now that we setup everything we can create an instance of our [`GoogleGenAIChatGenerator`](https://docs.haystack.deepset.ai/docs/googlegenaichatgenerator). This component supports both Gemini and Vertex AI. For this demo, we will set `api=\"vertex\"`, and pass our project_id as vertex_ai_project." ] }, { @@ -346,11 +346,13 @@ "id": "xk2SAGT_l25K" }, "source": [ - "Now you've seen some of what Gemini can do, as well as how to integrate it with Haystack. If you want to learn more:\n", - "- check out the Haystack [docs](https://docs.haystack.deepset.ai/docs) or [tutorials](https://haystack.deepset.ai/tutorials)\n", - "- Try out the [Gemini quickstart colab from Google](https://colab.research.google.com/github/google/generative-ai-docs/blob/main/site/en/tutorials/python_quickstart.ipynb#scrollTo=IqFXdgDFRvlU)\n", - "- Participate in the [Advent of Haystack](https://haystack.deepset.ai/advent-of-haystack)" + "Now you've seen some of what Gemini can do, as well as how to integrate it with Haystack. If you want to learn more, check out the Haystack [docs](https://docs.haystack.deepset.ai/docs) or [tutorials](https://haystack.deepset.ai/tutorials)" ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] } ], "metadata": { From 96e3ad695f6d80fd6d5b5fa49d876a646866d599 Mon Sep 17 00:00:00 2001 From: anakin87 Date: Fri, 8 Aug 2025 18:31:04 +0200 Subject: [PATCH 4/4] improvements + gitignore --- .gitignore | 205 ++++++++++++++++++++++ notebooks/vertexai-gemini-examples.ipynb | 212 +++++++++++++++++------ 2 files changed, 365 insertions(+), 52 deletions(-) create mode 100644 .gitignore diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..13b82e5 --- /dev/null +++ b/.gitignore @@ -0,0 +1,205 @@ +# Byte-compiled / optimized / DLL files +__pycache__/ +*.py[codz] +*$py.class + +# C extensions +*.so + +# Distribution / packaging +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +share/python-wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# PyInstaller +# Usually these files are written by a python script from a template +# before PyInstaller builds the exe, so as to inject date/other infos into it. +*.manifest +*.spec + +# Installer logs +pip-log.txt +pip-delete-this-directory.txt + +# Unit test / coverage reports +htmlcov/ +.tox/ +.nox/ +.coverage +.coverage.* +.cache +nosetests.xml +coverage.xml +*.cover +*.py.cover +.hypothesis/ +.pytest_cache/ +cover/ + +# Translations +*.mo +*.pot + +# Django stuff: +*.log +local_settings.py +db.sqlite3 +db.sqlite3-journal + +# Flask stuff: +instance/ +.webassets-cache + +# Scrapy stuff: +.scrapy + +# Sphinx documentation +docs/_build/ + +# PyBuilder +.pybuilder/ +target/ + +# Jupyter Notebook +.ipynb_checkpoints + +# IPython +profile_default/ +ipython_config.py + +# pyenv +# For a library or package, you might want to ignore these files since the code is +# intended to run in multiple environments; otherwise, check them in: +# .python-version + +# pipenv +# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. +# However, in case of collaboration, if having platform-specific dependencies or dependencies +# having no cross-platform support, pipenv may install dependencies that don't work, or not +# install all needed dependencies. +#Pipfile.lock + +# UV +# Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control. +# This is especially recommended for binary packages to ensure reproducibility, and is more +# commonly ignored for libraries. +#uv.lock + +# poetry +# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. +# This is especially recommended for binary packages to ensure reproducibility, and is more +# commonly ignored for libraries. +# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control +#poetry.lock +#poetry.toml + +# pdm +# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. +# pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python. +# https://pdm-project.org/en/latest/usage/project/#working-with-version-control +#pdm.lock +#pdm.toml +.pdm-python +.pdm-build/ + +# pixi +# Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control. +#pixi.lock +# Pixi creates a virtual environment in the .pixi directory, just like venv module creates one +# in the .venv directory. It is recommended not to include this directory in version control. +.pixi + +# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm +__pypackages__/ + +# Celery stuff +celerybeat-schedule +celerybeat.pid + +# SageMath parsed files +*.sage.py + +# Environments +.env +.envrc +.venv +env/ +venv/ +ENV/ +env.bak/ +venv.bak/ + +# Spyder project settings +.spyderproject +.spyproject + +# Rope project settings +.ropeproject + +# mkdocs documentation +/site + +# mypy +.mypy_cache/ +.dmypy.json +dmypy.json + +# Pyre type checker +.pyre/ + +# pytype static type analyzer +.pytype/ + +# Cython debug symbols +cython_debug/ + +# PyCharm +# JetBrains specific template is maintained in a separate JetBrains.gitignore that can +# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore +# and can be added to the global gitignore or merged into this file. For a more nuclear +# option (not recommended) you can uncomment the following to ignore the entire idea folder. +#.idea/ + +# Abstra +# Abstra is an AI-powered process automation framework. +# Ignore directories containing user credentials, local state, and settings. +# Learn more at https://abstra.io/docs +.abstra/ + +# Visual Studio Code +# Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore +# that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore +# and can be added to the global gitignore or merged into this file. However, if you prefer, +# you could uncomment the following to ignore the entire vscode folder +# .vscode/ + +# Ruff stuff: +.ruff_cache/ + +# PyPI configuration file +.pypirc + +# Marimo +marimo/_static/ +marimo/_lsp/ +__marimo__/ + +# Streamlit +.streamlit/secrets.toml + +.DS_Store \ No newline at end of file diff --git a/notebooks/vertexai-gemini-examples.ipynb b/notebooks/vertexai-gemini-examples.ipynb index 58c3035..a34e948 100644 --- a/notebooks/vertexai-gemini-examples.ipynb +++ b/notebooks/vertexai-gemini-examples.ipynb @@ -10,9 +10,11 @@ "\n", "*by Tuana Celik: [Twitter](https://twitter.com/tuanacelik), [LinkedIn](https://www.linkedin.com/in/tuanacelik/), Tilde Thurium: [Twitter](https://twitter.com/annthurium), [LinkedIn](https://www.linkedin.com/in/annthurium/) and Silvano Cerza: [LinkedIn](https://www.linkedin.com/in/silvanocerza/)*\n", "\n", - "**📚 Check out the [Gemini Models with Google Vertex AI Integration for Haystack](https://haystack.deepset.ai/blog/gemini-models-with-google-vertex-for-haystack) article for a detailed run through of this example.**\n", + "This is a notebook showing how you can use Gemini + Vertex AI with Haystack.\n", + "\n", + "To use Gemini models on the Gemini Developer API with Haystack, check out our [documentation](https://docs.haystack.deepset.ai/docs/googlegenaichatgenerator).\n", + "\n", "\n", - "This is a notebook showing how you can use Gemini with Haystack.\n", "\n", "Gemini is Google's newest model. You can read more about its capabilities [here](https://deepmind.google/technologies/gemini/#capabilities).\n", "\n" @@ -26,7 +28,13 @@ "source": [ "## Install dependencies\n", "\n", - "As a prerequisite, you need to have a Google Cloud Project set up that has access to Gemini. Following that, you'll only need to authenticate yourself in this Colab.\n", + "As a prerequisite, you need to have a Google Cloud Project set up that has access to Vertex AI and Gemini.\n", + "\n", + "Useful resources:\n", + "- [Vertex AI quick start](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)\n", + "- [Gemini API in Vertex AI quickstart](https://cloud.google.com/vertex-ai/generative-ai/docs/start/quickstart)\n", + "\n", + "Following that, you'll only need to authenticate yourself in this Colab.\n", "\n", "First thing first we need to install our dependencies including [Google Gen AI](https://haystack.deepset.ai/integrations/google-genai) integration:\n" ] @@ -39,7 +47,7 @@ }, "outputs": [], "source": [ - "!pip install --upgrade haystack-ai google-genai-haystack trafilatura" + "! pip install haystack-ai google-genai-haystack trafilatura" ] }, { @@ -48,14 +56,12 @@ "id": "tHDiEzI2OGZU" }, "source": [ - "To use Gemini you need to have a Google Cloud Platform account and be logged in using Application Default Credentials (ADCs). For more info see the [official documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc).\n", - "\n", - "Time to login!" + "Let's login using Application Default Credentials (ADCs). For more info see the [official documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc)." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "metadata": { "id": "iKvKRuRXOGZU" }, @@ -95,7 +101,7 @@ "id": "08TA9zAQlqy6" }, "source": [ - "## Use `gemini-2.0-flash`" + "## Use `gemini-2.5-flash`" ] }, { @@ -111,15 +117,15 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 6, "metadata": { "id": "f6Ql3qSlOGZV" }, "outputs": [], "source": [ - "from haystack_integrations.components.generators.googlegenai import GoogleGenAIChatGenerator\n", + "from haystack_integrations.components.generators.google_genai import GoogleGenAIChatGenerator\n", "\n", - "gemini = GoogleGenAIChatGenerator(model=\"gemini-2.0-flash\", api=\"vertex\", vertex_ai_project=project_id, vertex_ai_location=\"europe-west1\")" + "gemini = GoogleGenAIChatGenerator(model=\"gemini-2.5-flash\", api=\"vertex\", vertex_ai_project=project_id, vertex_ai_location=\"europe-west1\")" ] }, { @@ -135,17 +141,90 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 7, "metadata": { - "id": "QbqFt4IiOGZV" + "id": "QbqFt4IiOGZV", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "7e7bcd01-56b9-4727-aa9f-c9558f580c37" }, - "outputs": [], + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "The most interesting thing I know, and one of the most profound mysteries in all of science, is that **about 95% of the universe is made of something we cannot see or directly detect: dark energy and dark matter.**\n", + "\n", + "Imagine if 95% of the world around you was completely invisible and unknown, yet it fundamentally shaped everything you *could* see. That's our current situation with the cosmos.\n", + "\n", + "* **Dark Matter** makes up about 27% of the universe. We know it exists because of its gravitational effects – it holds galaxies together, prevents clusters from flying apart, and influenced the large-scale structure of the early universe. But it doesn't absorb, reflect, or emit light, making it \"dark.\" We don't know what particles it's made of.\n", + "* **Dark Energy** makes up about 68% of the universe. It's an even bigger enigma. We infer its existence because it's responsible for the accelerated expansion of the universe. It's essentially pushing the cosmos apart, overcoming the attractive force of gravity. Its nature is one of the biggest unsolved problems in physics.\n", + "\n", + "This means that all the stars, planets, galaxies, gas, and dust – everything we can observe with telescopes – makes up only about 5% of the universe's total mass-energy content. The vast majority of reality is utterly mysterious, and understanding it is one of the greatest scientific quests of our time. It dictates the fate of the cosmos itself.\n" + ] + } + ], "source": [ "from haystack.dataclasses import ChatMessage\n", + "\n", "messages = [ChatMessage.from_user(\"What is the most interesting thing you know?\")]\n", "result = gemini.run(messages = messages)\n", "for answer in result[\"replies\"]:\n", - " print(answer)" + " print(answer.text)" + ] + }, + { + "cell_type": "markdown", + "source": [ + "### Answer Questions about Images\n", + "\n", + "Let's try something a bit different! `gemini-2.5-flash` can also work with images, let's see if we can have it answer questions about some robots 👇\n", + "\n", + "We're going to download some images for this example. 🤖" + ], + "metadata": { + "id": "VjFgF37tcKB_" + } + }, + { + "cell_type": "code", + "source": [ + "from haystack.dataclasses import ImageContent\n", + "\n", + "urls = [\n", + " \"https://upload.wikimedia.org/wikipedia/en/5/5c/C-3PO_droid.png\",\n", + " \"https://platform.theverge.com/wp-content/uploads/sites/2/chorus/assets/4658579/terminator_endoskeleton_1020.jpg\",\n", + " \"https://upload.wikimedia.org/wikipedia/en/3/39/R2-D2_Droid.png\",\n", + "]\n", + "\n", + "images = [ImageContent.from_url(url) for url in urls]\n", + "\n", + "messages = [ChatMessage.from_user(content_parts=[\"What can you tell me about these robots? Be short and graceful.\", *images])]\n", + "result = gemini.run(messages = messages)\n", + "for answer in result[\"replies\"]:\n", + " print(answer.text)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "HNLuSPXKcUoi", + "outputId": "fdd4df25-8628-4162-9e0e-228ca21f3a69" + }, + "execution_count": 9, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "These are iconic robots from popular culture:\n", + "\n", + "1. **C-3PO:** A refined protocol droid, fluent in countless languages, known for his golden appearance and nervous demeanor.\n", + "2. **T-800 Endoskeleton:** A formidable, relentless combat machine, skeletal and chilling, from a dystopian future.\n", + "3. **R2-D2:** A courageous and resourceful astromech, full of personality, who communicates in beeps and whistles.\n" + ] + } ] }, { @@ -154,7 +233,7 @@ "id": "gLIIQ4PZmX-H" }, "source": [ - "## Function Calling with `gemini-2.0-flash`" + "## Function Calling with `gemini-2.5-flash`" ] }, { @@ -164,7 +243,7 @@ }, "source": [ "\n", - "With `gemini-2.0-flash`, we can also use function calling!\n", + "With `gemini-2.5-flash`, we can also use function calling!\n", "So let's see how we can do that 👇\n", "\n", "Let's see if we can build a system that can run a `get_current_weather` function, based on a question asked in natural language.\n", @@ -176,36 +255,46 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 11, "metadata": { "id": "4Wa_IoDDNg9V" }, "outputs": [], "source": [ "from haystack.components.tools import ToolInvoker\n", - "from haystack.tools import create_tool_from_function\n", + "from haystack.tools import tool\n", "from typing import Annotated\n", "\n", + "@tool\n", "def get_current_weather(\n", " location: Annotated[str, \"The city for which to get the weather, e.g. 'San Francisco'\"] = \"Munich\",\n", " unit: Annotated[str, \"The unit for the temperature, e.g. 'celsius'\"] = \"celsius\",\n", "):\n", - " return {\"weather\": \"sunny\", \"temperature\": 21.8, \"unit\": unit}\n", - "\n", - "weather_tool = create_tool_from_function(get_current_weather)" + " return {\"weather\": \"sunny\", \"temperature\": 21.8, \"unit\": unit}" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 12, "metadata": { - "id": "HD4G61Z0OGZX" + "id": "HD4G61Z0OGZX", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "dd732a1d-cd76-4504-b7f1-e91be9c0bb68" }, - "outputs": [], + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "[ChatMessage(_role=, _content=[TextContent(text=''), ToolCall(tool_name='get_current_weather', arguments={'unit': 'celsius', 'location': 'Berlin'}, id=None)], _name=None, _meta={'model': 'gemini-2.5-flash', 'finish_reason': 'stop', 'usage': {'prompt_tokens': 53, 'completion_tokens': 10, 'total_tokens': 126}})]\n" + ] + } + ], "source": [ - "\n", "user_message = [ChatMessage.from_user(\"What is the temperature in celsius in Berlin?\")]\n", - "replies = gemini.run(messages=user_message)[\"replies\"]\n", + "replies = gemini.run(messages=user_message, tools=[get_current_weather])[\"replies\"]\n", "print(replies)" ] }, @@ -223,20 +312,33 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 13, "metadata": { - "id": "c-DurWKOOSOk" + "id": "c-DurWKOOSOk", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "e43f370d-84c5-4d1c-9e97-89f8d26c3824" }, - "outputs": [], + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "[ChatMessage(_role=, _content=[ToolCallResult(result=\"{'weather': 'sunny', 'temperature': 21.8, 'unit': 'celsius'}\", origin=ToolCall(tool_name='get_current_weather', arguments={'unit': 'celsius', 'location': 'Berlin'}, id=None), error=False)], _name=None, _meta={})]\n", + "The temperature in Berlin is 21.8°C and it's sunny.\n" + ] + } + ], "source": [ - "tool_invoker = ToolInvoker(tools=[weather_tool])\n", + "tool_invoker = ToolInvoker(tools=[get_current_weather])\n", "tool_messages = tool_invoker.run(messages=replies)[\"tool_messages\"]\n", + "print(tool_messages)\n", "\n", "messages = user_message + replies + tool_messages\n", - "print(messages)\n", "\n", - "res = gemini_chat.run(messages = messages)\n", - "res[\"replies\"][0].text" + "res = gemini.run(messages = messages)\n", + "print(res[\"replies\"][0].text)" ] }, { @@ -254,7 +356,7 @@ "id": "Pi37EVlDenPw" }, "source": [ - "## Build a full Retrieval-Augmented Generation Pipeline with `gemini-2.0-flash`" + "## Build a full Retrieval-Augmented Generation Pipeline with `gemini-2.5-flash`" ] }, { @@ -263,7 +365,7 @@ "id": "fQz9_N46hniU" }, "source": [ - "As a final exercise, let's add the `GoogleGenAIChatGenerator` to a full RAG pipeline. In the example below, we are building a RAG pipeline that does question answering on the web, using `gemini-2.0-flash`" + "As a final exercise, let's add the `GoogleGenAIChatGenerator` to a full RAG pipeline. In the example below, we are building a RAG pipeline that does question answering on the web, using `gemini-2.5-flash`" ] }, { @@ -277,15 +379,15 @@ "from haystack.components.fetchers.link_content import LinkContentFetcher\n", "from haystack.components.converters import HTMLToDocument\n", "from haystack.components.preprocessors import DocumentSplitter\n", - "from haystack.components.rankers import TransformersSimilarityRanker\n", + "from haystack.components.rankers import SentenceTransformersSimilarityRanker\n", "from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder\n", "from haystack import Pipeline\n", "\n", "fetcher = LinkContentFetcher()\n", "converter = HTMLToDocument()\n", "document_splitter = DocumentSplitter(split_by=\"word\", split_length=50)\n", - "similarity_ranker = TransformersSimilarityRanker(top_k=3)\n", - "gemini = GoogleGenAIChatGenerator(model=\"gemini-2.0-flash\", api=\"vertex\", vertex_ai_project=project_id, vertex_ai_location=\"europe-west1\")\n", + "similarity_ranker = SentenceTransformersSimilarityRanker(top_k=3)\n", + "gemini = GoogleGenAIChatGenerator(model=\"gemini-2.5-flash\", api=\"vertex\", vertex_ai_project=project_id, vertex_ai_location=\"europe-west1\")\n", "\n", "prompt_template = [ChatMessage.from_user(\"\"\"\n", "According to these documents:\n", @@ -325,19 +427,31 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 17, "metadata": { - "id": "EhEx8xO7jMf9" + "id": "EhEx8xO7jMf9", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "6d09c658-b29b-42c8-a156-a1208d381217" }, - "outputs": [], + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "In Haystack, pipelines are structured as graphs. Specifically, Haystack 1.x pipelines were based on Directed Acyclic Graphs (DAGs). In Haystack 2.0, the \"A\" (acyclic) is being removed from DAG, meaning pipelines can now branch out, join, and cycle back to other components, allowing for more complex graph structures that can retry or loop.\n" + ] + } + ], "source": [ "question = \"What do graphs have to do with Haystack?\"\n", "result = pipeline.run({\"prompt_builder\": {\"question\": question},\n", " \"ranker\": {\"query\": question},\n", " \"fetcher\": {\"urls\": [\"https://haystack.deepset.ai/blog/introducing-haystack-2-beta-and-advent\"]}})\n", "\n", - "for answer in result[\"gemini\"][\"replies\"]:\n", - " print(answer)" + "for message in result[\"gemini\"][\"replies\"]:\n", + " print(message.text)" ] }, { @@ -348,16 +462,10 @@ "source": [ "Now you've seen some of what Gemini can do, as well as how to integrate it with Haystack. If you want to learn more, check out the Haystack [docs](https://docs.haystack.deepset.ai/docs) or [tutorials](https://haystack.deepset.ai/tutorials)" ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] } ], "metadata": { "colab": { - "include_colab_link": true, "provenance": [] }, "kernelspec": { @@ -380,4 +488,4 @@ }, "nbformat": 4, "nbformat_minor": 0 -} +} \ No newline at end of file