A guide on RAG Pipeline with kerasHub models #2150
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @laxmareddyp, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request introduces a comprehensive guide demonstrating a Retrieval-Augmented Generation (RAG) pipeline for medical image analysis, specifically focusing on brain MRI scans. The guide showcases how to leverage KerasHub models, including a Vision Transformer (ViT) for image feature extraction and a Gemma3 Causal Language Model for report generation. It highlights the benefits of RAG by comparing its context-aware report generation with a direct vision-language model approach, emphasizing the enhanced accuracy and contextual relevance provided by RAG.
Highlights
- New Guide for RAG Pipeline: Added a comprehensive guide demonstrating a Retrieval-Augmented Generation (RAG) pipeline for medical image analysis using KerasHub models.
- Model Integration: Showcases the integration of a Vision Transformer (ViT) for image feature extraction and a Gemma3 Causal Language Model for report generation, both sourced from KerasHub.
- Medical Imaging Application: Applies the RAG pipeline to brain MRI images from the OASIS dataset, demonstrating image processing, feature extraction, and similarity search for relevant medical reports.
- Contextual Report Generation: Implements a mechanism to retrieve the most similar medical report based on image features and uses it as context to generate more accurate and detailed radiology reports.
- Performance Comparison: Compares the RAG-based report generation approach with a direct Vision-Language Model (VLM) generation method, highlighting the benefits of contextual retrieval.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request adds a new guide demonstrating a RAG pipeline with KerasHub models. The guide is provided in Jupyter Notebook, Python script, and Markdown formats. The concept is great and the guide is well-structured. However, I've found several critical issues related to model selection and implementation logic that need to be addressed for the guide to be correct and functional. These include using a text-only model for a vision-language task, incorrect feature extraction logic, and bugs that will lead to runtime errors. I've also included some suggestions for improving code quality and robustness.
|
@laxmareddyp If this supposed to be a general guidline for using RAG in keras, I think instead of focusing on a particular dataset and modality (vision/image), its better to bring or cover how to use well-known libraries (ml-library agnostic) with keras, i.e. langchain, langgraph, chroma, etc. |
| 4. Generate comprehensive radiology reports using retrieved contex | ||
| 5. Compare RAG approach with direct vision-language model generation | ||
|
|
||
| This pipeline demonstrates how to build a sophisticated medical AI system that can: |
There was a problem hiding this comment.
this part id repetitive
|
|
||
| """ | ||
| ## Kaggle Credentials Setup | ||
|
|
There was a problem hiding this comment.
also add details on adding credentials as colab secrets.
| """ | ||
| ## RAG Pipeline | ||
|
|
||
| Implements the Retrieval-Augmented Generation (RAG) pipeline: |
There was a problem hiding this comment.
explain what RAG is - what components make it RAG and how that will be achieved
|
|
||
|
|
||
| """ | ||
| ## Vision-Language Model (Direct Approach) |
There was a problem hiding this comment.
Add a few sentences about - to compare the outputs of the RAG pipeline we just built. lets try how a generic VLM performs, etc ..
the guide should explain the flow of what and why
| This section loads models, prepares data, runs the RAG pipeline, and compares RAG with direct VLM generation. | ||
| """ | ||
|
|
||
| if __name__ == "__main__": |
There was a problem hiding this comment.
insted of running everything here under main. Lets execute it in the above cells one by one.
|
|
||
| """ | ||
| ## Welcome to Your RAG Adventure! | ||
|
|
There was a problem hiding this comment.
The language on this guide is more "fun" that we usually publish. I like it- but I would check with @hertschuh once!
|
Closing this PR due to accidentally one file is deleted, Will create new PR with same changes. |
No description provided.