* Support hallucination score

* Support hallucination score, deepeval part
* Support hallucination score, selfcheckgpt part
* Add workflow

---------

Signed-off-by:  Kang Wenjing <[email protected]>
Signed-off-by:  yzheng124 <[email protected]>
Co-authored-by: yzheng124 <[email protected]>

Author: WenjingKangIntel

.github/reusable-steps/categorize-projects/action.yml

@@ -16,6 +16,8 @@ outputs:
     value: ${{ steps.group-subprojects.outputs.qt }}
   js:
     value: ${{ steps.group-subprojects.outputs.js }}
+  unittest:
+    value: ${{ steps.group-subprojects.outputs.unittest }}
 
 runs:
   using: 'composite'
@@ -42,6 +44,8 @@ runs:
             qt+=("$dir")
           elif [ -f "$dir/main.py" ] && grep -q -- "--stream" "$dir/main.py"; then
             webcam+=("$dir")
+          elif [ -d "$dir/test" ]; then
+            unittest+=("$dir/test")
           else
             python+=("$dir")
           fi
@@ -53,13 +57,15 @@ runs:
         webcam_json=$(printf '%s\n' "${webcam[@]}" | jq -R -s -c 'split("\n") | map(select(length > 0))')
         qt_json=$(printf '%s\n' "${qt[@]}" | jq -R -s -c 'split("\n") | map(select(length > 0))')
         js_json=$(printf '%s\n' "${js[@]}" | jq -R -s -c 'split("\n") | map(select(length > 0))')
+        unittest_json=$(printf '%s\n' "${unittest_json[@]}" | jq -R -s -c 'split("\n") | map(select(length > 0))')
 
         echo "notebook=$notebook_json" >> $GITHUB_OUTPUT
         echo "python=$python_json" >> $GITHUB_OUTPUT
         echo "gradio=$gradio_json" >> $GITHUB_OUTPUT
         echo "webcam=$webcam_json" >> $GITHUB_OUTPUT
         echo "qt=$qt_json" >> $GITHUB_OUTPUT
         echo "js=$js_json" >> $GITHUB_OUTPUT
+        echo "unittest_json=$unittest_json" >> $GITHUB_OUTPUT
     - name: Print subprojects to test
       shell: bash
       run: |
@@ -69,3 +75,4 @@ runs:
         echo "Webcam subprojects: ${{ steps.group-subprojects.outputs.webcam }}"
         echo "Qt subprojects: ${{ steps.group-subprojects.outputs.qt }}"
         echo "JS subprojects: ${{ steps.group-subprojects.outputs.js }}"
+        echo "Unit test subprojects: ${{ steps.group-subprojects.outputs.unittest }}"

.github/workflows/sanity-check-demos.yml

@@ -119,3 +119,44 @@ jobs:
           command: npm start
           project: ${{ matrix.subproject }}
           timeout: 1m
+
+  unittest:
+    needs: find-subprojects
+    if: ${{ needs.find-subprojects.outputs.unittest != '[]' }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+        python: ["3.10", "3.12"]
+        subproject: ${{ fromJson(needs.find-subprojects.outputs.unittest) }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: ./.github/reusable-steps/setup-os
+      - name: Set up Python ${{ matrix.python }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - uses: ./.github/reusable-steps/setup-python
+        with:
+          python: ${{ matrix.python }}
+          project: ${{ matrix.subproject }}
+      - name: Login to HF
+        shell: bash
+        run: |
+          huggingface-cli login --token ${{ secrets.HF_TOKEN }}
+      - name: Install ollama
+        shell: bash
+        run: |
+          curl -fsSL https://ollama.com/install.sh | sh
+      - name: Start ollama and setup deepeval to use ollama
+        shell: bash
+        run: |
+          ollama serve &
+          ollama pull deepseek-r1 &
+          deepeval set-ollama deepseek-r1
+      - uses: ./.github/reusable-steps/timeouted-action
+        with:
+          script: python test.py
+          project: ${{ matrix.subproject }}
+          timeout: 5h

demos/virtual_ai_assistant_demo/requirements.txt

@@ -20,5 +20,7 @@ torch==2.5.1
 transformers==4.48.3
 pymupdf==1.24.10
 pyyaml==6.0.1
+selfcheckgpt==0.1.7
+deepeval==2.4.9
 
 gradio==5.12.0

demos/virtual_ai_assistant_demo/test/README.md

@@ -0,0 +1,50 @@
+# How to select hallucination computation algorithm
+
+Currently, two methods are availble: [deepeval](#use-deepeval-to-compute-hallucination-score) and [selfcheckgpt](#use-selfcheckgpt-to-compute-hallucination-score).
+
+If you have an evaluation dataset (i.e. both question and correct answer), you can choose [deepeval](#use-deepeval-to-compute-hallucination-score). However, if you do not have a labeled dataset, you can choose [selfcheckgpt](#use-selfcheckgpt-to-compute-hallucination-score). It will compute hallucination score based on the output consistency.
+
+# Use deepeval to compute hallucination score
+## Prerequisite libraries
+1. [deepeval](https://github.com/confident-ai/deepeval)
+2. [Ollama](https://github.com/ollama/ollama/blob/main/README.md)
+
+## How to set up
+1. Install deepeval:
+    ```
+    pip install -U deepeval
+    ```
+2. Install Ollama:
+    Please refer to [ollama](https://github.com/ollama/ollama/blob/main/README.md#ollama)
+
+3. Run Ollama, taking `deepseek-r1` as an example:
+    ```
+    ollama run deepseek-r1
+    ```
+4. Set deepeval to use Ollama for evaluation:
+    ```
+    deepeval set-ollama deepseek-r1
+    ```
+
+## How to run the test
+```
+python test.py --personality /path/to/personality.yaml --check_type deepeval
+```
+
+## More to read
+[deepeval hallucination](https://docs.confident-ai.com/docs/metrics-hallucination)
+
+# Use selfcheckgpt to compute hallucination score
+## Prerequisite libraries
+1. [selfcheckgpt](https://github.com/potsawee/selfcheckgpt)
+
+## How to set up and run the test
+1. Install deepeval:
+    ```
+    pip install selfcheckgpt==0.1.7
+    ```
+
+2. Run test
+    ```
+    python test.py --personality /path/to/personality.yaml --check_type selfcheckgpt
+    ```

demos/virtual_ai_assistant_demo/test/bartender_personality.txt

@@ -0,0 +1,20 @@
+Can you suggest some popular fruit-based drinks that are healthy and refreshing?
+Can you suggest some recipes using your favorite juices or ingredients?
+Can you suggest some refreshing drinks with watermelon or lime?
+Can you suggest some tropical juices or smoothies with kiwi or banana?
+What are the ingredients in a classic Martini?
+What are some popular drinks that use pomegranate juice?
+Can you suggest a cocktail that uses honey?
+What are the ingredients in a classic Daiquiri?
+Can you recommend a cocktail that uses apple cider?
+What are some popular drinks that use cranberry juice?
+Can you suggest a cocktail that uses chocolate?
+What are the ingredients in a classic Negroni?
+Can you recommend a cocktail that uses almond milk?
+What are some popular drinks that use grapefruit juice?
+Can you suggest a cocktail that uses lavender?
+What are the ingredients in a classic Pina Colada?
+Can you recommend a cocktail that uses maple syrup?
+What are some popular drinks that use lemon or lime juice?
+Can you suggest a cocktail that uses cinnamon?
+What are the ingredients in a classic Bloody Mary?

demos/virtual_ai_assistant_demo/test/culinara_personality.txt

@@ -0,0 +1,20 @@
+I'm planning to cook a classic spaghetti carbonara. What ingredients do I need?
+Can I substitute pancetta with bacon in my carbonara?
+I'm planning to make a vegan lasagna. What can I use instead of ricotta cheese?
+How long should I bake my lasagna for the best results?
+I'm making a chicken curry. What spices should I use for an authentic flavor?
+Can I use coconut milk instead of cream in my chicken curry?
+I'm planning to bake a chocolate cake. What type of cocoa powder is best?
+Can I use almond flour instead of all-purpose flour in my cake?
+I'm making a Caesar salad. What ingredients are essential for the dressing?
+Can I use Greek yogurt instead of mayonnaise in my Caesar dressing?
+I'm planning to cook a beef stew. What cut of beef is best for stewing?
+Can I use red wine instead of beef broth in my stew?
+I'm planning to cook a seafood paella. What types of seafood are best to use?
+Can I use brown rice instead of white rice in my paella?
+How do I achieve the perfect socarrat (crispy bottom) in my paella?
+I'm making a vegetarian chili. What beans are best to use?
+Can I add quinoa to my chili for extra protein?
+I'm planning to bake a batch of cookies. What type of sugar should I use?
+Can I substitute butter with coconut oil in my cookies?
+I'm making a Greek salad. What ingredients are essential?