Skip to content

RHAIENG-1758- Revise Tests for 2025b Onboarding: PandocMissing #2677

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+169 −54
Open

RHAIENG-1758- Revise Tests for 2025b Onboarding: PandocMissing #2677

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 66 additions & 54 deletions jupyter/minimal/ubi9-python-3.12/Dockerfile.cpu
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,26 @@ ARG BASE_IMAGE
# External image alias for UBI repository configuration
FROM registry.access.redhat.com/ubi9/ubi AS ubi-repos

############################
# Stage 1: PDF Tool Build #
############################
FROM registry.access.redhat.com/ubi9/python-312:latest AS pdf-builder

WORKDIR /opt/app-root/bin

# OS Packages needs to be installed as root
USER 0

# Copy scripts
COPY jupyter/utils/install_texlive.sh ./install_texlive.sh
COPY jupyter/utils/install_pandoc.sh ./install_pandoc.sh
RUN chmod +x install_texlive.sh install_pandoc.sh

RUN ./install_texlive.sh
RUN ./install_pandoc.sh
Comment on lines +20 to +25
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify the three install scripts exist and check their content

echo "=== Checking for install_texlive.sh ===" 
if [ -f ./jupyter/utils/install_texlive.sh ]; then
  echo "✓ Found install_texlive.sh"
  wc -l ./jupyter/utils/install_texlive.sh
else
  echo "✗ install_texlive.sh NOT found"
fi

echo ""
echo "=== Checking for install_pandoc.sh ===" 
if [ -f ./jupyter/utils/install_pandoc.sh ]; then
  echo "✓ Found install_pandoc.sh"
  wc -l ./jupyter/utils/install_pandoc.sh
else
  echo "✗ install_pandoc.sh NOT found"
fi

echo ""
echo "=== Checking for install_pdf_deps.sh ===" 
if [ -f ./jupyter/utils/install_pdf_deps.sh ]; then
  echo "✓ Found install_pdf_deps.sh"
  wc -l ./jupyter/utils/install_pdf_deps.sh
else
  echo "✗ install_pdf_deps.sh NOT found"
fi

echo ""
echo "=== Listing jupyter/utils directory ===" 
ls -la ./jupyter/utils/ 2>/dev/null || echo "Directory not found"

Repository: opendatahub-io/notebooks

Length of output: 192

🌐 Web query:

opendatahub-io/notebooks jupyter/utils install_texlive.sh install_pandoc.sh

💡 Result:

I searched the opendatahub-io/notebooks repository but I couldn't find files named install_texlive.sh or install_pandoc.sh under jupyter/utils (or elsewhere) in that repo. The repository index and jupyter directory listing I checked are here. [1]

Do you want me to (a) search other branches/older commits for those filenames, (b) search other repos or the web for equivalent install scripts, or (c) show example install scripts for TeX Live and Pandoc you can use?

References:
[1] opendatahub-io/notebooks (repo root / jupyter directory).

🌐 Web query:

opendatahub-io/notebooks install_pdf_deps.sh

💡 Result:

Do you want me to (pick one):

  • find and show the install_pdf_deps.sh file contents from the opendatahub-io/notebooks repo, or
  • explain how to run/install PDF dependencies for ODH notebooks, or
  • search the repo for related scripts and steps and summarize them?

Which would you like?

Critical: install_texlive.sh and install_pandoc.sh scripts are missing and will cause Docker build to fail.

Lines 20–21 attempt to COPY these scripts from jupyter/utils/, but they do not exist in the repository. The COPY commands will fail during the Docker build process. Additionally, the RUN commands at lines 24–25 that execute these scripts cannot succeed if the files are not present.

Add the missing scripts to jupyter/utils/ with implementations for all supported architectures (ppc64le, x86_64, aarch64, s390x) that output artifacts to /usr/local/texlive and /usr/local/pandoc respectively to match the cache mount paths expected in the dockerfile.


USER 1001

####################
# cpu-base #
####################
Expand All @@ -16,48 +36,39 @@ WORKDIR /opt/app-root/bin
# OS Packages needs to be installed as root
USER 0

### BEGIN upgrade first to avoid fixable vulnerabilities
# If we have a Red Hat subscription prepared, refresh it
RUN /bin/bash <<'EOF'
set -Eeuxo pipefail
if command -v subscription-manager &> /dev/null; then
subscription-manager identity &>/dev/null && subscription-manager refresh || echo "No identity, skipping refresh."
fi
EOF
# Copy pandoc from the pdf-builder stage so the runtime image supports
# nbconvert and notebook export operations during tests.
COPY --from=pdf-builder /usr/bin/pandoc /usr/bin/pandoc

# Inject the official UBI 9 repository configuration into the AIPCC base image.
# The Quay-based AIPCC image is "repo-less" by default (https://gitlab.com/redhat/rhel-ai/core/base-images/app#repositories), so dnf cannot upgrade or install packages.
# By copying ubi.repo from the public UBI 9 image, we enable package management for upgrades and installations.
COPY --from=ubi-repos /etc/yum.repos.d/ubi.repo /etc/yum.repos.d/ubi.repo

# upgrade first to avoid fixable vulnerabilities begin
# Problem: The operation would result in removing the following protected packages: systemd
# (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)
# Solution: --best --skip-broken does not work either, so use --nobest
RUN --mount=type=cache,target=/var/cache/dnf,sharing=locked,id=notebooks-dnf /bin/bash <<'EOF'
set -Eeuxo pipefail
dnf -y upgrade --refresh --nobest --skip-broken --nodocs --noplugins --setopt=install_weak_deps=0 --setopt=keepcache=1
EOF

### END upgrade first to avoid fixable vulnerabilities
RUN dnf -y upgrade --refresh --nobest --skip-broken --nodocs --noplugins --setopt=install_weak_deps=0 --setopt=keepcache=0 \
&& dnf clean all -y
# upgrade first to avoid fixable vulnerabilities end

# Install useful OS packages
RUN --mount=type=cache,target=/var/cache/dnf,sharing=locked,id=notebooks-dnf /bin/bash <<'EOF'
set -Eeuxo pipefail
dnf install -y --setopt=keepcache=1 perl mesa-libGL skopeo
EOF
RUN dnf install -y perl mesa-libGL skopeo && dnf clean all && rm -rf /var/cache/yum

# Other apps and tools installed as default user
USER 1001

### BEGIN Install micropipenv and uv to deploy packages from requirements.txt
RUN pip install --no-cache-dir --extra-index-url https://pypi.org/simple -U "micropipenv[toml]==1.9.0" "uv==0.9.6"
### END Install micropipenv and uv to deploy packages from requirements.txt

### BEGIN Install the oc client
RUN /bin/bash <<'EOF'
set -Eeuxo pipefail
curl -L https://mirror.openshift.com/pub/openshift-v4/$(uname -m)/clients/ocp/stable/openshift-client-linux.tar.gz \
-o /tmp/openshift-client-linux.tar.gz
tar -xzvf /tmp/openshift-client-linux.tar.gz oc
rm -f /tmp/openshift-client-linux.tar.gz
EOF
# Install micropipenv and uv to deploy packages from requirements.txt begin
RUN pip install --no-cache-dir -U "micropipenv[toml]==1.9.0" "uv==0.8.12"
# Install micropipenv and uv to deploy packages from requirements.txt end

### END Install the oc client
# Install the oc client begin
RUN curl -L https://mirror.openshift.com/pub/openshift-v4/$(uname -m)/clients/ocp/stable/openshift-client-linux.tar.gz \
-o /tmp/openshift-client-linux.tar.gz && \
tar -xzvf /tmp/openshift-client-linux.tar.gz oc && \
rm -f /tmp/openshift-client-linux.tar.gz
# Install the oc client end

####################
# jupyter-minimal #
Expand All @@ -83,35 +94,36 @@ COPY ${JUPYTER_REUSABLE_UTILS} utils/

USER 0

### BEGIN Dependencies for PDF export
RUN ./utils/install_pdf_deps.sh
ENV PATH="/usr/local/texlive/bin/linux:/usr/local/pandoc/bin:$PATH"

### END Dependencies for PDF export
# Dependencies for PDF export begin
RUN --mount=type=cache,from=pdf-builder,source=/usr/local/,target=/pdf_builder/,rw \
bash -c ' \
if [[ "$(uname -m)" == "ppc64le" ]]; then \
cp -r /pdf_builder/texlive /usr/local/; \
cp -r /pdf_builder/pandoc /usr/local/; \
else \
./utils/install_pdf_deps.sh; \
fi'

USER 1001

COPY ${MINIMAL_SOURCE_CODE}/pylock.toml ${MINIMAL_SOURCE_CODE}/start-notebook.sh ./

# Install Python dependencies from requirements.txt file
RUN /bin/bash <<'EOF'
set -Eeuxo pipefail
echo "Installing softwares and packages"
# This may have to download and compile some dependencies, and as we don't lock requirements from `build-system.requires`,
# we often don't know the correct hashes and `--require-hashes` would therefore fail on non amd64, where building is common.
uv pip install --strict --no-deps --no-cache --no-config --no-progress --verify-hashes --compile-bytecode --index-strategy=unsafe-best-match --requirements=./pylock.toml
# Disable announcement plugin of jupyterlab
jupyter labextension disable "@jupyterlab/apputils-extension:announcements"
# Replace Notebook's launcher, "(ipykernel)" with Python's version 3.x.y
sed -i -e "s/Python.*/$(python --version | cut -d '.' -f-2)\",/" /opt/app-root/share/jupyter/kernels/python3/kernel.json
# copy jupyter configuration
cp /opt/app-root/bin/utils/jupyter_server_config.py /opt/app-root/etc/jupyter
# Fix permissions to support pip in Openshift environments
chmod -R g+w /opt/app-root/lib/python3.12/site-packages
fix-permissions /opt/app-root -P
# Apply JupyterLab addons
/opt/app-root/bin/utils/addons/apply.sh
EOF
RUN echo "Installing softwares and packages" && \
# This may have to download and compile some dependencies, and as we don't lock requirements from `build-system.requires`,
# we often don't know the correct hashes and `--require-hashes` would therefore fail on non amd64, where building is common.
uv pip install --strict --no-deps --no-cache --no-config --no-progress --verify-hashes --compile-bytecode --index-strategy=unsafe-best-match --requirements=./pylock.toml && \
# Disable announcement plugin of jupyterlab \
jupyter labextension disable "@jupyterlab/apputils-extension:announcements" && \
# Replace Notebook's launcher, "(ipykernel)" with Python's version 3.x.y \
sed -i -e "s/Python.*/$(python --version | cut -d '.' -f-2)\",/" /opt/app-root/share/jupyter/kernels/python3/kernel.json && \
# copy jupyter configuration
cp /opt/app-root/bin/utils/jupyter_server_config.py /opt/app-root/etc/jupyter && \
# Fix permissions to support pip in Openshift environments \
chmod -R g+w /opt/app-root/lib/python3.12/site-packages && \
fix-permissions /opt/app-root -P && \
# Apply JupyterLab addons \
/opt/app-root/bin/utils/addons/apply.sh

WORKDIR /opt/app-root/src

Expand Down
20 changes: 20 additions & 0 deletions jupyter/utils/install_pandoc.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/bin/bash
set -euxo pipefail

# Mapping of `uname -m` values to equivalent GOARCH values
declare -A UNAME_TO_GOARCH
UNAME_TO_GOARCH["x86_64"]="amd64"
UNAME_TO_GOARCH["aarch64"]="arm64"
UNAME_TO_GOARCH["ppc64le"]="ppc64le"
UNAME_TO_GOARCH["s390x"]="s390x"

ARCH="${UNAME_TO_GOARCH[$(uname -m)]}"

if [[ "$ARCH" == "ppc64le" ]]; then

# Install Pandoc from EPEL repository
dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
dnf install -y pandoc
Comment on lines +16 to +17
Copy link
Contributor

@coderabbitai coderabbitai bot Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add error handling and checksum verification for external dependencies.

Lines 16–17 install an external RPM and Pandoc package without verifying integrity or handling installation failures. If dnf install fails silently or the RPM is compromised, the build continues unchecked. Apply strict error checking and pin package versions or checksums:

-  dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
-  dnf install -y pandoc
+  dnf install -y --setopt=skip_missing_names_on_install=False \
+    https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && \
+  dnf install -y --setopt=skip_missing_names_on_install=False pandoc && \
+  pandoc --version  # Verify installation
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
dnf install -y pandoc
dnf install -y --setopt=skip_missing_names_on_install=False \
https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && \
dnf install -y --setopt=skip_missing_names_on_install=False pandoc && \
pandoc --version # Verify installation
🤖 Prompt for AI Agents 
In jupyter/utils/install_pandoc.sh around lines 16–17, the script currently runs
dnf install on an external RPM and pandoc without integrity checks or robust
error handling; update the flow to (1) download the epel-release RPM to a
temporary file, (2) verify its checksum (and/or RPM GPG signature) against a
pinned value before installing, (3) install using dnf only if verification
succeeds, (4) pin the pandoc package/version (or install a specific released
RPM) and verify its checksum/signature as well, and (5) enable strict failure
behavior (e.g., set -e) and check dnf exit codes so the script exits with a
clear error message if any download, checksum verification, or install step
fails.

pandoc --version
Copy link
Member

@jiridanek jiridanek Nov 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You seem to be adding a step that installs pandoc where previously it was not installed, and at the same you're adding a skip in a test that was previously checking for pandoc being present.

I'd expect you'd do one or the other, not both, tbh.

Copy link
Contributor Author

@dibryant dibryant Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes you are correct I will change the test doc back and rebuild.


fi
83 changes: 83 additions & 0 deletions jupyter/utils/install_texlive.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
#!/bin/bash
set -euxo pipefail

# Mapping of `uname -m` values to equivalent GOARCH values
declare -A UNAME_TO_GOARCH
UNAME_TO_GOARCH["x86_64"]="amd64"
UNAME_TO_GOARCH["aarch64"]="arm64"
UNAME_TO_GOARCH["ppc64le"]="ppc64le"
UNAME_TO_GOARCH["s390x"]="s390x"

ARCH="${UNAME_TO_GOARCH[$(uname -m)]}"

if [[ "$ARCH" == "ppc64le" ]]; then

echo "Installing TeX Live from source for $ARCH..."

# Install build dependencies
dnf install -y gcc-toolset-13 perl make libX11-devel \
zlib-devel freetype-devel libpng-devel ncurses-devel \
gd-devel libtool wget tar xz \
https://mirror.stream.centos.org/9-stream/AppStream/ppc64le/os/Packages/libXmu-devel-1.1.3-8.el9.ppc64le.rpm \
https://mirror.stream.centos.org/9-stream/AppStream/ppc64le/os/Packages/libXext-devel-1.3.4-8.el9.ppc64le.rpm \
https://mirror.stream.centos.org/9-stream/AppStream/ppc64le/os/Packages/libICE-devel-1.0.10-8.el9.ppc64le.rpm \
https://mirror.stream.centos.org/9-stream/AppStream/ppc64le/os/Packages/libSM-devel-1.2.3-10.el9.ppc64le.rpm \
https://mirror.stream.centos.org/9-stream/AppStream/ppc64le/os/Packages/libXmu-1.1.3-8.el9.ppc64le.rpm \
https://mirror.stream.centos.org/9-stream/AppStream/ppc64le/os/Packages/libXaw-devel-1.0.13-19.el9.ppc64le.rpm \
https://mirror.stream.centos.org/9-stream/AppStream/ppc64le/os/Packages/libXaw-1.0.13-19.el9.ppc64le.rpm \
https://mirror.stream.centos.org/9-stream/AppStream/ppc64le/os/Packages/libXt-devel-1.2.0-6.el9.ppc64le.rpm \
https://mirror.stream.centos.org/9-stream/AppStream/ppc64le/os/Packages/flex-2.6.4-9.el9.ppc64le.rpm \
https://mirror.stream.centos.org/9-stream/AppStream/ppc64le/os/Packages/bison-3.7.4-5.el9.ppc64le.rpm
Comment on lines +21 to +30
Copy link
Contributor

@coderabbitai coderabbitai bot Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add checksum verification for all external downloads.

External downloads lack integrity verification, making builds non-reproducible and vulnerable to supply-chain risks. Downloads occur at:

  • Lines 21–30: CentOS RPMs (hardcoded URLs)
  • Line 33: TeX Live 2025 source archive
  • Line 55: TeX Live installer archive

Compute and verify SHA256 checksums immediately after each download:

   # Step 1: Download and extract the TeX Live source
-  wget https://ftp.math.utah.edu/pub/tex/historic/systems/texlive/2025/texlive-20250308-source.tar.xz
-  tar -xf texlive-20250308-source.tar.xz
+  TEXLIVE_SOURCE="texlive-20250308-source.tar.xz"
+  TEXLIVE_SOURCE_SHA256="<expected-hash-here>"
+  wget "https://ftp.math.utah.edu/pub/tex/historic/systems/texlive/2025/${TEXLIVE_SOURCE}"
+  echo "${TEXLIVE_SOURCE_SHA256}  ${TEXLIVE_SOURCE}" | sha256sum -c - || exit 1
+  tar -xf "${TEXLIVE_SOURCE}"

Apply the same pattern to the TeX Live installer (line 55) and consider pinning RPM package checksums or caching them locally.

Also applies to: 33-33, 55-55


# Step 1: Download and extract the TeX Live source
wget https://ftp.math.utah.edu/pub/tex/historic/systems/texlive/2025/texlive-20250308-source.tar.xz
tar -xf texlive-20250308-source.tar.xz
cd texlive-20250308-source

# Enable newer GCC toolchain
source /opt/rh/gcc-toolset-13/enable

# Create build directory and build
mkdir -p ../texlive-build
cd ../texlive-build
../texlive-20250308-source/configure --prefix=/usr/local/texlive
make -j"$(nproc)"
make install

# Symlink for pdflatex
ln -sf pdftex /usr/local/texlive/bin/powerpc64le-unknown-linux-gnu/pdflatex

# Cleanup sources to reduce image size
rm -rf /texlive-20250308-source /texlive-build

# Step 2: Run TeX Live installer for runtime tree setup
cd /
wget https://mirror.ctan.org/systems/texlive/tlnet/install-tl-unx.tar.gz
tar -xzf install-tl-unx.tar.gz
cd install-tl-2*/

Comment on lines +55 to +58
Copy link
Contributor

@coderabbitai coderabbitai bot Nov 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Unsafe glob expansion for directory traversal.

Line 57 uses cd install-tl-2*/ which relies on glob expansion. If extraction fails, produces zero matches, or produces multiple matches, the glob silently expands unpredictably. Add explicit existence and uniqueness checks:

   wget https://mirror.ctan.org/systems/texlive/tlnet/install-tl-unx.tar.gz
   tar -xzf install-tl-unx.tar.gz
-  cd install-tl-2*/
+  
+  # Find the extracted install-tl directory
+  INSTALL_TL_DIR=$(find . -maxdepth 1 -type d -name 'install-tl-*' | head -1)
+  if [[ -z "$INSTALL_TL_DIR" ]]; then
+    echo "Error: TeX Live installer directory not found after extraction"
+    exit 1
+  fi
+  cd "$INSTALL_TL_DIR"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
wget https://mirror.ctan.org/systems/texlive/tlnet/install-tl-unx.tar.gz
tar -xzf install-tl-unx.tar.gz
cd install-tl-2*/
wget https://mirror.ctan.org/systems/texlive/tlnet/install-tl-unx.tar.gz
tar -xzf install-tl-unx.tar.gz
# Find the extracted install-tl directory
INSTALL_TL_DIR=$(find . -maxdepth 1 -type d -name 'install-tl-*' | head -1)
if [[ -z "$INSTALL_TL_DIR" ]]; then
echo "Error: TeX Live installer directory not found after extraction"
exit 1
fi
cd "$INSTALL_TL_DIR"
🤖 Prompt for AI Agents 
In jupyter/utils/install_texlive.sh around lines 55 to 58, replace the unsafe
glob cd into install-tl-2*/ with explicit checks: after downloading and
extracting, verify tar succeeded, then list directories matching install-tl-2*
and ensure there is exactly one match (error and exit if zero or multiple),
assign that single directory to a variable and cd into that variable; this
prevents silent failures or ambiguous matches.

# Create a custom install profile
TEXLIVE_INSTALL_PREFIX="/usr/local/texlive"
cat <<EOF > texlive.profile
selected_scheme scheme-small
TEXDIR $TEXLIVE_INSTALL_PREFIX
TEXMFCONFIG ~/.texlive2025/texmf-config
TEXMFVAR ~/.texlive2025/texmf-var
option_doc 0
option_src 0
EOF

./install-tl --profile=texlive.profile --custom-bin=$TEXLIVE_INSTALL_PREFIX/bin/powerpc64le-unknown-linux-gnu

# TeX Live binary directory
TEX_BIN_DIR="/usr/local/texlive/bin/powerpc64le-unknown-linux-gnu"

# Create standard symlink 'linux' → arch-specific folder
ln -sf "$TEX_BIN_DIR" /usr/local/texlive/bin/linux

# Set up environment
export PATH="$TEXLIVE_INSTALL_PREFIX/bin/linux:$PATH"
pdflatex --version
tlmgr --version

fi
Loading