[BUG] ISO install fails - ISO image built with tag 3.0.20250806 fails to install

[smitesh-sutaria]

Bug Description

Earlier the ISO built with same tag worked.

When re-built today with this tag, the ISO fails to install.
Even able to reproducible on the Virtual Machine.

System Setup

Oracle Virtual machine, RPL/ADL.

Any hardware/VM, the ISO built with the tag 3.0.20250806 unable to install

Reproducible Steps

Follow the build guide to build ISO image for this tag 3.0.20250806

Install the generated ISO on virtual machine or any bare-metal hardware like ADL/RPL.

Installation fails

Root Cause Analysis

No response

[smitesh-sutaria]

Please note that, same steps worked few months/weeks back.
And as we are working on a tag, irrespective of timeline, it should work without issues.

Are there any other dependencies?

@anujm1 @sfonn @yangliang-intel

[yangliang-intel]

Hi @smitesh-sutaria,
Thanks for the feedback.
This is a known issue, and the fix has already been merged in PR: 478). The change will be included in the upcoming EMT 3.0 release (expected by sometime next week).

[smitesh-sutaria]

Thanks @yangliang-intel

The issue which we are hitting is a different one. (But eventually, at next step we will hit this root password issue as well).

Further investigation/debugging showed that same tag built on different timeframe generates different ISO size images.
Please note that, we have github workflow to do EMT build and generate ISO. Hence the scripts/workflow are same.

For ex: The one built 2 months ago generated ISO file size of ~925 MB. Built recently since few days, gave ISO file size of ~1 GB (This one does not get install)

Mounted both the ISOs and saw some differences in the RPMs as well.
For ex: The one on left is 4-NOV and to the right is 11-SEP. Same tag, same script.

Our understanding is, a EMT tag once available should generate the same image. Isn’t it ?

Can someone from your team look into building the https://github.com/open-edge-platform/edge-microvisor-toolkit/releases/tag/3.0.20250806 and see if it installs.
For us, building this tag and install fails on multiple hardware. (Older generated ISO works).

Further, just generated ISO from my local machine without making the toolchain (using older rpms), that ISO installed and hit into the password issue. But freshly build toolchain with ISO image does not even install. Fails at first step itself.

Is there something I am missing?

@anujm1 @krishnajs @magerstam @sfonn

[yangliang-intel]

Hi @smitesh-sutaria,
Thanks for providing the detailed info!
Our release team is checking on the build issue, and we’ll update you as soon as we have more details.

[cheeyanglee]

Hi @smitesh-sutaria

when build with REBUILD_PACKAGES=n, the build process uses prebuilt rpms, this will download latest available rpms from rpm repo , ignoring the version in SPECS hence the installed rpms would not match the tags.

Currently there is no way to rebuild an old tags with prebuild rpms (with matching versions) from the rpm repo.
However It should still possible to rebuild everything locally by doing a full rebuild. please note that full rebuild would take significantly longer time.
These are the steps to rebuild a tag with matching rpm/specs version locally:

# rebuild toolchain
sudo make toolchain \
    PACKAGE_URL_LIST="" \
    REPO_LIST="" \
    DISABLE_UPSTREAM_REPOS=y \
    REBUILD_TOOLCHAIN=y \
    REBUILD_TOOLS=y 

# rebuild packages/rpms local toolchain
sudo make build-packages REBUILD_TOOLS=y VALIDATE_TOOLCHAIN_GPG=n

# rebuild iso images
sudo make iso REBUILD_TOOLS=y VALIDATE_TOOLCHAIN_GPG=n REBUILD_PACKAGES=n CONFIG_FILE=./imageconfigs/full.json

we will have this updates in the document. #561

[sgolebiewski-intel]

Hi @smitesh-sutaria

when build with REBUILD_PACKAGES=n, the build process uses prebuilt rpms, this will download latest available rpms from rpm repo , ignoring the version in SPECS hence the installed rpms would not match the tags.

Currently there is no way to rebuild an old tags with prebuild rpms (with matching versions) from the rpm repo. However It should still possible to rebuild everything locally by doing a full rebuild. please note that full rebuild would take significantly longer time. These are the steps to rebuild a tag with matching rpm/specs version locally:

# rebuild toolchain
sudo make toolchain \
    PACKAGE_URL_LIST="" \
    REPO_LIST="" \
    DISABLE_UPSTREAM_REPOS=y \
    REBUILD_TOOLCHAIN=y \
    REBUILD_TOOLS=y 

# rebuild packages/rpms local toolchain
sudo make build-packages REBUILD_TOOLS=y VALIDATE_TOOLCHAIN_GPG=n

# rebuild iso images
sudo make iso REBUILD_TOOLS=y VALIDATE_TOOLCHAIN_GPG=n REBUILD_PACKAGES=n CONFIG_FILE=./imageconfigs/full.json

we will have this updates in the document. #561

Thank your for your feedback,

I will provide the required changes in #561, so we will have two sets of instructions:

• builds from the latest branch (pre-built rpms)
• builds from the specific tag (full rebuild)

[sgolebiewski-intel]

I have updated #561. I'd appreciate it if you would provide a further review.