script.sh incorrectly reported as not found

[bsloan-icl]

Hello,

We've deployed an Ubuntu Virtual Machine Scale Set (VMSS) via Terraform and used the azurerm_virtual_machine_scale_set_extension resource to run a custom init script. However, when using script and referencing ubuntu-init-script.sh the extension fails with the following error:

Error: VM has reported a failure when processing extension 'ubuntu_init_script' (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error message: 'Enable failed: failed to execute command: command terminated with exit status=127
[stdout]

[stderr]
/bin/sh: 1: /var/lib/waagent/custom-script/download/10/script.sh: not found

The associated log files are:
handler.log
waagent.log

Looking at the VMSS in Azure, the configuration looks fine and the script is base64 encoded. Decoding the base64 results in our init script as expected:

If we login to the VM, we can see the script exists in the correct path and has the correct contents:

We can also run the script manually and it runs as expected:

root@ub-2040000ZD:~# /bin/sh /var/lib/waagent/custom-script/download/10/script.sh
Installing apt packages.
Reading package lists... Done
Building dependency tree
Reading state information... Done
unzip is already the newest version (6.0-25ubuntu1.2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Installing GitHub CLI.
Hit:1 https://cli.github.com/packages stable InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu focal InRelease
Get:3 http://azure.archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB]
Hit:4 http://azure.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:5 http://azure.archive.ubuntu.com/ubuntu focal-security InRelease
Hit:6 https://packages.microsoft.com/repos/azure-cli focal InRelease
Fetched 128 kB in 1s (186 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree
Reading state information... Done
wget is already the newest version (1.20.3-1ubuntu2.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Hit:1 http://azure.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:4 http://azure.archive.ubuntu.com/ubuntu focal-security InRelease
Hit:5 https://cli.github.com/packages stable InRelease
Hit:6 https://packages.microsoft.com/repos/azure-cli focal InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree
Reading state information... Done
gh is already the newest version (2.52.0).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Installing Azure CLI.
Hit:1 https://cli.github.com/packages stable InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu focal InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:4 http://azure.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:5 http://azure.archive.ubuntu.com/ubuntu focal-security InRelease
Hit:6 https://packages.microsoft.com/repos/azure-cli focal InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
lsb-release is already the newest version (11.1.0ubuntu2).
ca-certificates is already the newest version (20230311ubuntu0.20.04.1).
curl is already the newest version (7.68.0-1ubuntu2.22).
gnupg is already the newest version (2.2.19-3ubuntu2.2).
apt-transport-https is already the newest version (2.0.10).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Types: deb
URIs: https://packages.microsoft.com/repos/azure-cli/
Suites: focal
Components: main
Architectures: amd64
Signed-by: /etc/apt/keyrings/microsoft.gpg
Hit:1 http://azure.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:4 http://azure.archive.ubuntu.com/ubuntu focal-security InRelease
Hit:5 https://cli.github.com/packages stable InRelease
Hit:6 https://packages.microsoft.com/repos/azure-cli focal InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
azure-cli is already the newest version (2.62.0-1~focal).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

We're not sure why the extension is logging an error saying the file doesn't exist when it does? I've added the the Terraform files and init script below for reference.

vmss.tf

resource "azurerm_virtual_machine_scale_set_extension" "tf_vmss_extension_ubuntu204" {
  name                         = "ubuntu_init_script"
  virtual_machine_scale_set_id = azurerm_linux_virtual_machine_scale_set.tf_vmss_ubuntu204.id
  publisher                    = "Microsoft.Azure.Extensions"
  type                         = "CustomScript"
  type_handler_version         = "2.0"
  auto_upgrade_minor_version   = true
  settings = jsonencode({
    "script" = base64encode(data.local_file.ubuntu_init_script.content)
  })
}

data.tf

data "local_file" "ubuntu_init_script" {
  filename = "./scripts/ubuntu-init-script.sh"
}

ubuntu-init-script.sh

#!bin/sh
# Sleep to allow VM resources to be provisioned
sleep 45
# Install packages
echo "Installing apt packages."
sudo apt-get install -y unzip
# Install GitHub CLI
echo "Installing GitHub CLI."
(type -p wget >/dev/null || (sudo apt update && sudo apt-get install wget -y)) \
&& sudo mkdir -p -m 755 /etc/apt/keyrings \
&& wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null \
&& sudo chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
&& sudo apt update \
&& sudo apt install gh -y
# Install Azure CLI
echo "Installing Azure CLI."
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
exit 0