Storage deployment

[gctucker]

The new API & Pipeline require some form of storage for the artifacts: kernel source code tarballs, build binaries, test logs etc. The default docker-compose setup provides SSH to upload files and Nginx to serve them in local containers. For a production deployment, we need to consider various solutions and choose which ones to support in general and which particular one to use for kernelci.org deployments.

• Determine file storage solution #9
• Evaluate chosen storage solution #263
• Deploy storage solution for production #264

[gctucker]

I've been able to upload a file to the Azure storage using the azure-storage-file-share Python package and download it again with a public URL:

curl "https://kciapistagingstorage1.file.core.windows.net/exp1/notifier.txt?<some-azure-token>"

The only issue here is that a token has to be included in the URL. I'll see if we can have a read-only token that doesn't expire or a way to make files publicly readable anonymously.

The file used for the experiment is notifier.txt which is a plain text log file from the API notifier service. In the URL above, exp1 is the name of the share or "container". There is no limit for the number of containers as per the Azure storage documentation so we can have one per node like directories with the SSH / Nginx storage solution.

I created a token via the Azure portal, this can then be saved in the TOML settings file like for an SSH key.

As such, this meets all the criteria for implementing a Storage based on Azure files. I believe we can use it for the Early Access phase as this will also give us the possibility to manage quotas for each user account. It will also help us evaluate whether we want to keep using this for production as the default storage. Users can also use alternative storage solutions since the API just requires a public URL for file downloads.