|
1 | | -# Packaging for High-Performance Computing (Notes) |
| 1 | +# Packaging for High-Performance Computing Demo Notes |
2 | 2 |
|
3 | | -**Note**: It is recommended to try out Spack in a fresh Docker container. To understand how Spack itself is installed, follow Step 1 in a fresh Ubuntu container. To make things simpler from Step 2 onwards, create a container from the [spack/ubuntu-jammy](https://hub.docker.com/r/spack/ubuntu-jammy) image, so that Spack is preinstalled. |
| 3 | +**Note**: It is recommended to try out Spack in a fresh Docker container. To understand how Spack itself is installed, follow Step 1 in a fresh Ubuntu Noble container (`docker run --rm -it ubuntu:noble`). To make things simpler from Step 2 onwards, create a container from the [spack/ubuntu-noble](https://hub.docker.com/r/spack/ubuntu-noble) image, so that Spack is preinstalled. |
4 | 4 |
|
5 | 5 | ## 1. Spack Setup/Installation |
6 | 6 |
|
7 | | -- Git repository of Python scripts |
| 7 | +- This demo (and Spack) needs Python, Git, a C/C++ compiler, patch, make, tar, and a few more tools. Basically `build-essential`, `git`, and `python` on Ubuntu Noble. |
| 8 | + |
| 9 | +- Get the Spack repository |
8 | 10 |
|
9 | 11 | ```bash |
10 | | - git clone -b v0.23.0 -c feature.manyFiles=true --depth=2 https://github.com/spack/spack.git |
| 12 | + git clone -b v1.1.0 --depth=2 https://github.com/spack/spack.git |
11 | 13 | ``` |
12 | 14 |
|
13 | | - - `v0.23.0` is currently the latest major release |
14 | | - |
15 | | -- **Note:** Install `curl`, `libcurl4-openssl-dev`, and `vim` to ensure that Spack v0.23.0 works in a fresh Ubuntu Jammy container. |
16 | | - |
17 | 15 | - Initializing Spack with |
18 | 16 |
|
19 | 17 | ```bash |
20 | 18 | . <spack_prefix>/share/spack/setup-env.sh |
21 | 19 | ``` |
22 | 20 |
|
23 | | - will set `SPACK_ROOT` and also add `spack` to `PATH`. |
24 | | - |
25 | | - Note that the `.` operator will run the commands in the supplied script as if we would type the commands supplied by the script in the shell ourselves. In bash the `.` operator is equivalent to `source`. However, `source` is not specified in POSIX and thus using `.` is likely to work on more platforms. |
| 21 | + will set `SPACK_ROOT` and also add `spack` to `PATH`. Note that the `.` operator will run the commands in the supplied script as if we would type the commands supplied by the script in the shell ourselves. In bash the `.` operator is equivalent to `source`. However, `source` is not specified in POSIX and thus using `.` is likely to work on more platforms. |
26 | 22 |
|
27 | 23 | - Finish Spack setup |
28 | 24 |
|
|
40 | 36 |
|
41 | 37 | prints the list of compilers that Spack has added. |
42 | 38 |
|
43 | | -- Find external packages (optional) |
| 39 | +- Find external packages |
44 | 40 |
|
45 | 41 | ```bash |
46 | 42 | spack external find |
47 | 43 | ``` |
48 | 44 |
|
49 | 45 | This command tries to find preinstalled software packages which are also in Spack's package database. This way we do not have to recompile everything from scratch. |
50 | 46 |
|
51 | | - **Note** `spack external find` is an experimental feature and might fail. System packages [can be defined manually](https://spack.readthedocs.io/en/latest/getting_started.html#system-packages). |
52 | | - |
53 | | - - Found packages (including version, configuration etc.) are stored in `~/.spack/packages.yaml`. There one can add further packages manually. **Note**: Maybe show content of this file. |
| 47 | + Found packages (including version, configuration etc.) are stored in `~/.spack/packages.yaml`. There one can add further packages manually. Look at contents of this file. |
54 | 48 |
|
55 | | -- Concretize a spec to trigger bootstrap process (optional) |
| 49 | +- Concretize a spec to trigger bootstrap process |
56 | 50 |
|
57 | 51 | ```bash |
58 | 52 | spack spec zlib |
59 | 53 | ``` |
60 | 54 |
|
61 | | - This will try to concretize the package and its dependencies. `clingo`, Spack's concretizer, is needed for this. If one calls this command the very first time, a bootstrapping process is triggered that installs `clingo`. |
| 55 | + This will try to concretize the package and its dependencies. `clingo`, Spack's concretizer, is needed for this. Running `spack spec` for the first time triggers a bootstrapping process that installs `clingo`. |
62 | 56 |
|
63 | 57 | ## 2. Package Installation/Management with Spack |
64 | 58 |
|
|
68 | 62 | spack info zlib |
69 | 63 | ``` |
70 | 64 |
|
71 | | -- Install a simple example package |
72 | | - |
73 | | - ```bash |
74 | | - spack spec zlib |
75 | | - ``` |
76 | | - |
77 | | - This will trigger the bootstrap process of Spack. It installs `clingo`, the dependency resolver, if not done before. |
78 | | - |
79 | | -- Install `zlib` |
| 65 | +- Install the package |
80 | 66 |
|
81 | 67 | ```bash |
82 | 68 | spack install --reuse zlib |
|
102 | 88 |
|
103 | 89 | Spack will fetch the archive and try to deduce some basic settings. It will also check for other releases and asks whether to add checksums. We want all checksums. |
104 | 90 |
|
105 | | -- Afterwards Spack drops us in the boilerplate package. |
| 91 | +- Spack immediately drops you in a boilerplate package file. |
106 | 92 |
|
107 | 93 | ```Python |
108 | 94 | class Helloworld(CMakePackage): |
|
133 | 119 |
|
134 | 120 | At least the `FIXME` statements should be fixed before releasing the package. The Spack package recipe is written in Python so we can also do common Python magic here. |
135 | 121 |
|
136 | | -- Spack has deduced a good number of information already. |
| 122 | +- Spack has deduced a good amount of information already. |
137 | 123 | - We work with CMake |
138 | 124 | - There are 3 releases of the software |
139 | | - - The URL to download packages |
140 | | - - The name of the Package we work with `class Helloworld`. This is also the name of the software now within Spack. |
| 125 | + - The URL to download the package |
| 126 | + - The name of the package we work with `class Helloworld`. This is also the name of the software now within Spack. |
141 | 127 | - We want to fix/extend the package with some standard information |
142 | 128 | - Package description |
143 | 129 | - Set URL to SSE homepage |
144 | | - - Add our GitHub username as maintainer |
145 | | - - Remove the `cmake_args` part as we only have a standard CMake arguments. Here we could give extra/special arguments specific to the software package. |
| 130 | + - Add GitHub username as maintainer |
| 131 | + - Remove the `cmake_args` part as there are only standard CMake arguments. |
146 | 132 |
|
147 | 133 | - Concretize the package |
148 | 134 |
|
|
158 | 144 | ^ [email protected]% [email protected]~doc+ncurses+openssl+ownlibs~qt build_type=Release patches=1c540040c7e203dd8e27aa20345ecb07fe06570d56410a24a266ae570b1c4c39,bf695e3febb222da2ed94b3beea600650e4318975da90e4a71d6f31a6d5d8c3d arch=linux-ubuntu20.04-skylake |
159 | 145 | ``` |
160 | 146 |
|
161 | | - We see that `cmake` is an implicit dependency as we need it for building our package. |
| 147 | + `cmake` is an implicit dependency as we need it for building our package. |
162 | 148 |
|
163 | 149 | - If the same Docker container as in step 2 is used, make sure to uninstall `zlib` before installing `helloworld`. |
164 | 150 |
|
|
180 | 166 | spack find |
181 | 167 | ``` |
182 | 168 |
|
183 | | -- Load installed package, run code and unload |
| 169 | +- Load installed package, run, and unload |
184 | 170 |
|
185 | 171 | ```bash |
186 | 172 | spack load helloworld |
187 | 173 | helloworld |
188 | 174 | spack unload helloworld |
189 | 175 | ``` |
190 | 176 |
|
191 | | -- Install different version of code |
| 177 | +- Install different version |
192 | 178 |
|
193 | 179 | ```bash |
194 | 180 | |
|
204 | 190 |
|
205 | 191 | or open `package.py` file in `${HOME}/var/spack/repos/builtin/packages/helloworld/` |
206 | 192 |
|
207 | | -- **Optional**: one could add `main` branch and thus GitHub repository |
| 193 | +- Add the `main` branch as a version |
208 | 194 |
|
209 | 195 | ```diff |
210 | | - + git = "https://github.com/Simulation-Software-Engineering/HelloWorld.git" |
| 196 | + + git = "https://github.com/Simulation-Software-Engineering/HelloWorld.git" |
211 | 197 | + |
212 | 198 | + version("main", branch="main") |
213 | 199 | ``` |
214 | 200 |
|
215 | | - This can also be used for `develop` branches etc. It is useful if one needs really the newest version of a package or if one develops software using Spack. |
| 201 | + This can also be used for `develop` branches etc. It is useful if the development state is required, or if one develops software using Spack. |
216 | 202 |
|
217 | 203 | - Add artificial dependencies |
218 | 204 |
|
|
221 | 207 | + depends_on("zlib@:1.2") |
222 | 208 | ``` |
223 | 209 |
|
224 | | - This means that the package depends on Python `3.0.0` or newer and newer if we use `helloworld` of version `0.3.0` or newer. The software also requires at most `zlib` in version `1.2.10` |
| 210 | + This states that the package depends on Python version `3.0.0` or newer if we use `helloworld` of version `0.3.0` or newer. The software also requires at most `zlib` in version `1.2.10` |
225 | 211 |
|
226 | | - - Show new dependencies |
| 212 | +- Show new dependencies |
227 | 213 |
|
228 | | - ```bash |
229 | | - spack spec helloworld |
230 | | - |
231 | | - spack info helloworld |
232 | | - ``` |
| 214 | + ```bash |
| 215 | + spack spec helloworld |
| 216 | + |
| 217 | + spack info helloworld |
| 218 | + ``` |
233 | 219 |
|
234 | | - The Python dependency will only show up for the newest version of our software package. |
| 220 | + The Python dependency will only show up for version `0.3.0`. |
235 | 221 |
|
236 | 222 | - Add an artificial variant |
237 | 223 |
|
|
247 | 233 | spack info helloworld |
248 | 234 | ``` |
249 | 235 |
|
250 | | - where Python now shows up as variant with its description. We can deactivate Python by specifying |
| 236 | + where Python now shows up as variant with its description. Python dependency is removed by doing |
251 | 237 |
|
252 | 238 | ```bash |
253 | 239 | spack info helloworld -python |
254 | 240 | ``` |
255 | 241 |
|
256 | | - `~` can be (often) used instead of `-`. There are [examples in the documentation](https://spack.readthedocs.io/en/latest/basic_usage.html#variants). |
257 | | - |
258 | 242 | ## Further material |
259 | 243 |
|
260 | 244 | ### References |
261 | 245 |
|
262 | | -- [EasyBuild](https://github.com/easybuilders/easybuild) |
263 | | -- [EasyConfigs](https://github.com/easybuilders/easybuild-easyconfigs) |
264 | 246 | - [Spack](https://spack.io/) |
265 | 247 | - [Spack docs](https://spack.readthedocs.io/en/latest/) |
266 | 248 | - [Spack 101 tutorials](https://spack-tutorial.readthedocs.io/en/latest/) |
| 249 | +- [EasyBuild](https://github.com/easybuilders/easybuild) |
| 250 | +- [EasyConfigs](https://github.com/easybuilders/easybuild-easyconfigs) |
267 | 251 | - [archspec project](https://github.com/archspec/) |
268 | 252 |
|
269 | | -### Talks |
| 253 | +### Interesting Talks On This Topic |
270 | 254 |
|
271 | | -Talks at FOSDEM |
| 255 | +FOSDEM talks: |
272 | 256 |
|
273 | 257 | - 2020: [Spack's new Concretizer](https://archive.fosdem.org/2020/schedule/event/dependency_solving_not_just_sat/) |
274 | 258 | - 2020: [Build for your microarchitecture: experiences with Spack and archspec](https://archive.fosdem.org/2020/schedule/event/archspec/) |
|
0 commit comments