Skip to content

Commit b1f4132

Browse files
Al Chu11chu11
Al Chu11
authored and
chu11
committed
tutorials: Add a flux cancel tutorial
1 parent 93dab21 commit b1f4132

File tree

2 files changed

+222
-0
lines changed

2 files changed

+222
-0
lines changed

‎tutorials/commands/flux-cancel.rst‎

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
.. _flux-cancel:
2+
.. _flux-pkill:
3+
4+
========================
5+
How to Cancel a Flux Job
6+
========================
7+
8+
Inevitably submitted jobs will have to be canceled for one reason or another. This tutorial
9+
will show you how.
10+
11+
----------------------------
12+
How to Cancel a Job by Jobid
13+
----------------------------
14+
15+
The basic way to cancel a job is through ``flux cancel``. All you have to do is specify
16+
the jobid on the command line. Here is a simple example after submitting a job.
17+
18+
.. code-block:: console
19+
20+
$ flux submit sleep 100
21+
ƒh35Dh5qRyq
22+
23+
$ flux jobs ƒh35Dh5qRyq
24+
JOBID USER NAME ST NTASKS NNODES TIME INFO
25+
ƒh35Dh5qRyq achu sleep R 1 1 13.33s corona174
26+
27+
$ flux cancel ƒh35Dh5qRyq
28+
29+
<snip wait a little bit>
30+
31+
$ flux jobs ƒh35Dh5qRyq
32+
JOBID USER NAME ST NTASKS NNODES TIME INFO
33+
ƒh35Dh5qRyq achu sleep CA 1 1 20.18s corona174
34+
35+
In the above example we submitted a simple job via ``flux submit`` that simply
36+
runs ``sleep``. Passing the resulting jobid to ``flux jobs`` shows that it is
37+
running (state is ``R``).
38+
39+
We cancel the job simply by passing the jobid to ``flux cancel``. After waiting
40+
a little bit, we see that the job is now canceled in ``flux jobs`` (state is ``CA``).
41+
42+
While we only passed one jobid to ``flux cancel`` in this example, multiple jobids can be
43+
passed on the commandline to cancel many jobs.
44+
45+
Note that in this particular example we happened to know the jobid of our job. If you do
46+
not know the the jobid of your job, you can always use ``flux jobs`` to see a list of all
47+
your currently active jobs.
48+
49+
---------------------------
50+
Canceling Many of Your Jobs
51+
---------------------------
52+
53+
When you need to cancel many or all of your jobs, you can use either the ``--all`` option with ``flux cancel``
54+
or the ``flux pkill`` command. Lets run through several examples with the ``--all`` option first.
55+
56+
``flux cancel --all`` allows you to cancel jobs without specifying jobids. By default it cancels all of your active
57+
jobs, but several options allow you to target a subset of the jobs.
58+
59+
To start off, lets create 100 jobs that will sleep infinitely. We will use the special ``--cc`` (carbon copy)
60+
option to ``flux submit`` that will submit 100 duplicate copies of the ``sleep`` job.
61+
62+
.. code-block:: console
63+
64+
$ flux submit --cc=1-100 sleep inf
65+
<snip - many job ids printed out>
66+
67+
$ flux jobs
68+
JOBID USER NAME ST NTASKS NNODES TIME INFO
69+
ƒjTWS5m3 achu sleep S 1 - -
70+
ƒjTWS5m4 achu sleep S 1 - -
71+
ƒjTWS5m5 achu sleep S 1 - -
72+
ƒjTWS5m6 achu sleep S 1 - -
73+
<snip - there are many jobs waiting to be run>
74+
ƒjTWS5m2 achu sleep R 1 1 8.858s corona212
75+
ƒjTWS5m1 achu sleep R 1 1 8.860s corona212
76+
ƒjTUx6Um achu sleep R 1 1 8.870s corona212
77+
ƒjTUx6Uk achu sleep R 1 1 8.870s corona212
78+
ƒjTUx6Uj achu sleep R 1 1 8.870s corona212
79+
ƒjTUx6Ui achu sleep R 1 1 8.871s corona212
80+
<snip - there are many jobs running>
81+
82+
As you can see, we have a lot of jobs waiting to run (state ``S``) and a lot of running jobs (state ``R``).
83+
84+
Lets first ``flux cancel --all`` without any options.
85+
86+
.. code-block:: console
87+
88+
$ flux cancel --all
89+
flux-cancel: Canceled 100 jobs (0 errors)
90+
91+
$ flux jobs
92+
JOBID USER NAME ST NTASKS NNODES TIME INFO
93+
94+
As you can see, all the jobs are now canceled. ``flux jobs``
95+
confirms there are no longer any of our jobs running or waiting to run.
96+
97+
There are several options to filter the jobs to cancel when using the ``--all`` option. Perhaps the most commonly used
98+
option is the ``-S`` or ``--states`` option. The ``--states`` option specifies the state(s) of a job to cancel. The most
99+
common states to target are ``pending`` and ``running``. Lets resubmit our 100 jobs and see the result
100+
of trying to cancel ``pending`` vs ``running`` jobs.
101+
102+
.. code-block:: console
103+
104+
$ flux submit --cc=1-100 sleep inf
105+
<snip - many job ids printed out>
106+
107+
$ flux cancel --all --states=pending
108+
flux-cancel: Canceled 52 jobs (0 errors)
109+
110+
$ flux cancel --all --states=running
111+
flux-cancel: Canceled 48 jobs (0 errors)
112+
113+
As you can see ``flux cancel --all --states=pending`` targeted the 52 pending jobs for cancellation and
114+
``flux cancel --all --states=running`` targeted the current 48 running jobs for cancellation.
115+
116+
--------------------------
117+
Cancelling with Flux Pkill
118+
--------------------------
119+
120+
The final way to cancel a job is via ``flux pkill``. There are a number of search and filtering options available in
121+
``flux pkill`` which can be seen in the :core:man1:`flux-pkill` manpage.
122+
123+
However, there are two common ways ``flux pkill`` is used. The first is to cancel a range of jobids. The jobid range can be specified
124+
via the format ``jobid1..jobidN``.
125+
126+
It is best shown with an example.
127+
128+
.. code-block:: console
129+
130+
$ flux submit --cc=1-5 sleep inf
131+
ƒ3vEobuhH
132+
ƒ3vEobuhJ
133+
ƒ3vEobuhK
134+
ƒ3vEq5tyd
135+
ƒ3vEq5tye
136+
137+
$ flux jobs
138+
JOBID USER NAME ST NTASKS NNODES TIME INFO
139+
ƒ3vEq5tye achu sleep R 1 1 14.23s corona212
140+
ƒ3vEq5tyd achu sleep R 1 1 14.23s corona212
141+
ƒ3vEobuhK achu sleep R 1 1 14.23s corona212
142+
ƒ3vEobuhJ achu sleep R 1 1 14.23s corona212
143+
ƒ3vEobuhH achu sleep R 1 1 14.23s corona212
144+
145+
Similar to before, we've submitted some sleep jobs. We see all five of the sleep jobs are
146+
running (state ``R``) in the ``flux jobs`` output.
147+
148+
We can inform ``flux pkill`` to cancel the set of 5 jobs by specifying the first and last jobid of this range.
149+
150+
.. code-block:: console
151+
152+
$ flux pkill ƒ3vEobuhH..ƒ3vEq5tye
153+
flux-pkill: INFO: Canceled 5 jobs
154+
155+
$ flux jobs
156+
JOBID USER NAME ST NTASKS NNODES TIME INFO
157+
158+
As you can see ``flux pkill`` canceled the five jobs in the range.
159+
160+
The other common way to ``flux pkill`` is used is to cancel jobs with matching job names. For example, you may
161+
submit several different types of jobs and give them different types of names to describe their function. ``flux pkill``
162+
can be used to match on the job names and cancel only the ones that match.
163+
164+
Lets submit several jobs and give them specific names using the ``--job-name`` option.
165+
166+
.. code-block:: console
167+
168+
$ flux submit --job-name=foo sleep inf
169+
ƒ6KjHNcxP
170+
171+
$ flux submit --job-name=foobar sleep inf
172+
ƒ6Limcmju
173+
174+
$ flux submit --job-name=boo sleep inf
175+
ƒ6NCaXCmV
176+
177+
$ flux submit --job-name=baz sleep inf
178+
ƒ6PjZG6jq
179+
180+
$ flux jobs
181+
JOBID USER NAME ST NTASKS NNODES TIME INFO
182+
ƒ6PjZG6jq achu baz R 1 1 38.06s corona212
183+
ƒ6NCaXCmV achu boo R 1 1 41.54s corona212
184+
ƒ6Limcmju achu foobar R 1 1 44.9s corona212
185+
ƒ6KjHNcxP achu foo R 1 1 47.15s corona212
186+
187+
188+
We've submitted four jobs, giving them the job names "foo", "foobar", "boo", and "baz".
189+
190+
Lets cancel the job "boo" via ``flux pkill``
191+
192+
.. code-block:: console
193+
194+
$ flux pkill boo
195+
flux-pkill: INFO: Canceled 1 job
196+
197+
$ flux jobs
198+
JOBID USER NAME ST NTASKS NNODES TIME INFO
199+
ƒ6PjZG6jq achu baz R 1 1 2.856m corona212
200+
ƒ6Limcmju achu foobar R 1 1 2.97m corona212
201+
ƒ6KjHNcxP achu foo R 1 1 3.008m corona212
202+
203+
As you can see, ``flux pkill`` canceled just one job, the one assigned the name "boo".
204+
205+
``flux pkill`` will actually search for all jobs matching the supplied name, so what would happen if we asked ``flux pkill``
206+
to cancel jobs with the matching name "foo".
207+
208+
.. code-block:: console
209+
210+
$ flux pkill foo
211+
flux-pkill: INFO: Canceled 2 jobs
212+
213+
$ flux jobs
214+
JOBID USER NAME ST NTASKS NNODES TIME INFO
215+
ƒ6PjZG6jq achu baz R 1 1 4.626m corona212
216+
217+
As you can see it didn't cancel 1 job, it canceled 2 jobs, the job "foo" and the job "foobar".
218+
219+
And that's it! If you have any questions, please
220+
`let us know <https://github.com/flux-framework/flux-docs/issues>`_.

‎tutorials/commands/index.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ Welcome to the Command Tutorials! These tutorials should help you to map specifi
77
with your use case, and then see detailed usage.
88

99
- ``flux submit/flux run`` (:ref:`flux-submit`): "Submit a job in a Flux instance"
10+
- ``flux cancel/flux cancelall/flux pkill`` (:ref:`flux-cancel`): "Cancel a job you submitted"
1011
- ``flux proxy`` (:ref:`ssh-across-clusters`): "Send commands to a Flux instance across clusters using ssh"
1112

1213
This section is currently 🚧️ under construction 🚧️, so please come back later to see more command tutorials!
@@ -17,4 +18,5 @@ This section is currently 🚧️ under construction 🚧️, so please come bac
1718
:caption: Command Tutorials
1819

1920
flux-submit
21+
flux-cancel
2022
ssh-across-clusters

0 commit comments

Comments
 (0)