Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion _episodes/02-storage-spaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,7 @@ See the next episode on [data management]({{ site.baseurl }}/03-data-management)
> ~~~
> {: .output}
> or 'UNKNOWN'
> (This command doesn't work on SL7 so use an AL9 window)
> (This command doesn't always work on SL7 so use an AL9 window)
{: .challenge}


Expand Down
12 changes: 10 additions & 2 deletions _episodes/03-data-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,14 @@ metacat namespace list # find likely namespaces

There are official looking ones like `hd-protodune-det-reco` and ones for users doing production testing like `schellma`. The default for general use is `usertests`

Creation of namespaces by non-privileged users is currently disabled. A tool is in progress which will automatically make one namespace for each user

### metacat web interface

Metacat also has a web interface that is useful in exploring file parentage [metacat gui](https://metacat.fnal.gov:9443/dune_meta_prod/app/gui)

### Example of finding reconstructed Monte Carlo

Let's look for some reconstructed Monte Carlo from the VD far detector.

~~~
Expand Down Expand Up @@ -498,14 +506,14 @@ which the locations of the file on disk and tape. We can use this to copy the f
It will complain because you haven't loaded all the information needed to read an artroot file but you should be able to read it.


> ## NOTE if you see a path in `/pnfs/usr/dune/tape_backed` at Fermilab or on eos, `ctapublic.cern.ch`
> ## NOTE if you see a path in `/pnfs/usr/dune/tape_backed` at Fermilab or on `eosctapublic.cern.ch`
> those are on tape and not generally accessible to the user. Try to get the file from the remaining one (in this case hep.manchester.ac.uk)
{: .callout}


## More finding files by characteristics using metacat

There isn't always an official dataset so you can also list files directly using metact.
There isn't always an official dataset so you can also list files directly using metacat.

To list raw data files for a given run:
~~~
Expand Down
35 changes: 18 additions & 17 deletions _extras/sl7_setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,23 +10,24 @@ keypoints:

( I put this command in a file called apptainer.sh so I don't have to retype all the time.)

### FNAL

~~~
/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer shell --shell=/bin/bash \
-B /cvmfs,/exp,/nashome,/pnfs/dune,/opt,/run/user,/etc/hostname,/etc/hosts,/etc/krb5.conf --ipc --pid \
/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-dev-sl7:latest
~~~
{: .language-bash}

### CERN

~~~
/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer shell --shell=/bin/bash \
-B /cvmfs,/afs,/opt,/run/user,/etc/hostname,/etc/krb5.conf --ipc --pid \
/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-dev-sl7:latest
~~~
{: .language-bash}
Start the Apptainer


> ## Choose your apptainer
>
> > ## gpvm apptainer
> > ~~~
> > {% include apptainer_gpvm.md %}
> > ~~~
> > {: .language-bash}
> {: .solution}
> > ## cern apptainer
> > ~~~
> > {% include apptainer_cern.md %}
> > ~~~
> > {: .language-bash}
> {: .solution}
{: .challenge}


## then do the following
Expand Down
8 changes: 5 additions & 3 deletions _includes/al9_token.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@

> ## RUCIO authentication does not work on AL9 for now
> Please use [SL7]({{ site.baseurl }}/sl7_setup) if you need to use rucio
> ## Note: The justin get-token method for authentication does not currently work on AL9
> The justin get-token command is not distributed on AL9/Spack currently.
> Please use [SL7]({{ site.baseurl }}/sl7_setup) if you need to use rucio.
>
> normal tokens (below) for xroot access do work
> normal tokens (below) for `xroot` access do work
>
{: .caution}

### getting a token for xroot access in AL9
Make certain you have [al9 set up]({{ site.baseurl }}/al9_setup)

<!-- Get rucio:
Expand Down
4 changes: 1 addition & 3 deletions _includes/apptainer_build.md
Original file line number Diff line number Diff line change
@@ -1,3 +1 @@
/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer shell --shell=/bin/bash \
-B /cvmfs,/exp,/nashome,/opt,/run/user,/etc/hostname,/etc/hosts,/etc/krb5.conf --ipc --pid \
/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-dev-sl7:latest
/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer shell --shell=/bin/bash -B /cvmfs,/exp,/nashome,/opt,/run/user,/etc/hostname,/etc/hosts,/etc/krb5.conf --ipc --pid /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-dev-sl7:latest
4 changes: 1 addition & 3 deletions _includes/apptainer_cern.md
Original file line number Diff line number Diff line change
@@ -1,3 +1 @@
/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer shell --shell=/bin/bash \
-B /cvmfs,/afs,/opt,/run/user,/etc/hostname --ipc --pid \
/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-dev-sl7:latest
/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer shell --shell=/bin/bash -B /cvmfs,/afs,/opt,/run/user,/etc/hostname --ipc --pid /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-dev-sl7:latest
4 changes: 1 addition & 3 deletions _includes/apptainer_gpvm.md
Original file line number Diff line number Diff line change
@@ -1,3 +1 @@
/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer shell --shell=/bin/bash \
-B /cvmfs,/exp,/nashome,/pnfs/dune,/opt,/run/user,/etc/hostname,/etc/hosts,/etc/krb5.conf --ipc --pid \
/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-dev-sl7:latest
/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer shell --shell=/bin/bash -B /cvmfs,/exp,/nashome,/pnfs/dune,/opt,/run/user,/etc/hostname,/etc/hosts,/etc/krb5.conf --ipc --pid /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-dev-sl7:latest
2 changes: 1 addition & 1 deletion _includes/sl7_setup_2025.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh

# do some data access setup
export IFDH_CP_MAXRETRIES=0\0\0\0\0 # no retries
export IFDH_CP_MAXRETRIES=0 # no retries
export METACAT_SERVER_URL=https://metacat.fnal.gov:9443/dune_meta_prod/app
export METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/dune

Expand Down
3 changes: 3 additions & 0 deletions _includes/sl7_token.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,6 @@ justin get-token
{: .language-bash}

You will have to do this sequence weekly as your justin access expires.

> ## Note:
> Despite the name of this command it gets you both a token and a special X.509 proxy and it is the latter you are actually using to talk to rucio in these SL7 examples