Merge pull request #11870 from github/hmakholm/pr/mergeback-2.12.0

Merge codeql-cli-2.12.0 back to main

Author: hmakholm

docs/codeql/codeql-cli/about-codeql-packs.rst

@@ -9,8 +9,8 @@ CodeQL packs are used to create, share, depend on, and run CodeQL queries and li
 
 There are two types of CodeQL packs: query packs and library packs.
 
-* Query packs are designed to be run. When a query pack is published, the bundle includes all the transitive dependencies and a compilation cache. This ensures consistent and efficient execution of the queries in the pack.
-* Library packs are designed to be used by query packs (or other library packs) and do not contain queries themselves. The libraries are not compiled and there is no compilation cache included when the pack is published.
+* Query packs are designed to be run. When a query pack is published, the bundle includes all the transitive dependencies and pre-compiled representations of each query, in addition to the query sources. This ensures consistent and efficient execution of the queries in the pack.
+* Library packs are designed to be used by query packs (or other library packs) and do not contain queries themselves. The libraries are not compiled separately.
 
 You can use the package management commands in the CodeQL CLI to create CodeQL packs, add dependencies to packs, and install or update dependencies. For more information, see ":ref:`Creating and working with CodeQL packs <creating-and-working-with-codeql-packs>`." You can also publish and download CodeQL packs using the CodeQL CLI. For more information, see ":doc:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`."
 
@@ -31,6 +31,16 @@ The other files and directories within the pack should be logically organized. F
 - Queries for specific products, libraries, and frameworks are organized into
   their own top-level directories.
 
+About published packs
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When a pack is published for use in analyses, the ``codeql pack create`` or ``codeql pack publish`` command verifies that the content is complete and also adds some additional pieces of content to it:
+
+* For query packs, a copy of each of the library packs it depends on, in the precise versions it has been developed with. Users of the query pack won't need to download these library packs separately.
+* For query packs, precompiled representations of each of the queries. These are faster to execute than it would be to compile the QL source for the query at each analysis.
+
+Most of this data is located in a directory named ``.codeql`` in the published pack, but precompiled queries are in files with a ``.qlx`` suffix next to the ``.ql`` source for each query. When analyzing a database with a query from a published pack, CodeQL will load these files instead of the ``.ql`` source. If you need to modify the content of a *published* pack, be sure to remove all of the ``.qlx`` files, since they may prevent modifications in the ``.ql`` files from taking effect.
+
 About ``qlpack.yml`` files
 --------------------------
 

docs/codeql/codeql-cli/creating-and-working-with-codeql-packs.rst

@@ -14,11 +14,13 @@ With CodeQL packs and the package management commands in the CodeQL CLI, you can
 
 There are two types of CodeQL packs: query packs and library packs.
 
-* Query packs are designed to be run. When a query pack is published, the bundle includes all the transitive dependencies and a compilation cache. This ensures consistent and efficient execution of the queries in the pack.
-* Library packs are designed to be used by query packs (or other library packs) and do not contain queries themselves. The libraries are not compiled and there is no compilation cache included when the pack is published.
+* Query packs are designed to be run. When a query pack is published, the bundle includes all the transitive dependencies and pre-compiled representations of each query, in addition to the query sources. This ensures consistent and efficient execution of the queries in the pack.
+* Library packs are designed to be used by query packs (or other library packs) and do not contain queries themselves. The libraries are not compiled separately.
 
 You can use the ``pack`` command in the CodeQL CLI to create CodeQL packs, add dependencies to packs, and install or update dependencies. You can also publish and download CodeQL packs using the ``pack`` command. For more information, see ":doc:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`."
 
+For more information about compatibility between published query packs and different CodeQL releases, see ":ref:`About CodeQL pack compatibility <about-codeql-pack-compatibility>`."
+
 Creating a CodeQL pack
 ----------------------
 You can create a CodeQL pack by running the following command from the checkout root of your project:
@@ -81,3 +83,13 @@ This command downloads all dependencies to the shared cache on the local disk.
    By default ``codeql pack install`` will install dependencies from the Container registry on GitHub.com.
    You can install dependencies from a GitHub Enterprise Server Container registry by creating a ``qlconfig.yml`` file.
    For more information, see ":doc:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`."
+
+Customizing a downloaded CodeQL pack
+---------------------------------------------------
+
+The recommended way to experiment with changes to a pack is to clone the repository containing its source code.
+
+If no source respository is available and you need to base modifications on a pack downloaded from the Container registry, be aware that these packs are not intended to be modified or customized after downloading, and their format may change in the future without much notice.  We recommend taking the following steps after downloading a pack if you need to modify the content:
+
+- Change the pack *name* in ``qlpack.yml`` so you avoid confusion with results from the unmodified pack.
+- Remove all files named ``*.qlx`` anywhere in the unpacked directory structure. These files contain precompiled versions of the queries, and in some situations CodeQL will use them in preference to the QL source you have modified.

docs/codeql/codeql-cli/publishing-and-using-codeql-packs.rst

@@ -53,6 +53,9 @@ To run a pack that someone else has created, you must first download it by runni
 
 This command accepts arguments for multiple packs.
 
+If you write scripts that specify a particular version number of a query pack to download, keep in mind that when you update your version of CodeQL to a newer one, you may also need to switch to a newer version of the query pack.  Newer versions of CodeQL *may* provide
+degraded performance when used with query packs that have been pinned to a very old version. For more information, see ":ref:`About CodeQL pack compatibility <about-codeql-pack-compatibility>`."
+
 Using a CodeQL pack to analyze a CodeQL database
 ------------------------------------------------
 
@@ -74,6 +77,45 @@ The ``analyze`` command will run the default suite of any specified CodeQL packs
 
    codeql <database> analyze <scope>/<pack> <scope>/<other-pack>
 
+.. pull-quote::
+
+   Note
+
+   The ``codeql pack download`` command stores the pack it downloads in an internal location that is not intended for local modification.  Unexpected (and hard to troubleshoot) behavior may result if the pack is modified after downloading. For more information about customizing packs, see ":ref:`Creating and working with CodeQL packs <creating-and-working-with-codeql-packs>`."
+
+.. _about-codeql-pack-compatibility: 
+
+About CodeQL pack compatibility
+-------------------------------
+
+When a query pack is published, it includes pre-compiled representations of all the queries in it. These pre-compiled queries are generally much faster to execute than it is to compile the QL source from scratch during the analysis. However, the pre-compiled queries also depend on certain internals of the QL evaluator, so if the version of CodeQL that performs the analysis is too different from the version that ran ``codeql pack publish``, it may be necessary to compile the queries from source instead during analysis. The recompilation happens automatically and will not affect the *results* of the analysis, but it can make the
+analysis significantly slower.
+
+It can generally be assumed that if a pack is published with one release of CodeQL, the precompiled queries in it can be used directly by *later* releases of CodeQL, as long as there is no more than 6 months between the release dates. We will make reasonable efforts to keep new releases compatible for longer than that, but make no promises.
+
+It can also be assumed that a pack published by the *latest* public release of CodeQL will be useable by the version of CodeQL that is used by code scanning and GitHub Actions, even though that is often a slightly older release.
+
+As an exception to the above, packs published with versions of CodeQL *earlier than 2.12.0* are not compatible with any earlier or later versions. These old versions did not write pre-compiled queries in a format that supported compatibility between releases. Packs published by these versions can still be *used* by newer versions, but the analysis will be slower because the queries have to be recompiled first.
+
+As a user of a published query pack, you can check that the CodeQL makes use of the precompiled queries in it by inspecting the terminal output from an analysis runs that uses the query pack. If it contains lines looking like the following, then the precompiled queries were used successfully:
+
+::
+
+   [42/108] Loaded /long/path/to/query/Filename.qlx.
+
+However, if they instead look like the following, then usage of the precompiled queries failed:
+
+::
+   
+   Compiling query plan for /long/path/to/query/Filename.ql.
+   [42/108 comp 25s] Compiled /long/path/to/query/Filename.ql.
+
+The results of the analysis will still be good in this case, but to get optimal performance you may need to upgrade to a newer version of the CodeQL CLI and/or of the query pack.
+
+If you publish query packs on the Container registry on GitHub.com for others to use, we recommend that you use a recent release of CodeQL to run ``codeql pack publish``, and that you publish a fresh version of your pack with an updated CodeQL version before the version you used turns 6 months old. That way you can ensure that users of your pack who keep *their* CodeQL up to date will benefit from the pre-compiled queries in your pack.
+
+If you publish query packs with the intention of using them on a GitHub Enterprise Server installation that uses its bundled CodeQL binaries, use the same CodeQL version to run ``codeql pack publish``. Newer versions might produce pre-compiled queries that the one in GitHub Enterprise Server may not recognize. Your GitHub Enterprise Server administrator may choose to upgrade to a newer version of CodeQL periodically. If so, follow their lead.
+
 .. _working-with-codeql-packs-on-ghes:
 
 Working with CodeQL packs on GitHub Enterprise Server