[META] Decouple interface generation pipeline from CMake

[hidmic]

Feature request

Feature description

As it stands, the rosidl interface generation pipeline is strongly coupled with CMake, and in particular with ament_cmake machinery. There's no easy way for external projects using different build systems (or build systems' generators) to generate their own interfaces, or provide their own generators and/or type support packages, but to delegate to a CMake project.

Pushing as much logic and data (e.g. expected output from any given script) into reusable libraries and scripts would greatly ease integration with other build systems. Most code generation already takes place in separate Python scripts that could be used as a starting point.

In addition to that, though orthogonal, making type support dynamic loading machinery optional may further simplify the process for users that do not need such functionality.

[hidmic]

I'll be putting together a concrete design proposal for a change in the coming weeks.

Early input on specific use cases that folks think warrant support via pipeline generalization is most welcomed. Perhaps @EricCousineau-TRI, @IanTheEngineer, @jacobperron, @koonpeng, or @gbiggs may have some.

[jacobperron]

Thanks for the ping!

One feature that is missing in the current pipeline is the ability to retroactively generate interfaces for a new language. For example, in order to generate support for rcljava (and perhaps other client libraries like rclrs), we need to compile interface packages (e.g. rcl_interfaces) from source.

Ultimately, it would be nice if a user compiling rosidl_generator_java didn't also have to clone all of the interfaces that they plan to use into the same workspace (i.e. a tool should detect any interface packages installed in an underlay and generate language support for them). Taking that one step further, it would be nice if I could sudo apt install ros-rolling-rosidl-generator-java and magically have Java language support for existing (and future) interface packages that are installed on my system. This step might be significantly more challenging depending on the design, I haven't thought about it too much.

[EricCousineau-TRI]

Thanks for filing this! I think we should be sure to monitor performance at the forefront - I'd assume for primarily Tier I platforms.

The main concern is that having CMake do a whole bunch of subprocess dispatch may slow things down a bit, esp. on Windows (DISCLAIMER: I currently have zero stake in Windows use cases, but just wanna make sure we don't disservice peeps).
At a minimum, we should identify a good benchmark to (try to) track any slowdowns this may incur (shifting complex logic from CMake to Python / whatevs), and ensure we can easily rerun those baselines.

Are there already existing benchmarks like that? (e.g. recording metrics from ament about configuration time in CMake?)

And dumb question: Can we assume that any system building ROS 2 components will have access to Python? (I think Python's a prereq, but just wanna make sure!)

[jacobperron]

Can we assume that any system building ROS 2 components will have access to Python?

Python is currently listed as a dependency in REP 2000, though there are efforts to define a "minimal" C++-only installation (see ros-infrastructure/rep#231). So, I suppose Python is not (or in the future may not be) a strict dependency for a system running ROS. But, I think Python is a reasonable requirement for the purposes of generating and building ROS messages. It would basically require an entire rewrite of the interface generation pipeline if we dropped Python (for better or worse).

[clalancette]

But, I think Python is a reasonable requirement for the purposes of generating and building ROS messages.

Yeah, I agree; requiring Python on the build machine seems fine. We just want to make sure it isn't required on the target (unless you are using rclpy and friends, of course).

[koonpeng]

This is my experience when working on a rosidl_generator_nodejs. A third party rosidl generator would probably need the following:

1. A list of all packages and their messages, services and actions.
2. A way to parse those message definitions.
3. The include dirs and link libraries for each package/message.

The list of all packages can be easily gotten from the ament index, but "2" and "3" are more complicated. The only way I know of to parse the message definitons is using rosidl_parser which requires python. For "3", the only reliable way currently is to use cmake. Since a 3rd party rosidl generator would probably need to generate the interfaces at "install time" or at "build time of the consuming application", that brings in depedency on python and cmake somewhere between run time and build time.

It would also be nice if a 3rd party idl generator can easily make reliable incremental builds, we will need to have a way that allows arbitrary build systems to detect changes and selectively rebuild only the messages that have changed.

---

After further testing, I realized that this may affect more than just build systems. There are multiple places in the ros ecosystem that assumes that a message's library can be found in <lib_prefix><package_name>__* (maybe because there is no way to get the library?), however that is not a valid assumption, it is possble for a "valid" message to not follow that convention.

https://github.com/koonpeng/ros_msg_with_diff_pkg_name This is an example message that exports it's library in <lib_prefix>some_other_package__*. Another message that does not follow the convention is libstatistics_collector/msg/DummyMessage, included as part of the base ros2 installation. It builds fine but when trying to publish this message with ros2 cli, you get the error

>>> [rcutils|error_handling.c:108] rcutils_set_error_state()
This error state is being overwritten:

  'Failed to find library 'ros_msg_test__rosidl_typesupport_fastrtps_c'
, at /tmp/binarydeb/ros-foxy-rosidl-typesupport-c-1.0.1/src/type_support_dispatch.hpp:73'

with this new error message:

  'Handle's typesupport identifier (rosidl_typesupport_c) is not supported by this library
, at /tmp/binarydeb/ros-foxy-rosidl-typesupport-c-1.0.1/src/type_support_dispatch.hpp:116'

rcutils_reset_error() should be called after error handling to avoid this.
<<<

>>> [rcutils|error_handling.c:108] rcutils_set_error_state()
This error state is being overwritten:

  'Handle's typesupport identifier (rosidl_typesupport_c) is not supported by this library
, at /tmp/binarydeb/ros-foxy-rosidl-typesupport-c-1.0.1/src/type_support_dispatch.hpp:116'

with this new error message:

  'type support not from this implementation, at /tmp/binarydeb/ros-foxy-rmw-fastrtps-cpp-1.2.4/src/publisher.cpp:86'

rcutils_reset_error() should be called after error handling to avoid this.
<<<
Failed to create publisher: type support not from this implementation, at /tmp/binarydeb/ros-foxy-rmw-fastrtps-cpp-1.2.4/src/publisher.cpp:86, at /tmp/binarydeb/ros-foxy-rcl-1.1.10/src/rcl/publisher.c:180

This also fails with rclpy and even rclcpp. I haven't been able to test this with ros2bag but I think it would likely fail as well. Given that none of the ros tools work with this message, maybe it was never intended for such a message to be "valid", in this case I would suggest adding a check in the cmake macro/functions to prevent such message from being built.

[ros-discourse]

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/common-messages-github-organization/18028/14

[hidmic]

Thank you all for the feedback. I'm still trying to put all the pieces together (this one's really tough 😅). In what follows, I'll reply while I layout my current train of thought.

---

As I understand it, ROS software is organized in packages. A package contains source code and data files, along with a build system (or build system generator) to turn that into usable artifacts (in the most general sense e.g. an executable binary, a shell script, a dynamic library, a Python module, a Java JAR file, etc.). Packages are the smallest distributable unit. Each package is defined in one place and built in full. Packages are usually built in groups using workspaces. There can only be one version or variation of a package in a workspace. However, workspaces can be overlayed such that a version or variation of a given package in the overlay takes precedence over a different version or variation of the same package in the underlay.

Our rosidl pipeline for interface generation fits right into that model. It is a collection of build system tools for parsing interface definitions, generating source code, building libraries, and installing and exporting these libraries, that the build system of any given package (well, currently any ament_cmake package) can use. Interfaces are always part of a package.

I intend to stick to this organization model.

---

Ultimately, it would be nice if a user compiling rosidl_generator_java didn't also have to clone all of the interfaces that they plan to use into the same workspace

@jacobperron I don't think this feature belongs to the pipeline. Interfaces in core packages that are distributed in binary underlays are already built, and rosidl_generator_java is a build system tool. You wouldn't expect upgrading cmake to trigger a re-build. And even then, you would still need all the sources and an install location for new artifacts, which have to belong to a package anyway.

I think pulling package sources into an overlay is the simplest, cleanest, and most explicit solution. Perhaps what we need here is better automation to generate custom overlays.

Are there already existing benchmarks like that? (e.g. recording metrics from ament about configuration time in CMake?)

@EricCousineau-TRI AFAIK no, there are no configuration-time nor build-time benchmarks.

And dumb question: Can we assume that any system building ROS 2 components will have access to Python?

@EricCousineau-TRI As mentioned above, Python is a prerequisite according to REP-2000 and the current pipeline relies on it significantly. That is not a justification considering that we're trying to decouple this from cmake (also a prerequisite and heavily relied on). However, I personally think that keeping it as a build time dependency and even using it as grounds for pipeline extension (i.e. substituting ament_cmake extensions with Python entrypoints) is in our best interest. It will allow us to reuse existing code with reasonable runtime overhead.

A third party rosidl generator would probably need the following:

1. A list of all packages and their messages, services and actions.
2. A way to parse those message definitions.
3. The include dirs and link libraries for each package/message.

@koonpeng I don't agree. A rosidl generator should only be concerned about generating source code for an interface definition during a package build. It shouldn't need to crawl underlays in any way.

By your description of rosidl_generator_nodejs, I'm guessing that you want to generate nodejs bindings using the output of an specific generator in a entire underlay. That's not a package build system tool. It can be made to work, but you'll probably end up with some bindings that are completely detached from the ROS organization model.

I can think of two solutions that stay within the model. One is to re-build with a new generator. Automating custom overlay generation can definitely help. The other solution is to go full dynamic and generate the bindings on the fly, in runtime, in process memory space. That's assuming you can go without actually building anything, loading dynamic libraries and working with opaque types and binary blobs.

It would also be nice if a 3rd party idl generator can easily make reliable incremental builds

@koonpeng Agreed, but this is entirely up to the build system. Generators must only provide enough information for it to be possible.

There are multiple places in the ros ecosystem that assumes that a message's library can be found in <lib_prefix><package_name>__* (maybe because there is no way to get the library?), however that is not a valid assumption, it is possble for a "valid" message to not follow that convention.

@koonpeng I think this is a related but orthogonal problem. Currently, only cmake has access to upstream artifacts information (e.g. via cmake modules), thus why some downstream packages make assumptions like these. Even if we can circumvent it by enforcing file names, this fundamental limitation still shows itself whenever trying to depend on ROS packages from a non cmake build.

One potential solution would be to have packages use the ament index to register their artifacts. This isn't quite what it was meant for though. Alternatively, packages could export this information using formats that can be consumed by tools (e.g. a json dump) and build systems and build system generators (e.g. cmake modules, pkg-config files, bazel Starlark files, gradle Groovy files, etc.).

[koonpeng]

@jacobperron Thanks for the useful insights!

By your description of rosidl_generator_nodejs, I'm guessing that you want to generate nodejs bindings using the output of an specific generator in a entire underlay. That's not a package build system tool. It can be made to work, but you'll probably end up with some bindings that are completely detached from the ROS organization model.

Yeah, what I am building is not really a rosidl generator, but more of a bindings generator if it makes sense. The nodejs ecosystem is too different that makes it unrealistic to somehow tie the bindings into the ROS ecosystem. So my current solution if to crawl through the packages and generate bindings for each message at build time (of the consuming application), would this be the recommended approach for a third party "rcl" library that is distributed outside the ROS ecosystem?

[hidmic]

So my current solution if to crawl through the packages and generate bindings for each message at build time (of the consuming application), would this be the recommended approach for a third party "rcl" library that is distributed outside the ROS ecosystem?

That last bit is key. Yes, you can definitely do this, but in doing so you're effectively leaving the ROS ecosystem. Another ROS package won't be able to depend on and use that generated code (at least not through standard channels).

I think your use case simply requires ROS packages to export their artifacts in a way you can consume. And if you can afford doing this in Python, you get the interface parser for free.

[jacobperron]

I don't think this feature belongs to the pipeline. Interfaces in core packages that are distributed in binary underlays are already built, and rosidl_generator_java is a build system tool. You wouldn't expect upgrading cmake to trigger a re-build. And even then, you would still need all the sources and an install location for new artifacts, which have to belong to a package anyway.

Sure, but perhaps we don't need to think of it as a re-build. All a generator package needs are the interface definition files (e.g. .msg, .srv, .action, and .idl files; or maybe just the .idl files), which are already installed. With this, language-specific code can be generated and built, basically separate from the package where the interface definition files came from. The question becomes where to install the built artifacts? Maybe it's possible to install them back with the original package, or maybe to a dedicated package with a mangled name (e.g. sensor_msgs__rosidl_generator_java). I admit it's a half-baked idea 🤷‍♂️

I think pulling package sources into an overlay is the simplest, cleanest, and most explicit solution.

This works well for developing rosidl generators. I guess the problem I'd like to see addressed is related to releasing and distributing packages for other languages. AFAIU, we can't distribute binaries via the buildfarm for languages that are not part of the "core" ROS installation. This means many ROS packages that depend on client libraries other than rclcpp and rclpy cannot be released and distributed on ROS infrastructure. E.g. We can't bloom-release rcljava because it requires java bindings for rcl_interfaces (at least, I don't think it's possible at the moment). This problem seems coupled with the design of the rosidl generation pipeline. Perhaps it's out-of-scope for this ticket, but I thought it would be good to bring up so it's kept in mind.

[koonpeng]

AFAIU, we can't distribute binaries via the buildfarm for languages that are not part of the "core" ROS installation.

Yeah, this is the main problem I see as well. Even if we make it possible to release packages for other languages, it wouldn't help third party developers who don't have access to the build farm to include their rosidl generators when building the messages.

[hidmic]

All a generator package needs are the interface definition files (e.g. .msg, .srv, .action, and .idl files; or maybe just the .idl files), which are already installed.

This is not strictly true. Generators are provided with interface definition files plus their dependencies. Packages usually don't but could affect that generated code arbitrarily.

Maybe it's possible to install them back with the original package

That breaks the assumption that packages are the smallest distributable unit. Packages built and/or released would no longer be static, but implicitly depend on the state of your installation. I also wonder how we would handle generators that are already present in the underlay and for which there's generated code already. Would re-building rosidl_generator_cpp in workspace B replace files in underlay A? What if we want to rollback?

or maybe to a dedicated package with a mangled name

Would these dedicated packages have their own package.xml? If so, it'd break all downstream packages that depend on interfaces to be found on a given package. If not, we'd be breaking REP-122 (i.e. dropping the correlation between ROS package name and artifact location). We would have to guard ourselves against name clashes in install spaces too.

I guess the problem I'd like to see addressed is related to releasing and distributing packages for other languages. AFAIU, we can't distribute binaries via the buildfarm for languages that are not part of the "core" ROS installation. This means many ROS packages that depend on client libraries other than rclcpp and rclpy cannot be released and distributed on ROS infrastructure.

This problem seems coupled with the design of the rosidl generation pipeline.

To make sure I understand, is this because third-party generators are not present in our core builds and so core interfaces are not generated for these other languages? That's a very good point I had not thought about.

What if, for each interface package as we know it today, we had an interface-only package, a package for each language, and a "metapackage" using group dependencies and ament index lookups to tie it all up? I know it's a big (and verbose?) change, but it'd give us total flexibility to build and install only what's needed when it's needed.

For instance, a CMake package that depends on geometry_msgs would indirectly depend on the geometry_msgs_files package, the geometry_msgs_cpp package, the geometry_msgs_py package, etc. On find_package(geometry_msgs REQUIRED), it'd use the ament index to find these group dependencies and bring them in (those that offer CMake modules). If we install and export correctly, language prefixes won't show in source code e.g. install C++ headers to <install-prefix>/include/geometry_msgs_cpp/geometry_msgs/..., export <install-prefix>/include/geometry_msgs_cpp.

Incidentally, by making each language-specific interface package target a specific set of build system tools, we can configure them more precisely. It'd definitely help me address a subtle problem we have with interface versioning today: the same interface package version can expose completely different APIs if it was built against different build system tool versions. We can land a breaking change w/o touching the package.

[hidmic]

Tagging the @ros2/team, my last proposal will spark controversy.