Simple YAML-backed modules and single-file script modules

[lihaoyi]

This PR implements a way to define "Simple Modules" based on mill.yaml files or single .java/.scala/.kt files with //| build headers. This should make Mill more appealing for small projects, where the use of a build.mill file adds significant boilerplate and complexity. Most small projects need minimal customization of the build, and so the full Scala .mill syntax provides no value over a more lightweight config-only approach. This PR also allows interop between simple YAML modules and custom module classes written in mill-build/src/, to allow a gradual transition to the more flexible programmatic configuration

Simple Module with mill.yaml

mill.yaml

extends: "mill.javalib.JavaModule.Simple"
mvnDeps:
- "net.sourceforge.argparse4j:argparse4j:0.9.0"
- "org.thymeleaf:thymeleaf:3.1.1.RELEASE"

test/mill.yaml

extends: "mill.javalib.JavaModule.Junit4"
moduleDeps: ["."]
mvnDeps:
- "com.google.guava:guava:33.3.0-jre"

> ./mill run --text hello
<h1>hello</h1>

> ./mill .:run --text hello # `.` for explicit root module, `:` as new external module separator


> ./mill test
...
+ foo.FooTests...simple ...  "<h1>hello</h1>"
+ foo.FooTests...escaping ...  "<h1>&lt;hello&gt;</h1>"

> ./mill test:testForked # explicit task

Single-File Module with .java, .scala, or .kt

Foo.scala

//| mvnDeps:
//| - "com.lihaoyi::scalatags:0.13.1"
//| - "com.lihaoyi::mainargs:0.7.6"
import scalatags.Text.all.*
import mainargs.{main, ParserForMethods}

object Foo {
  def generateHtml(text: String) = {
    h1(text).toString
  }

  @main
  def main(text: String) = {
    println(generateHtml(text))
  }

  def main(args: Array[String]): Unit = ParserForMethods(this).runOrExit(args)
}

> ./mill Foo.scala --text hello
compiling 1 Scala source to...
<h1>hello</h1>

> ./mill Foo.scala:run --text hello
<h1>hello</h1>

> ./mill show Foo.scala:assembly # show the output of the assembly task
".../out/Foo.scala/assembly.dest/out.jar"

> java -jar ./out/Foo.scala/assembly.dest/out.jar --text hello
<h1>hello</h1>

Implementation

• Each mill.yaml simple module or *.{java,scala,kt} script file on disk is instantiated into an ExternalModule in Mill, and can be configured via YAML data and depend on each other and on traditional "Programmatic Modules" (and vice versa). Each one instantiates a mill.simple.SimpleModule specified via extends:, and users can define their own concrete subclass of SimpleModule to use in their simple modules if the defaults do not do everything they want.

  • As they are just mill.Modules, you can run multiple simple modules or script files together with +, or run them together with programmatic modules, or use them with show or inspect. They also can depend on each other arbitrarily, since in the end they are all mill.Modules in the Mill daemon classloader

• Parsing the query selectors is done by first attempting a parse-and-resolution for normal programmatic modules and tasks, and only if that fails do we attempt a parse-and-resolution for simple yaml-backed modules. This preserves backwards compatibility by ensuring queries that resolved to programmatic modules in the past continue to do so unchanged, while allowing a concise syntax for referencing simple yaml-backed modules

• We re-use most of the infrastructure from Mill's existing YAML build headers, though refactored so the YAML metadata can apply to arbitrary modules rather than only the root meta-build module.

• As we need to instantiate these modules reflectively, we cannot use traits or abstract classes, and so the modules that mill.yaml files or single-file scripts use in extends need to be concrete classes. These are configured to take a SimpleModule.Config case class injected by Mill containing non-task metadata like the script file path and moduleDeps, though we can extend it to contain additional fields in future if necessary

  • Since we can no longer have abstract fields, mandatory fields within this concrete class modules need to be stubbed out with = Task { ??? }

TODO

• Simple equivalents of MavenModule/MavenKotlinModule/SbtModule
• More thorough examples of configuring simple modules
• Error reporting on invalid keys
• Docs overhaul
• Depending on simple modules from programmatic modules
• IDE support in mill.yaml files and build headers

Supersedes #5826

[lolgab]

Very cool feature! It's great not having to wait on the Scala compiler to compile the build.
I'd add some requests (that you can add to the TODOs if you want):

• Not require a build.mill
• Refresh the build when mill.yaml is changed.

[lefou]

If we somehow could avoid the need to have a concrete class, that would be cool. Instead of reflectively instantiating a class (which needs to be prepared for just this special use case), we want to use arbitrary existing Mill modules which are traits (either Mill provided like JavaModule or custom). We could ad-hoc generate the concrete module object from the yaml data. That would allow us to use the same templates in mill.yaml, single-file project Foo.scala or classic build.mill files.

[lihaoyi]

Code-generating backing Scala files for mill.yaml is definitely a possibility

It does mean we would not be able to avoid using the Scala compiler to compile things.

On the other hand, it would mean that we'd get the ability to refer to the mill.yaml code-generated modules statically, which is nice

But any error messages in the codegen would point at the generated code, which is less nice

Not obvious which way is better, but it's something we can consider

[lefou]

@lihaoyi

It does mean we would not be able to avoid using the Scala compiler to compile things.
...
But any error messages in the codegen would point at the generated code, which is less nice

I think the generated code should only use well tested code blocks, so the chance of a compile error is rather low, since if we managed to fully generate the code, we already know how to map each found YAML section to such a tested code block.

Recompilation is probably only needed, if the extends section changes. The generated task impls should just depend (task-wise) on the YAML and forward the respective data dynamically.

(We might even have pre-compiled backing impls for most polular traits like JavaModule and ScalaModule.)

[lefou]

We should move the discussion of the topic into a dedicated issue.

[lihaoyi]

CC @alexarchambault @arturaz if you have a moment to review this PR and feature proposal it would be great to get your thoughts as well

[lefou]

Due to time constraints, I couldn't make a thorough review. OTOH, I fear an accidental/premature merge of this feature before we had a proper discussion, which I think is necessary. I want to remind us that once, we merge, we need to stick to that feature, since it is rather tightly coupled (located inside the existing modules, special CLI handling, ...).

In addition to my previous comments and ideas, which are not directly related to this PR but to the general concept, here are some quick notes regarding this PR.

File names

The mill.yaml naming scheme feel incosistent with the current build.mill/package.mill scheme.

• root project: mill.yaml vs build.mill
• sub-project: mill.yaml vs. package.mill.

build.yaml/package.yaml would feel more consistent, although it does not contain the tool in the same.

Should we use build.mill.yaml/package.mill.yaml instead?

Option handling

Also, I think we should not skip the need to pass a --file/-f option for now. It feels premature, as it forces use to stick to that handling for a reasonable future, although we haven't gathered any experience and user feedback for that feature at all.

Specifying an alternative build file with --file/-f` is common practice for build tools and is probably not much a hassle. It would also allow us to mark it as "preview feature", so it is more explicit to users that it might change in future version, depending on the feedback and experience we got.

Simple modules

We should not make the simple modules like mill.javalib.JavaModule.Simple sub-classes or object-subs of the existing modules. For practical reasons. The existing module sources files are already large and overloaded. We should avoid a pattern that forces us to have even more stuff in a single source file. A separate class and file will IMHO ease maintenance and potential future changes. E.g. we might later come up with the ability to directly use module traits or decide to package it in a different jar.

To establish some naming pattern, we could place all the simple support files/classes in a simple sub-package, so mill.javalib.JavaModule.Simple becomes mill.javalib.simple.JavaModule.