Release Changes: Overview

[kflynn]

This discussion started with a Google doc with a bunch of changes being proposed for the next Gateway API release.

There is important background on why we need to change releases in the Google doc, but at this point, the Google doc is no longer maintained. This discussion is authoritative until we collect it into other documentation.

Overview of changes

1. We'll choose a date for standard-channel releases, but we won't choose the content.

   • The date should not slip: the content is allowed to be flexible.
   • Whatever content is ready to go at the date will ship; what content isn't will wait for the next release.
   • Ideally, we also pick the release number at the point its content is known.

2. To get a new item into the experimental channel, you need to get a Who/What/Why GEP merged as Provisional, and you need to line up implementations willing to sponsor the item by agreeing to implement it as an experimental feature.

   • Items affecting north-south aspects of Gateway API will need three sponsors. Items affecting only east-west aspects will need only two, because there are so few meshes involved with Gateway API that requiring three meshes is quite close to requiring unanimous support of the whole mesh ecosystem.
     • This is primarily shifting an existing requirement earlier in the process: you'll need implementations to go to standard, and in practice you need implementations to test things anyway, so let's just line sponsors up first.
   • We'll update the GEP template to include a Sponsors section.
   • The idea of having the Provisional GEP first is to give Gateway API better information about what the feature is before we agree to take it into experimental, and to give implementations good information about what they'll need to do before they agree to be sponsors.
     • The GEP can - and likely should! - be merged into Provisional before soliciting sponsors. There's no requirement that everything in this step happen in a single PR.

3. Once an item moves from Provisional to Implementable, it will be automatically dropped if it doesn't make any significant progress in six months.

   • "Significant" is deliberately vague. This is meant to be an easy bar to get over: if you're working on a thing, it won't get dropped. If no one is working on it, it'll fall out and clear bandwidth for something else.
   • There's no requirement that the original author be the one driving "progress". If you have to walk away from Gateway API for awhile, handing off to someone else is actively encouraged.
     • Honestly, even just letting people know that you have to go away for a bit but you expect to be back should be good enough.
   • This process will be handled with a public review, probably at the first meeting of every month.

4. We'll start doing monthly releases, tagged monthly-$year-$month (e.g. monthly-2025-10), of the experimental channel.

   • The point here is to make it faster to iterate in experimental.
   • Monthlies will only include experimental-install.yaml, because
   • Monthlies are not allowed to change the standard channel.
   • These will be snapshots of main. If you find a bug in a monthly, the solution is to upgrade to a newer monthly -- we won't do backports of fixes.
   • These deliberately don't use SemVer release numbers, because breaking changes to experimental resources and fields are always allowed between monthlies.
   • Note that we'll continue doing SemVer releases (1.5.0, 1.6.0, etc.) on a reqular cadence.
     • These will include both experimental-install.yaml and standard-install.yaml
     • These are not snapshots of main: they will have release branches to allow for backporting bugfixes.
     • Most likely, the experimental-install.yaml in a SemVer release will also be identical to the one contained in the monthly release right before it, though that's not required.

5. We'll use validating admission policies (VAP) to protect things.

   • An "upgrade VAP" will prevent applying experimental-channel CRDs over standard-channel CRDs.
     • If you really want to do this, you'll have to remove this VAP.
   • A "guardrails VAP" will prevent setting any experimental fields of a resource unless an annotation is also present.
     • Without the annotation, you get standard-channel functionality.

Sequencing

Changes 1, 2, and 3 are mostly just a matter of deciding to do them. With the exception of choosing the release number at the end, I think we mostly have consensus here.

• Change 1: irrespective of what we decide about release numbering, choosing to allow content to slip rather than allowing the date to slip doesn't require anything other than documentation.
• Change 2 requires adding a Sponsors section to the GEP template and a date field to the metadata.yaml.
• Change 3 would benefit from automation, but can be done by hand at first.

Change 4 requires release-engineering work plus the VAPs from change 5. This is extremely unlikely to happen before KubeCon in November.

What's Next?

Please reread the Release Changes doc since many of its comments are believed to be addressed. If you have a comment that is not addressed, please transfer it here (or, if you like, start a new discussion for it and drop the new discussion's link here). We'll try to reach what consensus we can asynchronously before, hopefully, concluding the discussion in the Gateway API meeting on 21 October.

Update from 21 October: after discussion today, we're still working on clarification, so keep updating this discussion as needed!

[youngnick]

Agreed on the changes and the sequencing, broadly.

I think that Change 2 does require adding a Sponsors section to the GEP template, but we could maybe add more than one field to the metadata.yaml schema - maybe a lastUpdated and a created date, or some similar thing, that will let us collect data on timings in the future.

[kflynn]

GitHub knows the last-updated date, and I'd strongly prefer that we use that rather than maintaining it by hand. But we will definitely need a Sponsors section in the GEP.

I like the idea of Sponsors including a link to the agreement, but honestly if the sponsor is the one making the change to list themselves in the GEP, that should be enough.

[candita]

Last-updated can come from someone posting PRs to fix typos or formatting, so we cannot rely on GitHub dates for an accurate analysis. I think we need the lastUpdated to differentiate real updates to content from updates for typos/formatting.

[kflynn]

For the purposes of "is this thing alive at all" I'm actually fine with typos/formatting fixes being counted as "alive" -- someone is paying attention. At least for the moment – we can revisit that later.

Of course, if it's harder to do that than just to look at the file, whatever. 😂

[snorwin]

I fully agree with changes 1–3. However, I have some reservations, particularly about removing the experimental channel from the release cycle and transforming it into monthly snapshots.

As mentioned in the comments of the Release Changes doc I’m wondering whether implementations would realistically be able to adopt and implement monthly experimental releases. IMO, there should be a clear commitment from at least five implementers to ensure that the intended goal of faster and broader feedback can actually be achieved.

[snorwin]

That’s a fair point, I hadn’t considered it from that perspective before.

The second concern is whether we can make these snapshots stable enough, and whether there are actual end users willing to install snapshot releases on their cluster. In our case, the most valuable feedback comes from users who are actively using the experimental channel.

[kflynn]

(Thanks for saying what I was going to, @mikemorris! 🙂)

[kflynn]

With respect to the stability of snapshots, I think there are probably three separate concerns worth teasing apart here.

1. A snapshot might have bugs.

   This is true: releases will have bugs at some point. Standard will too. We deal with this by normal software engineering, and by improving what tests we run before we cut a monthly.

2. Snapshots might somehow include a bunch of unintentional breakage.

   This is kind of getting at the idea of churn on main causing trouble. Even now, though, "experimental" doesn't mean "the wild wild West", and churning to no purpose is a thing we should avoid anyway.

   Ultimately, this gets back to "we need to be careful to keep main in a shippable state". We can, and should, figure out how to do more validation of this, but if you look over main it's really in pretty good shape here.

3. Snapshots might somehow break standard.

   This is the one I'm least concerned about, since we won't include standard-install.yaml in the monthlies.

Overall, I'm not terribly concerned about the monthlies being unstable just because they're monthlies. I'm frankly more concerned about making them cheap enough to do every month.

[snorwin]

@kflynn could we at least keep an experimental channel alongside the standard one for each release, even once we move to monthly snapshots?

Our users are unlikely to install snapshots, but they would probably continue using the experimental channel. That would at least give us the opportunity to collect more feedback from them.

In my view, the monthly snapshots are more targeted toward implementers and will naturally generate more feedback from that group. However, since we’re building the API for end users, I’d like to make sure they also have a chance to share their feedback.

[kflynn]

Sure, I don't see any reason we wouldn't ship both experimental-install.yaml and standard-install.yaml in SemVer releases. Seems like a good idea.

[shaneutt]

Strongly aligned on strengthening our "motivation-first proposals" philosophy.

Strongly aligned with making it easier, quicker and more iterative to work on Experimental GEPs, and Experimental should absolutely stay true to its namesake.

As we make Experimental easier and faster, we need to make Standard slower and more deliberate.

[kflynn]

Or at least it's OK to make Standard slower.

[mikemorris]

breaking changes are always allowed between monthlies

I think this needs the minor clarification of "breaking changes [to experimental fields or resources]" - we wouldn't be randomly breaking v1 standard channel fields or resources because of our compatibility policy.

[robscott]

Monthlies are not allowed to change the standard channel.

I think it may be clearer to say that "Monthly releases won't contain any standard channel artifacts". It may be impossible to prevent changes to standard channel on main, and if monthly releases are just tags of main at a predictable point in time, our best way to limit this is simply not to release any standard channel artifacts.

[mikemorris]

@robscott I'm not sure that's workable with the intent to include experimental fields on resources like HTTPRoute, unless you meant something other than excluding resources which have reached v1 already?

[robscott]

@mikemorris I'm basically just meaning that we'd exclusively include experimental-install.yaml in the release artifacts, not standard-install.yaml (see https://github.com/kubernetes-sigs/gateway-api/releases/tag/v1.4.0). For resources like HTTPRoute that are unfortunately still in the same API group for either channel, that will introduce some risk, but that's what we're trying to mitigate with the new VAPs.

[mikemorris]

Ah gotcha, that makes sense!

[kflynn]

🤦‍♂️ I'd never been thinking that a monthly would include standard-install.yaml, but this conversation made me realize that I'd never written that down anywhere. 🤦‍♂️ Updated the text again, take a look please! 🙂

[candita]

To get a new item into the experimental channel, you need to get a Who/What/Why GEP merged as provisional, and you need to line up three implementations (two for mesh) to sponsor it by committing to implement it as an experimental feature.

So, five implementations all together need to sign on? What if the feature is only meant for non-mesh or only meant for mesh? Or, do you mean if it's only valuable for mesh, then it only needs 2 implementations to sponsor it?

How about limiting the number of proposals that any one implementation can get behind, to one per release? If one implementation chooses to sponsor 2 or more features in a release, then how do we guarantee that one won't be dropped?

[candita]

And if the feature manages to be applicable for both north-south and east-west implementations?

[kflynn]

Then it's a north-south feature and needs three – and if it's complex enough that we're really concerned about things being vastly different between the two, I would expect the folks reviewing the GEP would ask for it to be split so that it can be tackled in stages.

[mikemorris]

If one implementation chooses to sponsor 2 or more features in a release, then how do we guarantee that one won't be dropped?

There is no guarantee that features won't be cut, that's kinda the point of choosing to prioritize release dates over release content. Even if an implementation is only sponsoring one feature it could be dropped if they're not able to prioritize GEP writing, API design and implementation.

And, at the moment, I'm OK with letting implementations decide how much bandwidth they have to spend on things

In prior cycles we have pushed back on individuals pitching 3+ features and suggested that they consider their priority and capacity to deliver if all would be selected. I think implementations sponsoring is a slightly lower bar as it's more a declaration of intent from project maintainers to review and implement, and different features sponsored by the same implementation may have involvement from separate individual humans.

There's somewhat two separate angles here - feature author and implementing project maintainer capacity, and Gateway API reviewer/maintainer capacity. Some projects may also want to register their support for a proposed feature, but won't consider implementing until it reaches the standard channel - I'm not quite sure if/how we should account for that. Recent cycles has biased toward GWAPI maintainer capacity by restricting inputs to the feature pipeline - this is an attempt to ease those input restrictions while still considering GWAPI maintainer capacity by limiting "what do I need to pay attention to now" for the upcoming fixed date instead of needing to follow everything in progress closely.

I think an explicit suggestion to consider maintainer bandwidth for sponsoring projects would be good, but I would maybe hold off on a formal constraint unless we see significant issues with too many features stalling from initial sponsorship but lack of follow-through. Lack of GWAPI reviewer/maintainer bandwidth is a bit of a different axis, where not having limits on sponsor capacity might actually be more helpful to allocate limited time, as it could help to inform prioritization by showing reviewers which features have broad demand from implementations. It's definitely a risk to watch out for though.

[mikemorris]

What if the feature is only meant for non-mesh or only meant for mesh?

I would agree that sponsors from the non-intended use case are not needed initially (e.g. I don't expect an AuthN proposal focused on user authentication/OAuth flows would require sponsorship from mesh implementations), but the potential implications or conflict/overlap with functionality for implementations in the other group do still need to be considered as part of the GEP process, especially as a feature advances towards standard - this is why explicit sections for N/S and E/W considerations were added to the GEP template in #3577.

[kflynn]

(Circling back here, I updated the language in the main discussion.)

[candita]

I would like to add to the record that requiring 3-5 sponsors after the Provisional stage makes it harder for those of us who are not implementers to get a feature accepted. Any feature proposed by an implementer has a head start on sponsors because they sponsor themselves. If we are translating sponsor numbers into "votes", then as an integrator, I have no vote in terms of sponsorship. I predict that eventually you won't have integrators or end-users participating because they have no vote.

I recommend that we allow integrators and/or users to sponsor proposals as well. They can participate in moving the feature along by reviewing, adding conformance tests, and other tasks that don't require an implementation. Ideally, three integrators can get together and be sponsors for features.

[virtualdxs]

Isn't it only 2 or 3 sponsors, depending on whether the feature is relevant to north/south traffic?