-
-
Notifications
You must be signed in to change notification settings - Fork 158
[RFC 0148] Pipe operator #148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nix is already considered to be overly complex by many. Of course we know that Nix solves hard problems and can't do so without exposing intrinsic complexity, but this needed complexity is already more than most people expect to have to deal with, coming from the broken but "easy" traditional methods.
This makes the addition of accidental complexity to Nix disproportionately harmful. It makes newcomers more likely to reject Nix, depriving themselves of a real solution to their packaging and deployment problems, while depriving us from valuable contributions.
I am opposed to the pipe function, and skeptical of a function application operator; especially one that reverses the evaluation order compared to normal function application.
rfcs/0148-pipe-operator.md
Outdated
|
||
## `builtins.pipe` | ||
|
||
`lib.pipe`'s functionality is implemented as a built-in function. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pipe
is not a good function because the order is not obvious. It's already available practically everywhere as lib.pipe
. It's also hardly ever used, so doing this for performance strips it of the last argument for making it a builtin.
Builtins need to satisfy more stringent requirements, they take resources from the Nix team, and they can never be changed or removed.
I would recommend to always use a library, which can change things arbitrarily, thanks to being locked or pinned.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's also hardly ever used
I think this is due to documentation issues rather than due to this function itself being bad. While it has a lot of potential of misuse, it can actually make code more readable if you put in the effort for your code to be readable from top to bottom.
I would recommend to always use a library, which can change things arbitrarily, thanks to being locked or pinned
This... makes sense, but then I'd make an argument that parts of nixpkgs.lib
that manipulate generic data should exist outside of builtins
and outside of nixpkgs
- in a separate repository. It is not weird: we already have library flakes like flake-utils
or flake-parts
.
This discussion is outside of the scrope of this RFC, though; I merely disagree that lib.pipe
not seeing much usage is due to technical reasons.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shepherds currently believe this conversation is resolved, outcome: inclined to leave pipe
in lib
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pipe is not a good function because the order is not obvious.
@roberth for what it’s worth, the commit where I introduced it has a rationale for the order in its message.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not a problem with your contribution, but with the constraints of the Nix syntax.
lib.pipe
is a necessary step, and its adoption shows that it is useful to many, so thank you for creating it!
rfcs/0148-pipe-operator.md
Outdated
## Change the `pipe` function signature | ||
|
||
There are many equivalent ways to declare this function, instead of just using the current design. | ||
For example, one could flip its arguments to allow a partially-applied point-free style (see above). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you could call this curried style.
While it is point free in the technical sense of not having a variable name to carry the data flow, point-free is generally only used in a context where higher order functions are used for more arbitrary data flows. For many, it also carries the connotation of the synonymously used pointless style.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've never heard that term and the resources I've read so far all talked about "point-free" programming style. Google does not seem to know much either, so some links would be appreciated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The flipped pipe function is just function composition on a list, eta expanded. Eta expansion / reduction do not change the meaning, so these two functions would be indistinguishable.
That means that we can divide the problem and drop some unnecessary terminology.
pipe = flip composeFunctions
(andcomposeFunctions = flip pipe
by virtue of aflip
law)composeFunctions
is isomorphic topipe
composeFunctions
returns a function, which makes it easier to use where a function is expected: the application of higher order functions such asmap
.
A lot of that is just overly formal thought. Instead we can simplify this section to:
flip pipe
is equivalent to function composition applied to a list. By using a function that composes a list of functions instead of pipe, we get back a function, which can be readily used in higher order functions such as map
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
point-free is also point-less, in a semantic sense 😛
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shepherds currently believe this conversation can be resolved by accepting or rejecting the suggestion below.
I am very well aware of the language's complexity. I am also aware of the fact that some of its users are very technical and into functional programming languages, and for many others this is a first contact with these concepts. This is why I suggest picking only one of these operators as a compromise. I don't have any overly strong preferences here, but I make a point against function composition instead of argument piping, because the former leads to the so-called "point-free" programming style which tends to be more confusing for new users. I picked I strongly oppose calling this proposal "accidental" complexity. I spent a lot of time thinking about the options and trying to balance expressiveness and complexity, finding a compromise that makes solving real Nix problems easier. The Nix language is full of weird quirks users eventually have to face: we even have a wiki article collecting such instances, and it is far from exhaustive. Feel free to call these accidental complexity. But nothing about this proposal is accidental. |
To be clear, I don't use accidental to describe this RFC. I only used it in accidental complexity, which I've used as a synonym for extrinsic complexity. I can see that you've put a lot of thought into it, and I respect that, but that does not necessarily make the change a net positive.
A language with more syntax may be easier to write, but is not more expressive unless the syntax comes with semantics that were not already covered by existing features and combinations of them.
What is a real Nix problem? At least the lack of a syntax won't end up on the quirks page. Are the quirks a problem? Probably. I'd be happy to see a proposal that simplifies the language or makes the syntax easier to learn, but those are hard problems with a lot of inertia, and up-front costs that aren't "repaid" for years into the future. |
And more people consider Nix awkward to read and write rather than being complex. Mostly because the standard library is poor and not discoverable.
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/pre-rfc-pipe-operator/28387/10 |
I disagree with your interpretation of the word "expressiveness". Because you can (almost) always already solve all existing problems with existing syntax in (almost) any programming language. That's the point of being Turing complete. Therefore the question then becomes, how well can a problem be expressed in the language. And I think this holds true even for changes that are purely syntactical, like this one. When adding new syntax to the language, one goal may indeed be to make it easier to write. Another one, and IMO much more important, is does it make the language easier to read. Depending on the feature and the language those may coincide, or one may have the other as a side benefit, or they may contradict each other. I recently learned Haskell, so I am very well aware of the trap of having a lot of powerful operators that allow it to write programs concisely and without parentheses, but which comes at the cost of readability.
Problems that one might face when writing Nix code, either for some flake/shell/system configuration or when contributing to Nixpkgs. What I mean with this, is that I don't want to solve problems that for example mostly occur when trying to solve AdventOfCode in Nix. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While the lib.pipe
function is pretty cool (thanks for introducing it to me!), I think going through with this RFC would take gigantic effort, and the positive results (if any) will be seen only after a few years. And I'm thinking those positive results aren't worth it, because they won't make Nixlang significantly easier to read for beginners, and they don't solve any technical issues with Nixpkgs or Nix.
rfcs/0148-pipe-operator.md
Outdated
|
||
## `builtins.pipe` | ||
|
||
`lib.pipe`'s functionality is implemented as a built-in function. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's also hardly ever used
I think this is due to documentation issues rather than due to this function itself being bad. While it has a lot of potential of misuse, it can actually make code more readable if you put in the effort for your code to be readable from top to bottom.
I would recommend to always use a library, which can change things arbitrarily, thanks to being locked or pinned
This... makes sense, but then I'd make an argument that parts of nixpkgs.lib
that manipulate generic data should exist outside of builtins
and outside of nixpkgs
- in a separate repository. It is not weird: we already have library flakes like flake-utils
or flake-parts
.
This discussion is outside of the scrope of this RFC, though; I merely disagree that lib.pipe
not seeing much usage is due to technical reasons.
rfcs/0148-pipe-operator.md
Outdated
like line numbers when some part of the pipeline fails. | ||
Additionally, it allows easy usage outside of Nixpkgs and increases discoverability. | ||
|
||
While Nixpkgs is bounds to minimum Nix versions and thus `|>` won't be available until |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is a great argument against going through with this RFC.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This would be an argument for never making any changes to improve the language at all, ever. Unless you see some shiny alternative language to which we'll realistically all migrate to within the next five years, any improvements to the Nix language are still expected to be beneficial.
I've actually thought about proposing this change for quite some time now, and the main thing that held me back was the idea is that Nix is a "lost cause" and that I should rather spending my energy towards some successor. But eventually, I realized that realistically, we'll be stuck with Nix for a long time to come, so better invest the energy to make it a bit more comfortable …
Also note that outside of Nixpkgs, the feature will be available almost immediately. There is a lot of Nix code outside of Nixpkgs, all these flakes, shells, and users' system configurations. Projects like home-manager, devenv, nur, etc. are all free to migrate to newer Nix versions at their own pace. We shouldn't ignore these.
Honestly, the most effort in this RFC is convincing people, and maybe also the coordination across many projects. But from the technical side of things, this is actually pretty easy to implement.
This only applies to Nixpkgs, see #148 (comment)
They also won't make it significantly harder for beginners to read Nix code IMO, while bringing quality of life improvements to everyday Nix development. I don't understand why only "technical" problems should be worth solving. |
This RFC is now open for shepherd nominations! |
@AndersonTorres because I don't know every programming language and just didn't come across OCaml during my research. Furthermore, the related work section is intended to roughly cover the design space, not to be an exhaustive list of every programming language. |
Updated the text according to some of the initial feedback. The more I think of it, the more I'm leaning towards also having |
I was thinking on it, precisely, two pipe operators! It makes the language more orthogonal: if we have The problems are:
|
I nominate @maralorn, who has a lot of experience with functional programming languages and programming language design. |
@roberth, what about you, would you like to be a shepherd? I'd rather like to have critical voices on board from the beginning … |
I am pretty certain there are a lot of people with more experience on this in the wider community. That doesn’t prevent me from having opinions on this particular bikeshed, though. 🤣 My availability depends on the timeframe here. I will be very busy in the next few weeks, after that I’d be on board. |
I've procrastinated writing this RFC for at least half a year, so waiting two more months won't be the end of the world |
I can do that |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/lix-mismatch-in-feature-name-compared-to-nix/59879/1 |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2025-02-17/60444/1 |
The pipe operators have been now implemented for quite a while, now I'd like to know about common usage patterns. Especially:
|
While writing open-location-code.nix, I frequently found myself reaching for Haskell's let
digitVal = digit: builtins.floor (builtins.stringLength (builtins.elemAt (builtins.match "(.*)${digit}.*" digits) 0) / 2);
# ...
lat' = builtins.floor ((lat + 90) * 8000);
long' = builtins.floor ((long + 180) * 8000);
encodeLatMsd = lat: valDigit (mod lat 20);
encodeLongMsd = long: valDigit (mod long 20); could have some parens removed: let
digitVal = digit: builtins.floor <| builtins.stringLength (builtins.elemAt (builtins.match "(.*)${digit}.*" digits) 0) / 2;
# ...
lat' = builtins.floor <| (lat + 90) * 8000;
long' = builtins.floor <| (long + 180) * 8000;
encodeLatMsd = lat: valDigit <| mod lat 20;
encodeLongMsd = long: valDigit <| mod long 20; On the other hand, I don't know if you would count open-location-code.nix as a "real-world use case"... Examining the results for the GitHub code search query {
"containers/registries.conf".source =
lib.mkForce
<| toml.generate "registries.conf" {
/* multi-line expression elided */
};
}; lib.mkIf config.shell.direnv.enableCompletionAutoloadingWorkaround
<| lib.mkOrder 2000 "..." There's also some more complex usage in an Advent of Code solution here: https://github.com/V1K1NGbg/Advent-Of-Code-2024/blob/755b780a3416dc03508a209dc8c072a8e2aa938c/2.2.nix, including this fun line using both concat [ (range 0 (n - 1) |> (map (elemAt xs))) ] <| groupsOf n (tail xs) |
Should the feedback be provided in this issue or through some other means? I don't want to spam here, but I do want to let the record show that I absolutely love this feature just as it is. |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2025-03-03/61097/1 |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2025-03-17/61795/1 |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2025-03-31/62432/1 |
RFCSC: @rhendric are there any updates on this PR? What needs to be done to move it forward? |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2025-04-28/63639/1 |
RFCSC: Since there has been little activity for quite a while, we'll draft this in the next RFCSC meeting (2 weeks from now), unless the shepherds indicate that they'll pick this back up soon. |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2025-05-12/64183/1 |
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2025-05-12/64183/5 |
What is needed to ratify this RFC? The feature is already available an an experimental feature; does it need to be made not experimental before the RFC is done, or can the RFC be considered done before then? Maybe it's also worth considering what criteria needs to be met for it to come out of being experimental? I'd propose that 1) being confident no changes are needed and 2) having waited a long enough period for feedback should be enough to make the feature stable. If the RFC will finish before the feature is stable, the RFC should make clear what criteria is needed for it to come out of stable. Otherwise we risk the feature remaining experimental indefinitely, because no one agrees what is needed for it to be stable 😁 |
See also #148 (comment) I think the pipe operator is pretty uncontroversial, however I am still unclear as to whether or not we want the backpipe operator. Symmetry and the occasional use case would speak in favor of that, ambiguities around a canonical formatting and readability as well as the precedence question against it. Additionally, the backpipe operator kind of intersects with function concatenation, so that would require more investigation and design work to figure out what is best here |
To me, the arguments against the back pipe weigh more heavily. |
As a personal anecdote, in my personal configs, Some (well, one) of these cases could be rejiggered to use someAttrSetFn
<| {
...
}
// lib.optionalAttrs predicate {
...
} Such expressions are often already within a few sets of multiple-line-spanning parentheses, and adding yet another parenthesis pair to the stack is substantially less readable to me than using |
It seems that everyone agrees with |
The text is a bit outdated, but this RFC is and always has been only about the forwards pipe operator. The backpipe is only discussed in prior art and the alternatives section. Implementing both pipe operators at once always has been a matter of experimentation to find out if it is actually worth it (turns out, a bit but not really). My suggestion would be to clean up the RFC and FCP it as-is, and leave the backpipe operator (or function concatenation) for someone else to champion. |
Are there any other programming languages that include or allow users to define an operator equivalent to If not, what makes Nix special so that we would want to do so? Are all those other languages wrong and we think we're doing language design better here? Is there any other purpose for which we might want to reserve the sequence of characters RFC author has the right to write the RFC however they want, but this is a troubling place to cleave the proposal in my opinion. The language with just |
Gleam only has the |
Okay, I'm not familiar with Gleam, so what makes Gleam special so that its creator made this design choice, and is Nix special in the same way? |
There's also
(?) denotes languages which don't seem to have backward pipe, but I have no experience with them beyond quick googling to prove/disprove this post. My guess is that I personally don't have any |
In OCaml, the equivalent of Elixir I don't know about either. Maybe there's something in the water in the Erlang ecosystem?
|
Thanks for pointing out my error about OCaml, I will update my previous post.
Makes perfect sense, thank you! i actually have replaced a lot of nested parentheses trading them for more lines with pipes aiming for a single line of code to do exactly one thing.
Also, for the record, I am not against the backward pipe. Forward pipe syntax has a special place in my heart and it's easy to see how the backward pipe could be similar to someone else. Nor am I making any statements about one syntax being superior to the other one :) |
I'm not sure why to be honest but I suspect it's similar to what was said prior.
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/rfcsc-meeting-2025-05-26/64791/1 |
Rendered
Discussion notice: please try to attach all discussions to a thread by using the code review feature. If your comment doesn't refer a specific line to attach to, use the header line instead.