Add association-list-based helper functions into Rosette backend

[gussmith23]

What are the reasons/motivation for this change?

Currently, the Rosette code emitted by Yosys requires the user to ensure they correctly order module inputs when creating the "inputs" struct to be passed to the Rosette function generated for the module. This makes it difficult to use the backend in an automated way, and may lead to silent failures -- e.g., if we provide the right number of ports but in the wrong order, it won't be clear why our module isn't behaving as expected. (As an aside: even more confusingly, it seems like the backend may currently reverse the order of the ports!)

Similarly on the outputs side, it would be nice to have a way to access outputs via their name.

This change adds an optional helper functions (guarded by a flag) that can construct inputs/convert outputs to/from string-keyed association lists. Users can thus explicitly specify and access each input/output port by name. This allows users to be more confident that they are constructing the input struct correctly, and allows users to more easily access outputs by name.

Explain how this is achieved.

Two new Rosette/Racket functions are (optionally) written out. One wraps over the current inputs struct constructor, taking a string-keyed association list mapping input port names to values. The other takes an outputs struct and converts it into a string-keyed association list mapping output port name strings to values.

If applicable, please suggest to reviewers how they can test the change.

I would appreciate feedback on how to test. I see that the Python tests currently run the generated Racket files. Here are two options:

1. Add the new -keyword-constructor flag in the existing tests, which will not actually cause the keyword constructors to be used, but it will cause them to be generated. This will at least test that we do not have syntax errors.
2. Add another set of nearly-identical Python tests that instantiate the input structs via the helper function, rather than the original constructor.

Testing

I test this PR by hooking into the existing tests for the Rosette backend. Essentially, for each test of the Rosette backend, there are now two tests: one that doesn't use the association list helpers (ie, what the tests used to do) and a test that does use the helpers. This is achieved via pytest's parametrize. This effectively doubles the number of Rosette tests, which may not be wanted if testing time is an issue.

TODO

• Add tests (see above, need guidance from maintainers)
• access_by_base_name feels unnecessary, but access doesn't work for my purposes because I can't actually get the IdString of the field name.

[gussmith23]

I am making one more change to this and then it will be ready for review. Converting to draft until then.

[gussmith23]

Ready for review.

[KrystalDelusion]

As an aside: even more confusingly, it seems like the backend may currently reverse the order of the ports!

Do you have an example of this? I remember they were backwards at one point so I had to reverse them, but they should be in the right order now:

module \$fa (A, B, C, X, Y);
input [WIDTH-1:0] A, B, C;
output [WIDTH-1:0] X, Y;

(struct _$fa_Inputs (A B C) #:transparent
  ; A (bitvector 1)
  ; B (bitvector 1)
  ; C (bitvector 1)
)
(struct _$fa_Outputs (X Y) #:transparent
  ; X (bitvector 1)
  ; Y (bitvector 1)
)

If there is any rearranging of ports I would suspect it's in Yosys or the frontend rather than the Rosette backend.

Similarly on the outputs side, it would be nice to have a way to access outputs via their name.

As far as I understand Rosette/Racket, this should be possible with (e.g.) _$fa_Outputs-X, similarly to how inputs are (internally) referenced by name (let ((A (_$fa_Inputs-A inputs))). IIRC that's why they're marked #:transparent for both input and output, and also why there are comments for each of the port names/expected types, though I totally agree that being able to construct the input struct from named values would be useful (and the current state is mostly a reflection of me being unfamiliar with Racket).

access_by_base_name feels unnecessary, but access doesn't work for my purposes because I can't actually get the IdString of the field name.

I think you want to iterate over auto output : ir.outputs() rather than auto name : output_struct.get_field_names(), because that'll give you the list of base names. But again, I'm not sure how necessary this is since you can already access outputs by name.

I would appreciate feedback on how to test.

I don't think adding the flag to the current tests is the right move, since that invites the possibility that the output stops being valid without the flag (though I realise this is already true for the -provides flag). I also don't think it's necessary to test the flag for the full suite of cells. I would say it's sufficient to add one test (or potentially a couple tests) specifically for checking that instantiation via the helper structs works. The current pytest framework may or may not be the best place to add that, but be aware that the functional tests are disabled by default and only run on the private runner during CI because of the additional requirements (though looking at it seems that even then the smtlib and rosette tests are skipped, which seems unideal...).

[gussmith23]

As an aside: even more confusingly, it seems like the backend may currently reverse the order of the ports!

Do you have an example of this? I remember they were backwards at one point so I had to reverse them, but they should be in the right order now:

module \$fa (A, B, C, X, Y);
input [WIDTH-1:0] A, B, C;
output [WIDTH-1:0] X, Y;

(struct _$fa_Inputs (A B C) #:transparent
  ; A (bitvector 1)
  ; B (bitvector 1)
  ; C (bitvector 1)
)
(struct _$fa_Outputs (X Y) #:transparent
  ; X (bitvector 1)
  ; Y (bitvector 1)
)

If there is any rearranging of ports I would suspect it's in Yosys or the frontend rather than the Rosette backend.

Example: https://github.com/gussmith23/lakeroad/blob/50226012951d66190ea0e36a67c75035943b9d31/modules_for_importing/xilinx_ultrascale_plus/DSP48E2.v

yosys -p "read_verilog -sv DSP48E2.v; prep; clk2fflogic; write_functional_rosette -provides -assoc-list-helpers"

This should produce a file with the ports reversed; that's true on my machine at least. If it's not true on yours...something extra weird is going on!

Similarly on the outputs side, it would be nice to have a way to access outputs via their name.

As far as I understand Rosette/Racket, this should be possible with (e.g.) _$fa_Outputs-X, similarly to how inputs are (internally) referenced by name (let ((A (_$fa_Inputs-A inputs))). IIRC that's why they're marked #:transparent for both input and output, and also why there are comments for each of the port names/expected types, though I totally agree that being able to construct the input struct from named values would be useful (and the current state is mostly a reflection of me being unfamiliar with Racket).

You're right that the accessors _$fa_Outputs-X are generated. However, they're hard to use programmatically. Imagine we have a module we're trying to use from Rosette, but we don't actually know its name/its ports at the time of writing the code. It would be natural, then, to take the name and ports list as strings. If we have an output port name o as a string "o", how do we actually use the accessor mod_Outputs-o? We have to generate the symbol for the function using quasiquoting and then eval it -- see https://docs.racket-lang.org/guide/eval.html. It's possible, but it's very worth avoiding, especially when there's easy ways around it.

As an aside, I really do feel like this should be easier in Racket. I'm shocked that there's not a default way to look up a field on a struct if you know the field's name as a string. It could be that I'm totally wrong about this, though; maybe it's easy!

access_by_base_name feels unnecessary, but access doesn't work for my purposes because I can't actually get the IdString of the field name.

I think you want to iterate over auto output : ir.outputs() rather than auto name : output_struct.get_field_names(), because that'll give you the list of base names. But again, I'm not sure how necessary this is since you can already access outputs by name.

Thanks! Fixed.

I would appreciate feedback on how to test.

I don't think adding the flag to the current tests is the right move, since that invites the possibility that the output stops being valid without the flag (though I realise this is already true for the -provides flag). I also don't think it's necessary to test the flag for the full suite of cells. I would say it's sufficient to add one test (or potentially a couple tests) specifically for checking that instantiation via the helper structs works. The current pytest framework may or may not be the best place to add that, but be aware that the functional tests are disabled by default and only run on the private runner during CI because of the additional requirements (though looking at it seems that even then the smtlib and rosette tests are skipped, which seems unideal...).

Any concrete suggestions on where I should put the tests? Should I not write them in pytest? Should I add them as "functional tests"? Are those the same things as the pytest tests?

[KrystalDelusion]

...
This should produce a file with the ports reversed; that's true on my machine at least. If it's not true on yours...something extra weird is going on!

Huh. It's not just backwards, it's alphabetised too, but only from the functional backends (and not just Rosette). I wonder if it's related to them being stored as a Yosys::hashlib::dict, and potentially below a certain size the order is maintained but above that they get sorted?

However, they're hard to use programmatically. Imagine we have a module we're trying to use from Rosette, but we don't actually know its name/its ports at the time of writing the code.

Ah right, that is true, particularly with the renaming/escaping that happens when it creates a unique name. In that case it does make sense to have an output converter as well.

Any concrete suggestions on where I should put the tests? Should I not write them in pytest? Should I add them as "functional tests"? Are those the same things as the pytest tests?

The tests/functional directory is gated by the make flag ENABLE_FUNCTIONAL_TESTS. Currently the tests/functional/run-test.sh script is just a single call to pytest -v -m "not smt and not rkt" "$@", which runs pytest, but skips the smt and rkt (Rosette) tests. I think what happened is that the smt and rkt tests were disabled because they have extra prerequisites that are not currently installed on the normal git runner, and would also break regular calls to make test without them. The ENABLE_FUNCTIONAL_TESTS was then added later, to limit running them to just the private runner which we also use to run the verific tests, meaning breaking make test is no longer a concern so the -m "not smt and not rkt" should be safe to remove.

Where you write the tests depends on if you prefer bash or python for scripting; if you prefer bash then you should put them in tests/functional/run-test.sh, if instead you prefer python then you're welcome to use the pytest infrastructure that already exists in that directory. In either case, any extra .v or .il files you need can go in the same directory.

[KrystalDelusion]

hierarchy; write_functional_rosette gives A B C.
prep; write_functional_rosette gives C B A... Looks like it's opt_clean that sorts them, but I'm unclear on why this affects the functional backends but not the others.

EDIT:
Ah, write_verilog uses module->ports which maintains order, the functional IR generator uses module->wires() (and then tests wire->port_input and wire->port_output), which gets sorted for faster look ups.

I can get it to not alphabetise if I use module->ports in functional.cc, but Yosys::hashlib::dict iterates in reverse??

iterator operator++() { index--; return *this; }

Relatively straightforward to solve in c++20 with for (auto input : ir.inputs() | std::views::reverse)... Less so in c++17

[gussmith23]

@KrystalDelusion I'm working on tests now, I will re-request you when it's ready! Feel free to unsubscribe for now.

[gussmith23]

@KrystalDelusion I have updated this PR and it's ready for another review. I have also added testing by hooking into the existing tests, adding an arg which controls whether this feature is used.

[gussmith23]

Failures from the last CI run look like this:

2025-07-01T22:31:26.7349954Z FAILED test_functional.py::test_rkt[not-6-12-S-True] - FileNotFoundError: [Errno 2] No such file or directory: 'racket'
2025-07-01T22:31:26.7350249Z FAILED test_functional.py::test_rkt[not-6-12-S-False] - FileNotFoundError: [Errno 2] No such file or directory: 'racket'
2025-07-01T22:31:26.7350541Z FAILED test_functional.py::test_rkt[not-6-12-U-True] - FileNotFoundError: [Errno 2] No such file or directory: 'racket'
2025-07-01T22:31:26.7350888Z FAILED test_functional.py::test_rkt[not-6-12-U-False] - FileNotFoundError: [Errno 2] No such file or directory: 'racket'
2025-07-01T22:31:26.7351214Z FAILED test_functional.py::test_rkt[not-32-16-S-True] - FileNotFoundError: [Errno 2] No such file or directory: 'racket'
...

@KrystalDelusion or whoever it may concern: it seems like the functional tests are getting run under the Verific tests -- is that expected? It seems like one of two options:

1. Verific testing workflow always ran the functional tests, and my changes broke them
2. Verific testing workflow didn't previously run the functional tests, and my changes somehow made them start running the functional tests

2 seems more likely to me, namely because the failure is so basic -- ie that Racket isn't installed in the environment. That implies to me that the Verific testing workflow isn't meant to run the functional tests.

[KrystalDelusion]

I also believe it is the latter, and mentioned earlier that the smt and rkt tests were being skipped and we should enable them so uhh, that's probably on me for not checking if the prerequisites were installed 😛

[gussmith23]

what would you like me to do here?

…

On Tue, Jul 1, 2025, 6:14 PM KrystalDelusion ***@***.***> wrote: *KrystalDelusion* left a comment (YosysHQ/yosys#5128) <#5128 (comment)> I also believe it is the latter, and mentioned earlier that the smt and rkt tests were being skipped and we should enable them so uhh, that's probably on me for not checking if the prerequisites were installed 😛 — Reply to this email directly, view it on GitHub <#5128 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJFZANELOSCL44JYIPT2YD3GMW5ZAVCNFSM6AAAAAB5K7ZBF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAMRWGAYTCMBVGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[KrystalDelusion]

what would you like me to do here?

Hopefully nothing, runner has just been updated and the test is re-running

[KrystalDelusion]

I think a couple extra tests (separate from test_rkt) could be added for checking failure states; like using the keyword helpers but then providing the wrong field names.

[KrystalDelusion]

Unless I'm missing it, this file isn't used anywhere (and has nothing in it).

[KrystalDelusion]

Is it possible for this error to occur?

[KrystalDelusion]

What makes this a destructor?

[KrystalDelusion]

@gussmith23 Can you take a look at the comments I left when you have a chance? Tests are running fine with parallelism enabled on CI now (shown on a test merge because the job can't be re-run anymore)