feat: twisted Edwards curve support with new execution #1858
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Note: this PR is not targeting `main`. I've used `TODO` and `TEMP` to mark places in code that will need to be cleaned up before merging to `main`. Beginning the refactor of online memory to allow different host types in different address spaces. Going to touch a lot of APIs. Focusing on stabilizing APIs - currently this PR will not improve performance. Tests will not all pass because I have intentionally disabled some logging required for trace generation. Only execution tests will pass (or run the execute benchmark). In future PR(s): - [ ] make `Memory` trait for execution read/write API - [ ] better handling of type conversions for memory image - [ ] replace the underlying memory implementation with other implementations like mmap Towards INT-3743 Even with wasteful conversions, execution is faster: Before: https://github.com/openvm-org/openvm/actions/runs/14318675080 After: https://github.com/openvm-org/openvm/actions/runs/14371335248?pr=1559
- make `VmSegmentExecutor` generic on `<Mem, Ctx, Ctrl>` where: - `Mem`: struct that implements `GuestMemory` - `Ctx`: struct that stores host context during execution - `Ctrl`: struct that implements pre/post segment execution hooks, termination condition and instruction execution logic - add `TracegenVmSegmentExecutor` that implements the current execution flow - move segmentation strategies to new file
- deleting `Vm{Adapter,Core}Chip` traits
- no more records, directly use trace buffer
- jal_lui chip is a demonstration of the new changes with working unit
tests
- changed unit tester
- [x] need to add some dummy volatile memory to the tester to balance
based on touched addresses
…1590) - introduce a new generic `InsExecutorE1` trait - add `InsExecutor::execute_e1` for rv32im instructions
…1589) Co-authored-by: Alexander Golovanov <[email protected]>
- fix some loadstore tests - remove records - wrap unsafe memory read/writes into safe wrappers --------- Co-authored-by: Jonathan Wang <[email protected]>
closes INT-3839 --------- Co-authored-by: Ayush Shukla <[email protected]>
- make `Rv32HintStoreChip` use the `NewVmChipWrapper` - rename `SingleTraceStep` to `TraceStep` and update it to work for chips whose execution creates multiple trace rows - comment out criterion execute benchmarks for now
one line fix. now that we're only initializing `TracingMemory` with `new`, we should remove this line from `with_image`
remove `memory/offline.rs` as we aren't using it anymore. Delete `VmAdapterChip` trait and `VmChipWrapper` since we also aren't using them anymore.
Made the rv32im tests pass and made all the testing files to have the same testing interface. Deleted the `test_adapter`. Kept all the test cases unchanged. The only commented test case remaining is the `store` test to the address space 4, which is failing because currently memory accesses with block size 4 are not supported with the address space 4. All the test files have 3 types of tests: Positive, Negative, and Sanity tests. All the test files have 2 helper functions: `create_test_chip`, `set_and_execute`. An important thing to notice about negative tests when expecting an interaction fail (aka ChallangePhase error) is that ther might be an imbalance created for the wrong reasons. For example, there might be an imbalance on the range checker bus created by the interactions: [send 1] (sent from the chip_air) [receive 2] (the execution did `add_count(2)` at some point) This is not a "valid" fail since 1 is still in the range of the range checker. Because of this a manual check is needed for all the negative checks. To see all the imbalances occurred during a test remove the 'disable_debug_builder();' line from the `run_negative_test` function and run the test. I am 95% sure that I wen through all the negative tests and checked that the imbalances occurred are correct. The `test_adapter` tried to address this issue by getting rid of interaction imbalances on the memory bus. But even with the `test _adapter` a manual check was necessary. To solve this I suggest that we somehow keep all the interactions that occur during the test and automatically check that actually an invalid interaction has happened on a specified bus. Resolves INT-3975 --------- Co-authored-by: Ayush Shukla <[email protected]>
Fixed an error in divrem negative tests. The trace pranking was done incorrectly. 2 instructions were being called (so the trace had height 2) each time but only one of the rows was being modified. Changed it so only one instruction is called each time Also, made the setup_tracing the default
Implemented e1 and e3 for HeapBranch, Heap, and VecHeap adapters. Updated the Bigint circuit correspondingly. Had to make some changes in the interfaces of rv32im Steps. In particular - Changed Reads type `([u8; N], [u8; N])` into `Into<[[u8;N];2]>` and Writes type `[u8; N]` into `From<[[u8;N];1]>`. This change corresponds to what we used to do with the previous integration API in order to make the interfaces to match. - Got rid of TraceAdapterContext in a lot of places. This is because the same Step can be using different AdapterSteps that require different TraceContexts. Or even the AdapterStep might require a `TraceContext` that the Step doesn't have. The easy solution was to implement AdapterSteps in a similar way as in the previous integration API. That is, added the necessary fields to the AdapterStep structs. I am thinking maybe deleting the `TraceContext` from the interface makes sense. I am not sure if there is a better way to do this Important Note: the tests don't run right now because a lot of the read/write operations are done in address space 2 with block size 32 but currently only block size 4 is supported by the memory. Resolves INT-3980
Closes INT-4013
Resolves INT-3801. - Added memory access adapters. To improve: * Allocate the trace buffer once before filling it as opposed to pushing to `Vec` how it's done now, * Maybe not call `get_f` too often (although I don't know how to avoid it normally). - Added volatile and persistent boundary chips tracegen, - Added merkle chip tracegen as described [here](https://docs.google.com/document/d/12cH7ZYRFWHgflpPzOILb7bg5XExdyWOL4vwrQ9HFGkQ/edit?tab=t.0#heading=h.hrg0oexxgu9). To improve: * Parallelize at least something, * Maybe support passing this struct between segments. - `VmChipTestBuilder` now has `::default_persistent`, so all tests in `extensions/rv32im/circuit` pass both with volatile and persistent memory interface.
`cargo` complains that `uuid` has a conflict checksum.
I used to handle creating new blocks in a wrong way when `align > initial_block_size`, now I hopefully do it right. Also added persistent base alu tests, although nothing changed for the persistent case, and added a dummy access in all of them that used to fail.
This resolves INT-4012 by not using memory controller's memory in E1 execution.
implemented e1 and e3 for `VecHeapTwoReads` and `eq_mod` rv32 adapters. Implemented e1 and e3 for mod-builder. Updated the `algebra` and `ecc` extensions accordingly. Deleted all the pairing chips All the tests successfully run. Also, added back the address space 4 loadstore tests. Resolves INT-3914
- add codspeed walltime measurement job - tweak execution benchmarks to be heavier and more representative
…sts) (#1659) This resolves INT-3913. As a _side effect_, this removes `GuestMemory` trait -- it is a struct now with underlying `AddressMap<PAGE_SIZE>` (I didn't make `type GuestMemory = ...` because the vaguely called `read` and `write` methods would be too vaguely called for `AddressMap`). `VmStateMut` is generic over `MEM` though. I didn't fully implement `TraceStep` and `StepExecutorE1` for the phantom chip because the chip is relatively easy and I'm not sure it would be better expressible in terms of `NewVmChipWrapper`. `PhantomSubExecutor` also changed a little (now accepts `u32` instead of `F`, for example, and also `GuestMemory` instead of what it needed before).
Implemented e1, e3 for sha2 and keccak extensions. I feel like the trace generation of sha2 is more readable now. Also, implemented an arbitrary length read function (used in both e1 executions) I think eventually we should reimplement Plonky3's trace generation for Keccak so we don't have to allocate a separate trace matrix for the perm cols. TODOs: - port the tests of keccak to the new framework - spend some time (not much) on optimizing the keccak trace gen Resolves INT-3966 Resolves INT-3921
- Added a `execute_metered` function that keeps a track of `trace_heights` during execution and suspends execution if the trace height, number of cells or number of interactions goes above a fixed threshold Resolves INT-3752
We removed support for memory read/writes that aren't 4-byte aligned (except risc-v loadh, loadb), so now the keccak guest binding must handle the input misalignment cases
Fixes misaligned calls to sha256. Moved Aligned_buf to openvm platform so both `sha2` and `keccak` can access it.
Previously toolchain version was hardcoded to some version of rust 1.86 nightly. Sometimes users may want to use custom version, e.g. for some newest nightly Rust feature.
- make `PAGE_SIZE` a const generic in `PagedVec` - use fixed size `[T; PAGE_SIZE]` array instead of slice - update `PagedVec::get` to return a copy of data and not allocate pages - reduce bounds checking - maybe we can just remove all bounds checking in `PagedVec` if we can guarantee the pointers are bounds-checked somewhere else? 🤔 - store a `u8` offset in `AccessMetadata` instead of a `u32` start pointer - pack `timestamp` and `block_size` values into a single `u32` - make `prev_access_time` function a bit more readable - use a stack allocated array (smallvec) instead of heap allocated vector for intermediate variables
…1945) Cherry-picked from #1891 without the twisted Edwards macros. The `complex_init!` and `sw_init!` macros are updated so that instead of parsing a `Path`, it parses a `LitStr`. This makes it so that the `_init!` macros do not require any path imports in the file where it is called, which greatly improves the developer experience. The main files changed aside from test files and auto-generates `openvm_init.rs` files are: - extensions/algebra/complex-macros/src/lib.rs - extensions/ecc/sw-macros/src/lib.rs In both macros, the change essentially boils down to changing: (before) ```rust let items = input.parse_terminated(<Expr as Parse>::parse, Token![,])?; ``` to (after) ```rust let items = input.parse_terminated(<LitStr as Parse>::parse, Token![,])?; ``` and then associated type changes (mostly simplifications) in converting tokens to strings. There is one additional change needed only for the `sw_init!` macro case. Previously the linked function ```rust extern "C" fn #setup_extern_func() ``` defined by the `sw_init!` macro did use the path of the struct within the function implementation. After this PR, we avoid this import by moving that part of the logic into the `fn set_up_once()` function defined for the struct itself. This changed the function signature of the extern function to: ```rust extern "C" fn #setup_extern_func(uninit: *mut core::ffi::c_void, p1: *const u8, p2: *const u8) ``` [Here](https://www.diffchecker.com/0ncTMwe2/) is the diff of the code that was moved from the `#setup_extern_func()` into the `set_up_once()` function. --------- Co-authored-by: Avaneesh Kulkarni <[email protected]>
- rename `execute_e3_time_ms` to `execute_preflight_time_ms` and now it includes `memory_finalize_time_ms`. - the `single_trace_gen` info spans are skipped if record is empty just to reduce log verbosity
Now that our traits are unified around Executor This was a find/replace excluding the following: codec.rs,memcpy.s,memset.s,ECDSA.md,**/recursion/**,**/pairing/**,**/ff_derive/** closes INT-4350
This resolves INT-4549. The assertion was a sanity check and it essentially checked that the "type size" should be greater than one. I replaced it with an `debug_assert_eq!(align, min block size from config)`. While we are there, I also replaced some `cell_size` being `size_of::<T>()` or 1 with another value from config.
`total_proof_time_ms` can now be measured directly with a span instead of by summing up individual parts, which is more accurate. The increase in times from benchmarks includes some additional setup time.
This resolves INT-4550. Merkle tests checked that, when we touch some memory, - the finalized root that is obtained by "build a tree on initial memory, then update with the touched memory" is the same as if we just built the tree on the final memory, - the merkle trace does the interactions we want it to. I updated them so that they check the same but use the new architecture. The general memory tests checked that we can do all kinds of accesses (persistent/volatile, native/rv32 address space, etc etc) and nothing breaks. Apparently, I did the same in `online.rs`, so I just moved my tests to that file, where I think they indeed belong.
Addresses the following warning: ``` warning: package `alloy-eips v1.0.20` in Cargo.lock is yanked in registry `crates-io`, consider running without --locked warning: package `alloy-serde v1.0.20` in Cargo.lock is yanked in registry `crates-io`, consider running without --locked ```
Avaneesh-axiom
changed the base branch from
feat/new-execution
to
feat/ed25519
Avaneesh-axiom
force-pushed
the
feat/edwards-curve-new-execution
branch
from
5b3127b to
be1e5ec
Compare
Avaneesh-axiom
force-pushed
the
feat/edwards-curve-new-execution
branch
from
efd381d to
1849d45
Compare
Commit: 16a8d32 |
jonathanpwang
force-pushed
the
feat/ed25519
branch
from
b371303 to
426091b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Similar to #1255 but incorporated changes to tracegen design.