EVM integration progress #4

jsign · 2023-08-09T01:16:59Z

This PR made progress in implementing a workable EVM.

We reached the point that the first exec-spec-test passes.

There're still many caveats and TODOs, since having a full implementation means a careful impl of:

1559
Gas refunds
And many many other things.

Getting one working is good news.
The exec-spec-tests test is only running the first test in that file. We can bump this number to allow further tests to run, and keep improving the implementation so more tests pass. But that will work for the future.

Sample of runs:

$ zig build run
info: Welcome to phant! 🐘
info(vm): evmone info: name=evmone, version=0.11.0-dev, abi_version=10
debug(vm): run block number=100
debug(vm): call depth=0 sender=a94f5374fce5edbc8e2a8697c15331677e6ebf0b recipient=0000000000000000000000000000000000004142
debug(vm): non-contract call
debug(vm): execution result: status_code=0, gas_left=79000

(main.zig doesn't have any "interesting" stuff, but it works).

Running tests is more interesting:

$ zig build test --summary all         
run test: error: [execspectests] (debug): ##### Executing fixture 000-fork=Shanghai-CALL-sufficient_gas-opcode_call #####
[vm] (info): evmone info: name=evmone, version=0.11.0-dev, abi_version=10
[vm] (debug): run block number=1
[vm] (debug): call depth=0 sender=a94f5374fce5edbc8e2a8697c15331677e6ebf0b recipient=cccccccccccccccccccccccccccccccccccccccc
[vm] (debug): contract call, codelen=19
[vm] (debug): access_account(addr=0000000000000000000000000000000000000100)
[vm] (debug): call depth=1 sender=cccccccccccccccccccccccccccccccccccccccc recipient=0000000000000000000000000000000000000100
[vm] (debug): contract call, codelen=15
[vm] (debug): get_tx_context()
[vm] (debug): access_account(addr=2adc25665018aa1fe0e6bc666dac8fc2697ff9ba)
[vm] (debug): internal call exec result: status_code=3, gas_left=0
[vm] (debug): internal call exec result: status_code=0, gas_left=99954154
[vm] (debug): execution result: status_code=0, gas_left=99954154
checking account state: 0000000000000000000000000000000000000100
checking account state: 2adc25665018aa1fe0e6bc666dac8fc2697ff9ba
checking account state: a94f5374fce5edbc8e2a8697c15331677e6ebf0b
checking account state: cccccccccccccccccccccccccccccccccccccccc
Build Summary: 5/5 steps succeeded; 4/4 tests passed
test success
└─ run test 4 passed 4ms MaxRSS:3M
   └─ zig test Debug native cached 14ms MaxRSS:37M
      ├─ zig build-lib ethash Debug native cached 12ms MaxRSS:34M
      └─ zig build-lib evmone Debug native cached 36ms MaxRSS:36M

There are many things in this PR, but I'll create some comments if anyone is curious.

This is sufficient since evmone.h already includes evmc.h. Also, this is necessary since doing @cImport("evmc.h") would duplicate types creating weird compiler errors claiming that two types are different when that isn't the case.

Signed-off-by: Ignacio Hagopian <[email protected]>

…ll implementation Signed-off-by: Ignacio Hagopian <[email protected]>

Signed-off-by: Ignacio Hagopian <[email protected]>

jsign · 2023-08-11T21:52:29Z

build.zig

-    unit_tests.addIncludePath(LazyPath{ .path = "evmone" });
+    unit_tests.addIncludePath(LazyPath{ .path = "evmone/include/evmone" });
+    unit_tests.addIncludePath(LazyPath{ .path = "evmone/evmc/include" });
+    if (target.getCpuArch() == .x86_64) {
+        // On x86_64, some functions are missing from the static library,
+        // so we define dummy functions to make sure that it compiles.
+        unit_tests.addCSourceFile(.{
+            .file = .{ .path = "src/glue.c" },
+            .flags = &cflags,
+        });
+    }


@gballet, I added an analogous configuration for unit_tests executable.
At some point we can "unify" this since it will be always something that should be configured both for exe (main) and unit_test (tests).

Not a big deal for now.

jsign · 2023-08-11T21:53:57Z

src/exec-spec-tests/execspectests.zig

+        for (self.blocks) |encoded_block| {
+            var out = try allocator.alloc(u8, encoded_block.rlp.len / 2);
+            defer allocator.free(out);
+            const rlp_bytes = try std.fmt.hexToBytes(out, encoded_block.rlp[2..]);

-            var txns = try allocator.alloc(Transaction, block.transactions.len);
+            const block = try Block.init(rlp_bytes);
+
+            var txns = try allocator.alloc(Transaction, encoded_block.transactions.len);
            defer allocator.free(txns);
-            for (block.transactions, 0..) |tx_hex, i| {
+            for (encoded_block.transactions, 0..) |tx_hex, i| {
                txns[i] = try tx_hex.to_vm_transaction(allocator);
            }

-            evm.run_txns(txns);
+            try evm.run_block(block, txns);
        }


(Context: we're in the exec-spect-test runner)

Now we RLP-decode the block RLP, and run the block with our VM.
Note in L85 now the API is basically: "Please, run this block".

Still missing returning the result, but we'll get there.

jsign · 2023-08-11T21:56:11Z

src/exec-spec-tests/execspectests.zig

        // 3. Verify that the post state matches what the fixture `postState` claims is true.
        var it = self.postState.map.iterator();
        while (it.next()) |entry| {
-            var account_state = try entry.value_ptr.*.to_vm_accountstate(allocator, entry.key_ptr.*);
-            _ = account_state;
+            var exp_account_state: AccountState = try entry.value_ptr.*.to_vm_accountstate(allocator, entry.key_ptr.*);
+            std.debug.print("checking account state: {s}\n", .{std.fmt.fmtSliceHexLower(&exp_account_state.addr)});
+            const got_account_state = try db.get(exp_account_state.addr);
+            if (!std.mem.eql(u8, &got_account_state.addr, &exp_account_state.addr)) {
+                return error.post_state_addr_mismatch;
+            }
+            if (got_account_state.nonce != exp_account_state.nonce) {
+                log.err("expected nonce {d} but got {d}", .{ exp_account_state.nonce, got_account_state.nonce });
+                return error.post_state_nonce_mismatch;
+            }
+            if (got_account_state.balance != exp_account_state.balance) {
+                log.err("expected balance {d} but got {d}", .{ exp_account_state.balance, got_account_state.balance });
+                return error.post_state_balance_mismatch;
+            }
+            if (got_account_state.storage.count() != exp_account_state.storage.count()) {
+                return error.post_state_storage_size_mismatch;
+            }
        }


Now I added code to verify that executing the block had the expected side-effects as described by the test.

The test has a postState fields, so you can validate that executing the block changed the account state.
For example, see here.

jsign · 2023-08-11T21:57:03Z

src/exec-spec-tests/execspectests.zig

        }
+        // TODO(jsign): verify gas used.


Uhm, see now that I forgot to add this which should work in this PR already... I've verified this manually.
I'll eventually do it here.

jsign · 2023-08-11T21:57:39Z

src/exec-spec-tests/execspectests.zig

+        try std.testing.expect(try entry.value_ptr.*.run(test_allocator));
+        count += 1;
+
+        // TODO: Only run the first test for now. Then we can enable all and continue with the integration.


For now only one the first test in the json file; we can keep bumping this and making the EVM impl more complete by passing more and more tests.

jsign · 2023-08-11T22:03:30Z

src/vm/vm.zig

+        if (remaining_gas.* < charge) {
+            return error.OutOfGas;
+        }
+        remaining_gas.* -= charge;
    }

    // ### EVMC Host Interface ###

    fn get_tx_context(ctx: ?*evmc.struct_evmc_host_context) callconv(.C) evmc.struct_evmc_tx_context {


Already implemented this EVMC API... EVMOne calls this to get some info while executing opcodes.

For example, this first exec-spec-test calls the COINBASE opcode, and EVMOne call us on this method to get what is the address. I'd guess in many other cases it will do the same.

jsign · 2023-08-11T22:04:03Z

src/vm/vm.zig

+    fn set_storage(
+        ctx: ?*evmc.struct_evmc_host_context,
+        addr: [*c]const evmc.evmc_address,
+        key: [*c]const evmc.evmc_bytes32,
+        value: [*c]const evmc.evmc_bytes32,
+    ) callconv(.C) evmc.enum_evmc_storage_status {
+        const vm: *VM = @as(*VM, @alignCast(@ptrCast(ctx.?)));
+
+        const skey = std.mem.readIntSlice(u256, &key.*.bytes, std.builtin.Endian.Big);
+        const svalue = std.mem.readIntSlice(u256, &value.*.bytes, std.builtin.Endian.Big);
+
+        vm.statedb.set_storage(addr.*.bytes, skey, svalue) catch unreachable; // TODO(jsign): manage catch.
+
+        return evmc.EVMC_STORAGE_ADDED; // TODO(jsign): fix
    }


I had also to do some minimal implementation of this one, since this first test saves some result to storage.

jsign · 2023-08-11T22:04:41Z

src/vm/vm.zig

    }

-    fn call(ctx: ?*evmc.struct_evmc_host_context, msg: [*c]const evmc.struct_evmc_message) callconv(.C) evmc.struct_evmc_result {
+    fn call(


This first test makes to internal contract calls, which is interesting... so I had to do some reasonable implementation of this.

jsign · 2023-08-11T22:09:42Z

src/vm/vm.zig

-        // Create the new context to be used to do the call.
-        vm.exec_context = ExecutionContext{ .address = util.from_evmc_address(msg.*.recipient) };
+        // Check if the target address is a contract, and do the appropiate call.
+        const recipient_account = vm.statedb.get(util.from_evmc_address(msg.*.code_address)) catch unreachable; // TODO(jsign): fix this.


The reason for catch unreachable is just a quick way to ignore errors that can only happen due to the allocator not being able to allocate memory.

Why is that needed for get is just that if the account doesn't exists, then it's created. Creating it requires a storage : AutoHashMap(u256, u256) to be created which requires an allocator.

I'm not convinced about that... probably I'd like to make that storage optional and avoid this failure case. (i.e: get can't fail).

But.. that's a story for another day.

jsign · 2023-08-11T22:10:41Z

src/vm/vm.zig

+            // Restore previous context after call() returned.
+            vm.context.?.execution = prev_exec_context;


Showing a bit how "restoring the execution context" works for calls(...).
Kind of "stack based" context.

jsign · 2023-08-11T22:26:24Z

src/vm/vm.zig

The interesting changes

jsign force-pushed the jsign-vm-continuation branch 2 times, most recently from 92c765b to 9108033 Compare August 11, 2023 21:45

jsign added 14 commits August 11, 2023 18:46

all: only do @cImport for evmone.h

88c1413

This is sufficient since evmone.h already includes evmc.h. Also, this is necessary since doing @cImport("evmc.h") would duplicate types creating weird compiler errors claiming that two types are different when that isn't the case.

vm/util: add helper to convert to evmc_bytes32

30f61ba

Signed-off-by: Ignacio Hagopian <[email protected]>

vm: add tx_context

b199cd4

Signed-off-by: Ignacio Hagopian <[email protected]>

vm: simplifications

03c4bc5

Signed-off-by: Ignacio Hagopian <[email protected]>

vm: avoid long lines

baf47a0

Signed-off-by: Ignacio Hagopian <[email protected]>

types/block: improvements

562bb16

Signed-off-by: Ignacio Hagopian <[email protected]>

vm: add block as the unit of execution & improve tx context & evmc ca…

ddbb14f

…ll implementation Signed-off-by: Ignacio Hagopian <[email protected]>

vm: more progress

30c4c81

Signed-off-by: Ignacio Hagopian <[email protected]>

vm: implement txn sender nonce and fee charges

acef54e

Signed-off-by: Ignacio Hagopian <[email protected]>

exec-spec-test: first test works

d0051b9

Signed-off-by: Ignacio Hagopian <[email protected]>

vm: avoid logic repetition

6801848

Signed-off-by: Ignacio Hagopian <[email protected]>

vm: refactor VM contexts

73c4875

Signed-off-by: Ignacio Hagopian <[email protected]>

fix main

273fce8

Signed-off-by: Ignacio Hagopian <[email protected]>

fix build

d8eed51

Signed-off-by: Ignacio Hagopian <[email protected]>

jsign force-pushed the jsign-vm-continuation branch from 9108033 to d8eed51 Compare August 11, 2023 21:46

jsign changed the title ~~WIP: EVM integration progress~~ EVM integration progress Aug 11, 2023

jsign commented Aug 11, 2023

View reviewed changes

jsign merged commit ecea4b6 into main Aug 11, 2023
4 checks passed

jsign commented Aug 11, 2023

View reviewed changes

src/vm/vm.zig

Copy link

Owner Author

jsign Aug 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The interesting changes

jsign deleted the jsign-vm-continuation branch December 8, 2023 11:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EVM integration progress #4

EVM integration progress #4

jsign commented Aug 9, 2023 •

edited

Loading

jsign Aug 11, 2023

jsign Aug 11, 2023

jsign Aug 11, 2023

jsign Aug 11, 2023

jsign Aug 11, 2023

jsign Aug 11, 2023

jsign Aug 11, 2023

jsign Aug 11, 2023

jsign Aug 11, 2023

jsign Aug 11, 2023

jsign Aug 11, 2023

		// Restore previous context after call() returned.
		vm.context.?.execution = prev_exec_context;

EVM integration progress #4

EVM integration progress #4

Conversation

jsign commented Aug 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsign commented Aug 9, 2023 •

edited

Loading