smitebot: add bench command to measure Nyx execution speed by erickcestari · Pull Request #160 · lnfuzz/smite

erickcestari · 2026-07-21T13:20:44Z

The goal is to tell whether a change concretely improved the target's throughput or not: run bench before and after and compare execs/sec.

It runs a single input through the target's Nyx VM many times, timing the snapshot-restore-plus-target-run loop. Each exec is one snapshot restore plus one target run, so the numbers isolate VM/snapshot and target speed from AFL++'s mutation and scheduling overhead.

Like start, it builds the image and sets up the sharedir by default (pass --no-build to reuse an existing one), and loads libnyx.so at runtime the same way afl-fuzz does. --repeat averages over several fresh VM boots so a real change stands out from boot/snapshot variance, and --timeout bounds a single execution (default 2s).

Chand-ra

Looks like a great addition overall. Following are some improvement suggestions.

erickcestari · 2026-07-23T15:14:52Z

Thanks for the review @Chand-ra ! I've added the coverage metric, which is a nice feature to have. Specially when we start using the bedrock as one of the hypervisor.

Ashish-Kumar-Dash

I had a few thoughts/nits, otherwise its a solid PR! The only real concern I have is the order in which we merge status, bench and print-ir since one of us will have to rebase, but I think it'd be much better if bench,print-ir could get merged and then we land status. I don't think it will be an issue on your part, since I'd have to do the rebase then :)

morehouse · 2026-07-21T17:35:10Z

+smitebot bench campaign.toml --input testcase.bin --iterations 5000
+```
+
+- `--input`: Input file to execute repeatedly. Defaults to a single `0x00` byte.


I think --input should be required -- it should be rare that anyone wants to benchmark an empty program.

It's useful for quickly checking performance improvements. I've used it to quickly test JVM optimization because the executor checks if the target is live, which does a ping-pong.

I think it is worth mentioning in the documentation then: the default input can be used to quickly check performance improvements due to factors like target optimization.

morehouse · 2026-07-23T17:05:39Z

+    /// Start of the `result` section within the buffer.
+    const RESULT: usize = 896;
+    /// `result.dirty_pages` (`u32`): pages restored for the last exec.
+    pub const DIRTY_PAGES: usize = RESULT + 16;
+    /// `result.runtime_usec` (`u32`): microsecond part of guest payload runtime.
+    pub const RUNTIME_USEC: usize = RESULT + 28;
+    /// `result.runtime_sec` (`u32`): second part of guest payload runtime.
+    pub const RUNTIME_SEC: usize = RESULT + 32;


These offsets are all correct AFAICT. But it seems fragile -- it would be nice if Nyx provided a clearer interface.

erickcestari · 2026-07-23T21:09:49Z

Thanks @morehouse and @Ashish-Kumar-Dash for the review.

Chand-ra

GitHub started acting up so I couldn't use the "suggest change" functionality🤦. Looks mostly good to me, following are a couple of suggestions.

Chand-ra · 2026-07-24T05:57:40Z

+smitebot bench campaign.toml --input testcase.bin --iterations 5000
+```
+
+- `--input`: Input file to execute repeatedly. Defaults to a single `0x00` byte.


I think it is worth mentioning in the documentation then: the default input can be used to quickly check performance improvements due to factors like target optimization.

Chand-ra

Took a quick look and verified that all the pointers from the previous feedback were addressed, looks good to me now!

Ashish-Kumar-Dash · 2026-07-30T13:51:37Z

@erickcestari I pushed some recent commits, you might have to rebase

erickcestari · 2026-07-30T14:29:01Z

@erickcestari I pushed some recent commits, you might have to rebase

Thanks! I've rebased it now.

morehouse · 2026-08-04T17:46:33Z

+    /// Maximum input buffer size in bytes for the Nyx VM. Defaults to the input
+    /// size rounded up to a 4 KiB page; pass 1048576 to match AFL++'s 1 MiB buffer.
+    #[arg(long)]
+    max_input_size: Option<u32>,


Is there ever a reason to use this flag now that we automatically calculate the buffer size?

Actually I'll remove the adaptive buffer size, since when running with AFL++ the value used is always the 1MiB MAX_FILE. So I believe it makes more sense to bench in the same way AFL++ runs. Also I'll keep an option to change the max buffer size to test if changing it makes any difference on performance.

erickcestari · 2026-08-06T15:41:38Z

Thanks @morehouse. It was an insightful review. It's ready for another one.

morehouse · 2026-08-07T17:14:50Z

+    /// Defaults to AFL++'s 1 MiB, the buffer a real campaign runs with. Lower it
+    /// to measure what a smaller buffer buys: the guest allocates the whole
+    /// buffer on every execution and the snapshot resets every page of it
+    /// dirtied, so buffer size shows up directly in the Nyx overhead and dirty
+    /// pages per exec.


I don't think this logic checks out. Assume the input size is X pages and the buffer size is Y > X pages. Then regardless of what Y is, only X pages ever get dirtied and need to be reset.

Have you actually observed differences in performance with varying buffer sizes?

I'll provide a benchmark later. Currently I'm in a laptop without the charger which would result in unreliable benchmark.

Actually, that's not true at all. When I ran it, it ended up using an E-core instead of a P-core because I hadn't pinned the CPU. It produced a different result, which led me to think the difference was caused by max-input-size.

bench-exec needs the same Nyx sharedir setup as start, so lift the helper out of start.rs into utils instead of duplicating it.

The goal is to tell whether a change concretely improved the target's throughput or not: run bench before and after and compare execs/sec. It runs a single input through the target's Nyx VM many times, timing the snapshot-restore-plus-target-run loop. Each exec is one snapshot restore plus one target run, so the numbers isolate VM/snapshot and target speed from AFL++'s mutation and scheduling overhead. Like start, it builds the image and sets up the sharedir by default (pass --no-build to reuse an existing one), and loads libnyx.so at runtime the same way afl-fuzz does. --repeat averages over several fresh VM boots so a real change stands out from boot/snapshot variance, and --timeout bounds a single execution (default 2s). Adds FFI bindings to libnyx (loaded via dlopen) and a small stats module for latency percentiles and mean/stddev aggregation.

erickcestari mentioned this pull request Jul 21, 2026

workloads+smite-scenarios: patch CLN to sync with new blocks faster #156

Merged

erickcestari force-pushed the bench-exec-s branch from 5170540 to 528ffd7 Compare July 21, 2026 15:14

Chand-ra suggested changes Jul 22, 2026

View reviewed changes

Comment thread smitebot/src/commands/start.rs Outdated

Comment thread smitebot/src/commands/bench.rs Outdated

Comment thread smitebot/README.md Outdated

Comment thread smitebot/src/commands/bench_exec.rs

erickcestari mentioned this pull request Jul 22, 2026

smite-scenarios: warm up and freeze Eclair's JIT before the snapshot #161

Open

erickcestari force-pushed the bench-exec-s branch 2 times, most recently from 15cf725 to c2c557f Compare July 23, 2026 15:13

Ashish-Kumar-Dash suggested changes Jul 23, 2026

View reviewed changes

Comment thread smitebot/src/commands/bench_exec.rs

Comment thread smitebot/README.md Outdated

Comment thread smitebot/src/libnyx.rs Outdated

Comment thread smitebot/src/commands/bench_exec.rs Outdated

Comment thread smitebot/Cargo.toml Outdated

morehouse reviewed Jul 23, 2026

View reviewed changes

erickcestari force-pushed the bench-exec-s branch 3 times, most recently from d6d5659 to 931a8b2 Compare July 23, 2026 20:53

Chand-ra suggested changes Jul 24, 2026

View reviewed changes

erickcestari force-pushed the bench-exec-s branch 3 times, most recently from bfb96ea to 470fce8 Compare July 24, 2026 17:27

Chand-ra approved these changes Jul 28, 2026

View reviewed changes

erickcestari force-pushed the bench-exec-s branch 2 times, most recently from c80f707 to 76314ac Compare July 30, 2026 14:28

erickcestari requested a review from morehouse July 31, 2026 14:14

morehouse requested changes Aug 4, 2026

View reviewed changes

erickcestari force-pushed the bench-exec-s branch 2 times, most recently from 01a9562 to 28fe6ec Compare August 6, 2026 15:37

erickcestari requested a review from morehouse August 6, 2026 15:40

morehouse reviewed Aug 7, 2026

View reviewed changes

erickcestari force-pushed the bench-exec-s branch from 28fe6ec to 508fe79 Compare August 10, 2026 23:32

smitebot: move setup_nyx into utils

7ce8e40

bench-exec needs the same Nyx sharedir setup as start, so lift the helper out of start.rs into utils instead of duplicating it.

erickcestari force-pushed the bench-exec-s branch from 508fe79 to c306fc7 Compare August 10, 2026 23:40

erickcestari requested a review from morehouse August 11, 2026 00:02

erickcestari added 2 commits August 11, 2026 14:03

smitebot: document bench-exec in README

c13e9f0

erickcestari force-pushed the bench-exec-s branch from c306fc7 to c13e9f0 Compare August 11, 2026 17:03

Uh oh!

Conversation

erickcestari commented Jul 21, 2026

Uh oh!

Chand-ra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erickcestari commented Jul 23, 2026

Uh oh!

Ashish-Kumar-Dash left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

erickcestari commented Jul 23, 2026

Uh oh!

Chand-ra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Chand-ra left a comment

Choose a reason for hiding this comment

Uh oh!

Ashish-Kumar-Dash commented Jul 30, 2026

Uh oh!

erickcestari commented Jul 30, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

erickcestari commented Aug 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!