Cut wasted CPU in the ingest accept path #1

New issue

Open

opened 2026-09-20 08:58:37 +00:00 by franz · 0 comments

franz commented

2026-09-20 08:58:37 +00:00

Owner

Profiling the ingest path under load turns up three hot spots that aren't doing useful work. Numbers are perf shares of total server CPU during a 25,200 events/s soak on SQLite (ingest_batch_size = 20000, eight dedicated vCPU).

Overall split: 84% of cycles on the tokio workers, 15% on the single SQLite writer thread. The writer is the serialization point and caps throughput at ~28,000 events/s regardless of how many cores the process gets, but the work below sits on the accept side and inflates what each event costs before it ever reaches the writer.

1. `AppState` clone/drop - 4.3%

AppState in src/server.rs is #[derive(Clone)] over ~25 Arc-ish fields, and axum clones it per request. That's ~25 atomic increments on the way in and ~25 decrements on the way out, for every single event.

Switching the router to State<Arc<AppState>> collapses that to one refcount pair. Self-contained change, and the largest single win of the three.

2. zstd payload compression - 9.6%

compress_payload in src/ingest/models.rs runs zstd at level 3 on the tokio workers. Level 1 is a fraction of the cost for a modest size increase on JSON, and the blobs are already small. Worth measuring level 1 against level 3 for both CPU and resulting database size before picking.

3. `serde_json` into a generic `Value` - 8.8%

The ingest path parses envelopes into serde_json::Value, which means a BTreeMap<String, Value> and an allocation per key. BTreeMap::insert alone shows up at 0.8%. A typed struct for the fields ingest actually reads would cut most of this, though it's the largest change of the three and needs care around the lenient-parsing behaviour.

Allocator traffic (malloc/free, 6.5%) is mostly downstream of 2 and 3, so it should shrink with them.

Reference

Raw benchmark data, per-run CSVs and the symbolized profile are in brain under projects/stackpit/benchmarks/2026-09-20-ccx43/ (see e-perf/perf-symbols.txt). Benchmark numbers are in the README.

Profiling the ingest path under load turns up three hot spots that aren't doing useful work. Numbers are `perf` shares of total server CPU during a 25,200 events/s soak on SQLite (`ingest_batch_size = 20000`, eight dedicated vCPU). Overall split: 84% of cycles on the tokio workers, 15% on the single SQLite writer thread. The writer is the serialization point and caps throughput at ~28,000 events/s regardless of how many cores the process gets, but the work below sits on the accept side and inflates what each event costs before it ever reaches the writer. ### 1. `AppState` clone/drop - 4.3% `AppState` in `src/server.rs` is `#[derive(Clone)]` over ~25 `Arc`-ish fields, and axum clones it per request. That's ~25 atomic increments on the way in and ~25 decrements on the way out, for every single event. Switching the router to `State<Arc<AppState>>` collapses that to one refcount pair. Self-contained change, and the largest single win of the three. ### 2. zstd payload compression - 9.6% `compress_payload` in `src/ingest/models.rs` runs zstd at level 3 on the tokio workers. Level 1 is a fraction of the cost for a modest size increase on JSON, and the blobs are already small. Worth measuring level 1 against level 3 for both CPU and resulting database size before picking. ### 3. `serde_json` into a generic `Value` - 8.8% The ingest path parses envelopes into `serde_json::Value`, which means a `BTreeMap<String, Value>` and an allocation per key. `BTreeMap::insert` alone shows up at 0.8%. A typed struct for the fields ingest actually reads would cut most of this, though it's the largest change of the three and needs care around the lenient-parsing behaviour. Allocator traffic (`malloc`/`free`, 6.5%) is mostly downstream of 2 and 3, so it should shrink with them. ### Reference Raw benchmark data, per-run CSVs and the symbolized profile are in brain under `projects/stackpit/benchmarks/2026-09-20-ccx43/` (see `e-perf/perf-symbols.txt`). Benchmark numbers are in the README.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

franz/stackpit#1

No description provided.

Rows
Columns

Cut wasted CPU in the ingest accept path #1

1. AppState clone/drop - 4.3%

2. zstd payload compression - 9.6%

3. serde_json into a generic Value - 8.8%

Reference

1. `AppState` clone/drop - 4.3%

3. `serde_json` into a generic `Value` - 8.8%