Crashing an Agave validator during snapshot bootstrap with one oversized account length
An Agave validator that bootstraps from a snapshot served by an untrusted peer — the default configuration of agave-validator run when --known-validators is not set — can be made to panic and abort during snapshot load by a single malformed field: one account-storage file (AppendVec) that declares an account whose data_len exceeds the 10 MiB per-account maximum triggers a fatal error in accounts-index generation, before the snapshot hash is ever checked, from a crafted storage file as small as ~136 bytes.
Why this matters
The exposed party is any operator bringing a validator up from a snapshot: a newly-joining node, or one restarting without a usable local snapshot. In the default configuration — agave-validator run with no --known-validators — the node will accept a snapshot from a peer it has no reason to trust, and a peer that serves a crafted snapshot can crash it deterministically. The crash is availability-only: no memory corruption, no funds or consensus impact, and a validator already in steady state is unaffected.
Two properties make it worse than a one-off abort. First, it is a boot-time failure, so the usual steady-state defences do not apply — the node dies before it is a participant. Second, the snapshot's cryptographic hash gate does not save you: the hash that snapshot verification checks is produced by the same index-generation pass that panics, so the integrity check that would reject the malformed snapshot never runs. Re-fetching from the same or a colluding peer reproduces the crash, which is a boot-loop until the operator changes snapshot source or sets a known-validator constraint.
How the attack works
Three trust assumptions compose (mechanism cited to Agave v4.1.0):
- Entry point: the snapshot download a bootstrapping validator fetches from a peer. The malformed field rides inside an AppendVec account-storage file in the snapshot archive.
- Preconditions: the victim is in snapshot-fetch state (joining, or restarting without a local snapshot) and
--known-validatorsis unset, so the attacker's node can be selected as the snapshot source. Remote, unauthenticated, pre-consensus. - Processing path:
- Reconstruction skips per-account validation. When a snapshot storage is rebuilt,
AppendVec::new_for_startuptrusts the storage length and, when it equals the file size (which the attacker controls), returns without runningsanitize_layout_and_length(). The per-accountsanitize()that does run checks only the executable byte and lamports — it never boundsdata_len(accounts-db/src/append_vec.rs). - Index generation treats an oversized
data_lenas fatal. Building the accounts index reads every account via a File-backed scan whose reader buffer is capped atSTORE_META_OVERHEAD + MAX_PERMITTED_DATA_LENGTH. The scan requestsSTORE_META_OVERHEAD + data_lenbytes with no cap ondata_len; whendata_lenexceeds 10 MiB the reader returns a quota error and the scan returnsErr.generate_index_for_slotcalls.expect("must scan accounts storage")on that result — so it panics (accounts-db/src/accounts_db.rs).
- Reconstruction skips per-account validation. When a snapshot storage is rebuilt,
- Boundary crossed: attacker-controlled snapshot bytes reach a fatal
.expect()on an internal path, before the snapshot hash gate — because the hash that gate checks is produced by the very pass that aborts. - Resulting behaviour: a deterministic crash/abort of the bootstrapping node. Index generation scans every storage, so one malicious AppendVec among many legitimate ones is enough; no valid account data is required.
The mechanism is a reachable panic on unvalidated input, not a memory-safety bug. MAX_PERMITTED_DATA_LENGTH is 10 MiB.
Affected systems
- Severity: High for availability of bootstrapping nodes in default config; not steady-state, not RCE (per NR-2026-002).
- Chain: Solana.
- Implementation: Agave (
agave-validator). Verified builds:- v4.1.0 – v4.1.2 — v4.1.2 is the current stable release; the two affected files (
accounts-db/src/append_vec.rs,accounts-db/src/accounts_db.rs) are byte-identical at v4.1.0 and v4.1.2, and v4.1.0's copy is byte-identical to the build the crash was measured against (per NR-2026-002). Every v4.1.x release to date is affected. - 4.2.0-alpha.0 (
3e9a16d6e6) — the build measured. - master (as of 2026-07-01) — present and, if anything, easier to reach: the per-account layout sanitisation was removed from the startup path (length is derived from file size, so the skip is unconditional) and the alternate Mmap storage backing was removed (File-backed only), so the panic is the sole outcome.
- v4.1.0 – v4.1.2 — v4.1.2 is the current stable release; the two affected files (
- Component: snapshot-load AppendVec reconstruction (
new_for_startup) → accounts-index generation (generate_index_for_slot). - Gated on: the victim being in snapshot-fetch state and
--known-validatorsbeing unset. A validator in steady state, or one restricted to known snapshot sources, is not exposed by this path. No other chains or clients are claimed.
Evidence
- Publicly documented: none as a CVE. Agave's
SECURITY.mdacknowledges maliciously-crafted snapshots as a known trust-on-first-use limitation with improvement "actively underway"; this brief adds a concrete, reproduced instance of that class, it does not claim a new public disclosure. - Independently reproduced by NullRabbit (lab): two unit-level tests run against the real Agave deserializer craft a 136-byte AppendVec with
data_len = MAX_PERMITTED_DATA_LENGTH + 1and drive it through the realnew_for_startup(the path that skips per-account sanitisation).test_malformed_snapshot_oversized_data_len_scan_errorsshows the realscan_accountscall index generation makes returnsErr(Io(QuotaExceeded));test_malformed_snapshot_oversized_data_len_panics_index_genshows the exactscan_accounts(..).expect("must scan accounts storage")pattern panics with the production message. - Observed under controlled conditions: the measured panic is
must scan accounts storage: Io(Custom { kind: QuotaExceeded, error: "requested more bytes than allowed capacity range" }), against 4.2.0-alpha.0 (byte-identical functions in v4.1.0). On recentmasterthenew_for_startupsignature changed, so the test bodies need a one-line signature update to compile there; the mechanism is unchanged. - Wire signature (traffic reproducer): a driver crafts the malicious snapshot archive (a
versionfile, a bank-manifest stub, andaccounts/<slot>.<id>= the 136-byte AppendVec whoseStoredMeta.data_len = MAX_PERMITTED_DATA_LENGTH + 1), serves it over loopback HTTP, and fetches it — reproducing the download an untrusted peer would serve, with the malformed account length present on the wire. It captures the malicious serve/fetch, not a live validator crash; the crash itself is proven by the unit-level reproducer. The captured bundle ships in theNullRabbit/nr-bundles-publicdataset (primitivesol_snapshot_oversized_datalen_indexgen_panic). - Not established: no live/mainnet validator was crashed. The crash is proven at the deserializer and index-generation level; the boot-loop consequence follows from the crash being deterministic, and is not separately load-tested.
Operator actions
Available today; no upstream fix required (per NR-2026-002):
- Set
--known-validators(primary). This constrains snapshot selection so an attacker's node cannot be chosen as your snapshot source, which removes the entry point. - When bootstrapping, prefer a trusted local or known snapshot source rather than an arbitrary peer.
- Detection idea (unconfirmed): alert on validator aborts during snapshot load whose panic references
must scan accounts storage, and on snapshot fetches from sources outside your known set. Treat this as a monitoring heuristic, not a validated signature.
Canonical references
- NRDAX technique: NRDAX-T0352 — Unvalidated field-length panic (family
state_import_abuse). - NullRabbit advisory: NR-2026-002 — Agave snapshot bootstrap crash via malformed AppendVec account length (published 2026-07-02).
- Evidence bundle (Hugging Face): NullRabbit/nr-bundles-public — primitive
sol_snapshot_oversized_datalen_indexgen_panic. - Reproducers:
NR-2026-002/reproducers/—append_vec_datalen_panic_tests.rs(unit-level, proves the crash) and itsREADME.md(both reproducers, scope). - Upstream primary sources:
anza-xyz/agaveaccounts-db/src/append_vec.rs(new_for_startup,sanitize_layout_and_length),accounts-db/src/accounts_db.rs(generate_index_for_slot); vendorSECURITY.md.
Related attacks
- Same technique class: NRDAX-T0352 (unvalidated field-length panic) also has instances in QUIC's
quiche—quiche_0rtt_packet_len_panicandquiche_payload_overflow_crash— where an unvalidated length field likewise drives a panic. Those are separate, reverse-engineered public CVEs in a different implementation, linked here only by the shared mechanism shape, not asserted to be the same bug. - Same family:
state_import_abuse— malformed bulk-state import reaching a reachable fault during import. NR-2026-002 is a single-finding advisory; no same-advisory sibling primitive is claimed.
Track attacks affecting Agave and Solana on NRDAX and in NullRabbit research.
Analysis plus reproducer. No weaponised proof-of-concept code.
Related Posts
A getProgramAccounts filter that matches nothing returns 37 bytes and occupies a pool thread for the whole scan
Agave evaluates getProgramAccounts filters inline during the scan, not via an index, so a never-matching filter holds a spawn_blocking thread for the full walk and returns an empty 37-byte array. The measured signal is pool saturation under concurrency: aggregate throughput pegs flat at ~60 req/s while per-worker rate drops 8x.
How unauthenticated simulateTransaction requests saturate an Agave RPC node's executor pool
Agave runs the simulateTransaction handler synchronously on its shared Tokio executor threads, so unauthenticated, gas-free simulate requests pin those workers and add queue latency across the validator's entire JSON-RPC tier — not just the simulate path.
NR-2026-001 - Three Agave RPC architectural findings
Three architectural findings in the Agave JSON-RPC layer at v3.1.9: response amplification on getMultipleAccounts, Tokio executor saturation via simulateTransaction, and spawn_blocking pool saturation via getProgramAccounts. Architectural patterns, not rate-limit DoS - operator rate limits don't close them.
