Crashing an Agave validator during snapshot bootstrap with one oversized account length
An Agave validator that bootstraps from a snapshot served by an untrusted peer — the default configuration of agave-validator run when --known-validators is not set — can be made to panic and abort during snapshot load by a single malformed field: one account-storage file (AppendVec) that declares an account whose data_len exceeds the 10 MiB per-account maximum triggers a fatal error in accounts-index generation, before the snapshot hash is ever checked, from a crafted storage file as small as ~136 bytes.
Why this matters
The exposed party is any operator bringing a validator up from a snapshot: a newly-joining node, or one restarting without a usable local snapshot. In the default configuration — agave-validator run with no --known-validators — the node will accept a snapshot from a peer it has no reason to trust, and a peer that serves a crafted snapshot can crash it deterministically. The crash is availability-only: no memory corruption, no funds or consensus impact, and a validator already in steady state is unaffected.
Two properties make it worse than a one-off abort. First, it is a boot-time failure, so the usual steady-state defences do not apply — the node dies before it is a participant. Second, the snapshot's cryptographic hash gate does not save you: the hash that snapshot verification checks is produced by the same index-generation pass that panics, so the integrity check that would reject the malformed snapshot never runs. Re-fetching from the same or a colluding peer reproduces the crash, which is a boot-loop until the operator changes snapshot source or sets a known-validator constraint.
How the attack works
Three trust assumptions compose (mechanism cited to Agave v4.1.0):
- Entry point: the snapshot download a bootstrapping validator fetches from a peer. The malformed field rides inside an AppendVec account-storage file in the snapshot archive.
- Preconditions: the victim is in snapshot-fetch state (joining, or restarting without a local snapshot) and
--known-validatorsis unset, so the attacker's node can be selected as the snapshot source. Remote, unauthenticated, pre-consensus. - Processing path:
- Reconstruction skips per-account validation. When a snapshot storage is rebuilt,
AppendVec::new_for_startuptrusts the storage length and, when it equals the file size (which the attacker controls), returns without runningsanitize_layout_and_length(). The per-accountsanitize()that does run checks only the executable byte and lamports — it never boundsdata_len(accounts-db/src/append_vec.rs). - Index generation treats an oversized
data_lenas fatal. Building the accounts index reads every account via a File-backed scan whose reader buffer is capped atSTORE_META_OVERHEAD + MAX_PERMITTED_DATA_LENGTH. The scan requestsSTORE_META_OVERHEAD + data_lenbytes with no cap ondata_len; whendata_lenexceeds 10 MiB the reader returns a quota error and the scan returnsErr.generate_index_for_slotcalls.expect("must scan accounts storage")on that result — so it panics (accounts-db/src/accounts_db.rs).
- Reconstruction skips per-account validation. When a snapshot storage is rebuilt,
- Boundary crossed: attacker-controlled snapshot bytes reach a fatal
.expect()on an internal path, before the snapshot hash gate — because the hash that gate checks is produced by the very pass that aborts. - Resulting behaviour: a deterministic crash/abort of the bootstrapping node. Index generation scans every storage, so one malicious AppendVec among many legitimate ones is enough; no valid account data is required.
The mechanism is a reachable panic on unvalidated input, not a memory-safety bug. MAX_PERMITTED_DATA_LENGTH is 10 MiB.
Affected systems
- Severity: High for availability of bootstrapping nodes in default config; not steady-state, not RCE (per NR-2026-002).
- Chain: Solana.
- Implementation: Agave (
agave-validator). Verified builds:- v4.1.0 – v4.1.2 — v4.1.2 is the current stable release; the two affected files (
accounts-db/src/append_vec.rs,accounts-db/src/accounts_db.rs) are byte-identical at v4.1.0 and v4.1.2, and v4.1.0's copy is byte-identical to the build the crash was measured against (per NR-2026-002). Every v4.1.x release to date is affected. - 4.2.0-alpha.0 (
3e9a16d6e6) — the build measured. - master (as of 2026-07-01) — present and, if anything, easier to reach: the per-account layout sanitisation was removed from the startup path (length is derived from file size, so the skip is unconditional) and the alternate Mmap storage backing was removed (File-backed only), so the panic is the sole outcome.
- v4.1.0 – v4.1.2 — v4.1.2 is the current stable release; the two affected files (
- Component: snapshot-load AppendVec reconstruction (
new_for_startup) → accounts-index generation (generate_index_for_slot). - Gated on: the victim being in snapshot-fetch state and
--known-validatorsbeing unset. A validator in steady state, or one restricted to known snapshot sources, is not exposed by this path. No other chains or clients are claimed.
Evidence
- Publicly documented: none as a CVE. Agave's
SECURITY.mdacknowledges maliciously-crafted snapshots as a known trust-on-first-use limitation with improvement "actively underway"; this brief adds a concrete, reproduced instance of that class, it does not claim a new public disclosure. - Independently reproduced by NullRabbit (lab): two unit-level tests run against the real Agave deserializer craft a 136-byte AppendVec with
data_len = MAX_PERMITTED_DATA_LENGTH + 1and drive it through the realnew_for_startup(the path that skips per-account sanitisation).test_malformed_snapshot_oversized_data_len_scan_errorsshows the realscan_accountscall index generation makes returnsErr(Io(QuotaExceeded));test_malformed_snapshot_oversized_data_len_panics_index_genshows the exactscan_accounts(..).expect("must scan accounts storage")pattern panics with the production message. - Observed under controlled conditions: the measured panic is
must scan accounts storage: Io(Custom { kind: QuotaExceeded, error: "requested more bytes than allowed capacity range" }), against 4.2.0-alpha.0 (byte-identical functions in v4.1.0). On recentmasterthenew_for_startupsignature changed, so the test bodies need a one-line signature update to compile there; the mechanism is unchanged. - Wire signature (traffic reproducer): a driver crafts the malicious snapshot archive (a
versionfile, a bank-manifest stub, andaccounts/<slot>.<id>= the 136-byte AppendVec whoseStoredMeta.data_len = MAX_PERMITTED_DATA_LENGTH + 1), serves it over loopback HTTP, and fetches it — reproducing the download an untrusted peer would serve, with the malformed account length present on the wire. It captures the malicious serve/fetch, not a live validator crash; the crash itself is proven by the unit-level reproducer. The captured bundle ships in theNullRabbit/nr-bundles-publicdataset (primitivesol_snapshot_oversized_datalen_indexgen_panic). - Not established: no live/mainnet validator was crashed. The crash is proven at the deserializer and index-generation level; the boot-loop consequence follows from the crash being deterministic, and is not separately load-tested.
Operator actions
Available today; no upstream fix required (per NR-2026-002):
- Set
--known-validators(primary). This constrains snapshot selection so an attacker's node cannot be chosen as your snapshot source, which removes the entry point. - When bootstrapping, prefer a trusted local or known snapshot source rather than an arbitrary peer.
- Detection idea (unconfirmed): alert on validator aborts during snapshot load whose panic references
must scan accounts storage, and on snapshot fetches from sources outside your known set. Treat this as a monitoring heuristic, not a validated signature.
Canonical references
- NRDAX technique: NRDAX-T0352 — Unvalidated field-length panic (family
state_import_abuse). - NullRabbit advisory: NR-2026-002 — Agave snapshot bootstrap crash via malformed AppendVec account length (published 2026-07-02).
- Evidence bundle (Hugging Face): NullRabbit/nr-bundles-public — primitive
sol_snapshot_oversized_datalen_indexgen_panic. - Reproducers:
NR-2026-002/reproducers/—append_vec_datalen_panic_tests.rs(unit-level, proves the crash) and itsREADME.md(both reproducers, scope). - Upstream primary sources:
anza-xyz/agaveaccounts-db/src/append_vec.rs(new_for_startup,sanitize_layout_and_length),accounts-db/src/accounts_db.rs(generate_index_for_slot); vendorSECURITY.md.
Related attacks
- Same technique class: NRDAX-T0352 (unvalidated field-length panic) also has instances in QUIC's
quiche—quiche_0rtt_packet_len_panicandquiche_payload_overflow_crash— where an unvalidated length field likewise drives a panic. Those are separate, reverse-engineered public CVEs in a different implementation, linked here only by the shared mechanism shape, not asserted to be the same bug. - Same family:
state_import_abuse— malformed bulk-state import reaching a reachable fault during import. NR-2026-002 is a single-finding advisory; no same-advisory sibling primitive is claimed.
Track attacks affecting Agave and Solana on NRDAX and in NullRabbit research.
Analysis plus reproducer. No weaponised proof-of-concept code.
Related Posts
How unauthenticated simulateTransaction requests saturate an Agave RPC node's executor pool
Agave runs the simulateTransaction handler synchronously on its shared Tokio executor threads, so unauthenticated, gas-free simulate requests pin those workers and add queue latency across the validator's entire JSON-RPC tier — not just the simulate path.
NR-2026-001 - Three Agave RPC architectural findings
Three architectural findings in the Agave JSON-RPC layer at v3.1.9: response amplification on getMultipleAccounts, Tokio executor saturation via simulateTransaction, and spawn_blocking pool saturation via getProgramAccounts. Architectural patterns, not rate-limit DoS - operator rate limits don't close them.
Expensive work before authentication: the RPC pattern we keep finding
The DoS class that scales against validators isn't volumetric. It's small requests that cost the node real work before it authenticates the caller. The pattern we keep finding across clients, and the fix.
