13 분 소요

Summary

BBXFuzz addresses a recurring limitation of mutation-based fuzzing: coverage can stall in front of branch conditions that require input structure, parser state, exact strings, numeric boundaries, or code-flow awareness. AFL++ is effective at executing inputs and measuring coverage, but blind mutation often cannot infer why a near-miss input failed the next branch.

BusyBox is not the motivation by itself. It is the evaluation target that makes the motivation concrete. Its applets include small languages and structured formats: awk scripts, shell scripts, ed commands, vi commands, dpkg archives, HTTP requests, compressed inputs, and more. That variety creates many cases where raw byte mutation can spend a long time near a parser path without producing the input shape that satisfies the next condition.

The design insight was to split the work by what each component is good at:

  • AFL++ stays in charge of execution, coverage, crashes, and truth.
  • A local grammar mutator keeps feeding syntactically useful inputs without network calls.
  • A local solver gets the first chance at simple stalled branches.
  • Codex is used only when a coverage stall needs source-aware reasoning, with the blocked branch condition, code flow, solver diagnostics, and closest corpus input.
  • Any Codex output goes back into the fuzzer as seed files and grammar changes, not as a trusted conclusion.

The supported claim is therefore narrow: LLM-assisted stall breaking produced test artifacts that AFL++ could validate. This post does not claim that “the LLM found bugs.”

BBXFuzz design: AFL++ runs local grammar mutation first, then coverage stalls are escalated to Codex through a local solver and source-backed review tasks.
BBXFuzz keeps the fast path local and escalates only when the fuzzer has concrete coverage evidence that it is stuck.

Problem

The core problem was not just invalid syntax. The core problem was that mutation-only fuzzing often cannot explain why a near-miss input failed a branch condition.

BusyBox is a multi-call binary, and each applet has a different input contract. Some applets read scripts from files, some read commands from stdin, some parse binary archive formats, and some are better tested through small harnesses. Treating them all as raw bytes made the fuzzer waste work on invalid inputs, but even syntactically valid inputs could still stop before a hard branch.

This motivates a constrained use of LLMs: not as replacement fuzzers, and not as random test-case generators, but as source-aware assistants invoked when AFL++ has coverage evidence of a stall. The workflow therefore constrains each model task to a small, evidence-backed request:

  1. Here is the blocked branch.
  2. Here is the enclosing C source.
  3. Here is the closest input that already reaches the area.
  4. Explain why the current input fails the condition.
  5. Generate a few minimal bypass seeds and update the grammar if the source reveals a missing token, constant, or production.

This framing is important because the fuzzer, not the model, decides whether the answer was useful.

Motivating Example: A Provenance Chain, Not Causal Proof

The historical ash stall is useful for explaining the workflow, but the available artifacts do not link that stall to a later ASAN-confirmed bug. A cleaner real example is a dpkg stall artifact from issue #3743 and PR #3748. It is presented as provenance evidence, not as proof that LLM assistance caused a bug discovery.

In that run, BBXFuzz detected a coverage stall in the dpkg applet after 171,777 executions. The campaign had 690 of 5,411 edges, 492 corpus entries, and 709 stuck branches. The local solver had produced 0 seeds and 0 new edges across 2 runs, so the workflow escalated the top blocked branches into a Codex stall task.

The merged PR added five .deb seeds and updated the dpkg grammar:

commit 4e03c954 seeds+grammar: stall #3743 bypass for dpkg (#3748)
  grammars/dpkg_grammar.json
  seeds/dpkg/stall_3743/seed_001.dpkg
  seeds/dpkg/stall_3743/seed_002.dpkg
  seeds/dpkg/stall_3743/seed_003.dpkg
  seeds/dpkg/stall_3743/seed_004.dpkg
  seeds/dpkg/stall_3743/seed_005.dpkg

One of those seeds was not a random byte blob. It had the outer shape of a Debian archive: an ar container, a debian-binary member, and compressed metadata members. That matters because the artifact was structurally plausible enough for AFL++ to accept and mutate in the normal queue.

The stall prompt included the branch source context and the call chain that reached it. One representative source excerpt from the prompt was:

unsigned FAST_FUNC string_array_len(char **argv)
{
	char **start = argv;

	while (*argv)
		argv++;

	return argv - start;
}

This code is not the later crash sink. It is included here because it captures an important engineering issue in coverage-stall handling: the immediate blocked branch reported by coverage instrumentation may be a generic library branch, while the closest corpus input still exposes the target input structure. In this case, the closest input was already a Debian archive-like file, and the generated artifact preserved that structure:

ar archive
  debian-binary: "2.0\n"
  control archive member
  data archive member

The local artifact chain is direct: a checked-in seed from PR #3748 is byte-identical to an AFL++ queue entry tagged as coming from a Codex PR, and a later crashing input records that queue entry as its mutation parent. This post deliberately omits the seed hash, queue filename, crash filename, sink function, source line, stack trace, and vulnerable code because the disclosure is still pending.

This example supports the claim that LLM-assisted stall breaking produced test artifacts that AFL++ could validate. It does not prove that vanilla AFL++ or grammar-only BBXFuzz would have failed to find the same crash under the same budget. Tier 1 also injected grammar seeds during the surrounding campaign, and the historical logs do not isolate every later edge or crash to a single intervention. Without the ablation study below, the dpkg chain should be read as an artifact-provenance example, not a causal performance result.

The model still did not “find the bug.” It produced a structured test artifact. AFL++ accepted and mutated that artifact. The resulting crash was then reproduced, minimized, and triaged through the normal ASAN and root-cause workflow.

Approach

BBXFuzz has three practical phases.

Phase 0: Initialize the Applet

Before fuzzing an applet, the tool builds applet-specific artifacts: a grammar JSON file, initial seeds, and variant-hunting prompts. The applet registry records the BusyBox source files, input type, input description, and AFL++ feeding mode for each target. This keeps awk, hush, vi, dpkg, udhcpc, and the other applets from being treated as one generic byte stream.

Phase 1: Mutate Locally

The local Tier 1 mutator reads the grammar and produces new seeds without calling an LLM. It combines grammar derivation, corpus-aware mutation, and strategy templates. The current default configuration uses a 60-second mutation cycle, 50 seeds per batch, and a 1 MB generated-seed cap.

Tier 1 is the high-throughput layer. It is low-cost, repeatable, and does not need a model in the loop.

Phase 2: Solve or Escalate Stalls With Source Context

When edge growth stays below the configured threshold across the stall window, BBXFuzz takes a coverage snapshot, finds stuck branches, and extracts nearby C context. A local regex-based constraint solver tries first. If it can patch a seed for a simple branch condition, the fuzzer can keep moving without Codex.

When the solver produces no seeds, or after repeated solver-only rounds, the stalled branch is escalated into a reviewable task. The work item is intentionally narrow: use the supplied evidence to generate bypass seeds and update the grammar.

The important detail is that the prompt is not just “make more inputs.” It carries the branch condition, function body, call chain, covered siblings, closest corpus input, and solver diagnostics when available. That is what lets the model reason about the actual path constraint and code flow instead of guessing.

The Design Insight

The central design boundary is:

Use models for source-aware decisions; use fuzzers for execution truth.

The design does not replace AFL++. It uses the model to read the source context that AFL++ has already identified as interesting, then produce small artifacts that AFL++ can immediately accept or reject.

This also changed the grammar from a static file into a campaign memory. When a stall reveals a missing keyword, magic value, command form, or boundary token, the grammar can be updated so future local mutations start closer to that path.

Evidence Boundary

The project currently registers 24 BusyBox applet targets, with four blacklisted because they reached coverage plateaus. The live applet set spans editors, shells, coreutils-like tools, archive parsers, and network-facing inputs.

For most vulnerability reports from this campaign, there is not enough public-safe attribution to determine whether plain AFL++ mutations, grammar mutations, solver seeds, or Codex-generated artifacts were decisive. The dpkg chain above is an exception for artifact provenance, not for causal improvement. This post does not publish minimized PoCs, exact sink locations, crash hashes, queue filenames, or exploit-oriented details.

The workflow can be summarized as the following validation loop:

  1. AFL++ discovers coverage reality.
  2. BBXFuzz extracts the stuck source context.
  3. The local solver tries cheap seed patches first.
  4. Codex proposes small seed and grammar changes when local solving is not enough.
  5. AFL++ tests those changes.
  6. Any crashes are reproduced and triaged separately from the model-generated suggestion.

This loop converts coverage stalls into source-backed artifacts without treating unverified model output as evidence.

What the Ablation Should Measure

A search of the local project state did not identify a completed, apples-to-apples vanilla AFL++ baseline campaign for the same BusyBox build, target set, seed corpus, CPU budget, and 24-hour window. The post therefore does not claim a quantified BBXFuzz-over-vanilla improvement.

A valid ablation has to run the same targets under the same environment:

Variant Enabled components Question it answers
Vanilla AFL++ Initial corpus only What does AFL++ reach without BBXFuzz guidance?
Grammar only Tier 1 grammar mutation How much comes from syntactically valid local mutation?
Grammar plus solver Tier 1 and local branch solver How much comes from cheap deterministic stall breaking?
Codex seeds only Tier 2 seeds, no grammar persistence Does one-time LLM seed generation help after stalls?
Full BBXFuzz Grammar, solver, Codex seeds, grammar updates Does the persistent seed plus grammar loop improve over all lower tiers?

The metrics should be collected per applet and per campaign window:

  • Edge coverage over time.
  • Time to first new edge after a detected stall.
  • Number of generated seeds that survive AFL++ execution and corpus minimization.
  • Unique crashes after deduplication and ASAN reproduction.
  • Number of Codex calls, merged PRs, rejected PRs, and stale or duplicate seeds.
  • Prompt bytes, prompt tokens, completion tokens, model, latency, and cost per call.
  • 24-hour totals for all of the above.

The main hypotheses are testable. If BBXFuzz improves performance, the likely causes are not an opaque LLM effect. They are fewer invalid inputs from applet grammars, stall gating that avoids constant model use, source context that makes generated seeds more relevant, local solver attempts that avoid unnecessary Codex calls, and grammar updates that amortize one useful stall analysis across later local mutations.

Prompt-Token and Cost Boundary

Current GitHub-mode BBXFuzz logs preserve prompt files, issue counts, seed PRs, grammar versions, and turnaround observations. They do not preserve exact API token usage because the historical workflow did not call a metered API directly from the fuzzer. Archived completion text is also incomplete, so completion tokens and end-to-end cost are missing.

The available archive supports prompt-size measurement. An offline tokenizer pass over 2,111 archived stall_*.md prompt files from the work-20260406_224550 session used the cl100k_base tokenizer, which is an OpenAI GPT-3.5/GPT-4-era tokenizer. It may not match the exact Codex model tokenizer used by the historical workflow, and it will not match Anthropic tokenization or newer OpenAI tokenizers exactly. This is a prompt-size proxy, not billing data.

Prompt-size proxy Bytes Words cl100k_base tokens
Minimum 2,778 427 661
25th percentile 18,190 1,989 5,615
Median 20,449 2,448 6,906
75th percentile 26,104 3,978 10,059
90th percentile 29,408 4,799 12,179
Maximum 29,430 5,243 14,006
Mean 21,998 3,015 7,998

The dpkg stall prompt used in the example above was 28,751 bytes, 4,785 words, and 11,718 cl100k_base proxy tokens, near the 90th percentile of that archive. A publication-quality evaluation should log prompt tokens, completion tokens, model identity, wall time, retries, and whether the resulting seeds were accepted by AFL++.

Observed Properties

The implementation forces model output into artifacts that the fuzzing campaign can evaluate. A seed either survives execution and affects coverage or it does not. A grammar update either generates useful input shapes or it does not. A crash report remains separate from model output and requires independent reproduction and triage.

The reviewable task workflow also makes the campaign auditable. Each escalated stall is represented by an evidence bundle, a seed update, and a grammar diff, rather than by an unstructured model conversation.

Limitations

First, the current evidence does not include a completed vanilla-AFL++ baseline or the ablation matrix above. Without that experiment, this post cannot claim a quantified improvement over AFL++ or isolate the contribution of the LLM tier.

Second, the historical GitHub-mode workflow preserved prompts and PR artifacts, but not complete API accounting. A publication-quality implementation should record prompt tokens, completion tokens, latency, model identity, retry count, and the post-fuzzing outcome for each Tier 2 request.

Third, attribution remains incomplete. A merged seed PR should be linked to later coverage changes, queue survival, minimization results, and crash provenance so that productive stall-breaking artifacts can be separated from harmless but unproductive ones.

Finally, the public disclosure path must remain separate from the fuzzing workflow. Pending vulnerability details should not be published merely because the fuzzer generated a useful internal report.

Prior work shows that LLM assistance can improve fuzzing when the model is constrained to a well-defined role and the execution system remains the oracle. The table below compares BBXFuzz against the closest systems along three axes: when the model is invoked, what artifact the model produces, and how the artifact is validated.

Work Venue/status Trigger for LLM use LLM output unit Validation role Distinction from BBXFuzz
Asmita et al. USENIX Security 2024 Before fuzzing and crash reuse Target-specific initial seeds AFL++ evaluates generated seed corpora BBXFuzz invokes the model after an observed coverage stall and persists both seed files and grammar deltas.
FuzzingBrain V2 arXiv preprint, submitted May 20, 2026 Agent-selected suspicious points PoCs, reproduction artifacts, and reports Fuzzing grounds agent output BBXFuzz is not an end-to-end vulnerability agent; its Tier 2 output is limited to applet fuzzing artifacts.
HyLLfuzz arXiv preprint, 2024 with 2026 revision Greybox roadblocks Modified inputs from LLM-assisted concolic reasoning Coverage-guided execution validates reachability BBXFuzz has similar stall timing, but uses seed and grammar PRs rather than a concolic-execution framing.
HLPFUZZ USENIX Security 2025 Complex constraints in language processors Constraint-solving inputs Hybrid fuzzing validates deeper states BBXFuzz targets BusyBox applets and command/file/network formats rather than language processors as a class.
Branch Flipper AIxCC public technical report Fuzz blockers Blocker-unlocking seeds, including binary formats via Kaitai Coverage feedback grounds retries BBXFuzz has weaker binary-format modeling, but integrates applet grammars and reviewable seed PRs into a live AFL++ campaign.
ChatAFL NDSS 2024 Protocol structure and state guidance Protocol grammars and next-message predictions Stateful protocol fuzzing validates coverage BBXFuzz is not protocol-state specific; it escalates applet stalls after AFL++ coverage evidence.
Fuzz4All ICSE 2024 General LLM-powered fuzzing loop Generated or mutated language inputs Target feedback guides prompt evolution BBXFuzz is narrower and more auditable: applet-specific stalls become concrete seed and grammar diffs tied to AFL++ queue behavior.
TitanFuzz ISSTA 2023 Deep-learning library API fuzzing Generated or mutated Python programs DL-library execution and coverage validate programs BBXFuzz targets BusyBox parsers and command/file/network inputs rather than deep-learning API programs.

The resulting contribution is narrower than general LLM-driven fuzzing. BBXFuzz studies source-aware stall breaking for a live AFL++ BusyBox campaign, where model output is restricted to test artifacts and all behavioral claims are mediated by fuzzer execution, ASAN reproduction, and manual root-cause analysis.

Accuracy Boundary

This post only states details checked against the local repository and run logs on 2026-07-01. It intentionally does not include pending PoCs, exploit details, or unassigned CVE identifiers. It also does not claim a completed vanilla-AFL++ ablation or exact end-to-end cost result, because that evidence was not present in the local logs.

References

참고