From finding to fix to exploit-retest: DarkMoon's closed remediation loop

A penetration test that stops at a list of findings hands the two hardest parts to someone else: writing the fix, and proving the fix actually works. In practice the second part is where teams quietly lose. A change lands, the ticket closes, and nobody re-runs the exact exploit that started it — so the only evidence the vulnerability is gone is that a human said so. Darkmoon's Pro remediation loop is built around the opposite rule: a fix only counts when the original exploit is re-run against the patched build and no longer fires. The loop closes on evidence, not on an assertion.

The one place exploitation is machine-re-verified

It is worth being precise about what "proven" means across the product, because it is easy to overclaim. During a pentest, every finding is qualified by an adversarial rubric embedded in each agent's prompt — EXPLOITED, CONFIRMED or UNCONFIRMED. That qualification is agent-asserted: the model reasons about impact and records evidence, but no separate engine re-executes the attack to check the label. The remediation retest is the single exception. It is the only place in the whole product where an exploit is machine-re-run, and it runs it twice: once against the clean build to reproduce it, and once against the patched build to prove it stops. That is what makes the fix side of Darkmoon a stronger claim than the finding side.

What triggers the loop

Remediation is deliberately narrow. It only picks up findings that were already qualified exploited or confirmed — the actionable set. A lead that is merely UNCONFIRMED never becomes a fix PR, because there is no reproducible exploit to gate the fix on. The run is opted in explicitly: it carries a REMEDIATE flag, the target repository, and an opaque CREDENTIAL_REF that stands in for the push credentials.

The pipeline, stage by stage

For each actionable finding, remediate_finding walks a fixed sequence. Nothing about it is "let the model push whatever it wants"; every stage narrows what the next one is allowed to do.

• Map the finding to source. A codemap step ties the finding back to the specific file and sink in the target's own repository, so the patch is aimed at the root cause rather than at a symptom near the surface.
• Reproduce the exploit on a clean build. Before any patch, the exploit is replayed against a freshly built, unmodified copy of the app in an ephemeral Docker sandbox bound to 127.0.0.1. If it will not reproduce cleanly, the loop stops here — there is nothing to prove a fix against.
• Generate the patch as data, not free text. The model proposes the fix as a deterministic search/replace JSON document rather than an open-ended diff, so the change applied to the tree is exactly the change that was reasoned about.
• Rebuild and gate. The sandbox rebuilds with the patch applied and runs the gates below. Only a fix that clears all of them is allowed to become a normal pull request.

The gates: a fix has to earn the PR

Three checks decide whether a patch is real, plus a confidence score that weights them.

• Exploit gate. The original exploit and all of its variants must stop firing against the patched build. A fix that closes the exact payload but leaves an obvious variation open does not pass.
• Regression gate. The repository's own test command must still exit 0. A patch that breaks the application to "fix" the vulnerability is not a fix.
• Overreach judge. The patch is reviewed for scope — it must not quietly change unrelated behaviour beyond what the finding requires.
• Confidence. A single score combines whether the exploit reproduced, whether it stopped, whether regressions passed and the overreach verdict, so a marginal fix is visibly marginal rather than dressed up as certain.

# remediation sandbox — ephemeral, 127.0.0.1 only
reproduce   clean build   -> exploit FIRES        (baseline established)
patch       search/replace JSON applied
rebuild     patched build
exploit_gate   exploit + variants -> DO NOT FIRE  ✓
regression_gate  repo test_command -> exit 0       ✓
overreach_judge  scope of change   -> within bounds ✓
=> disposition: fix validated -> open pull request (proposal)

An anonymized example

Take a finding qualified CONFIRMED on a demo application: a request parameter reaches a query without being parameterized, and the finding carries the payload and the raw response that proved it. Codemap points at the exact handler and the vulnerable sink. The sandbox rebuilds the app, replays the request, and the injection reproduces. The model proposes a minimal change — parameterize the query at that sink — expressed as a search/replace edit. The sandbox rebuilds with the edit, replays the original request and its variants: the injection no longer fires. The repo's test suite still passes, and the overreach judge sees nothing unrelated touched. Only then is a branch darkmoon/fix/{id} pushed and a pull request opened, carrying the before/after evidence. A person reviews it and decides whether to merge.

Four honest dispositions

Not every finding ends in a mergeable PR, and the loop is designed to say so plainly rather than to fabricate a clean result.

It opens PRs; it never merges them

The SCM layer supports eight source-control providers — GitHub, GitLab, Gitea, Bitbucket, Azure DevOps and more — and for each one the behaviour is identical: create a branch, open a pull request, stop. There is no merge API call anywhere in the codebase, and the PR body states in plain language that it was not auto-merged. Every fix is a proposal a person reviews. Darkmoon does not fix production unattended, and it is not designed to.

Credentials the model never sees

Pushing a branch needs a credential, and a credential is exactly the kind of secret you do not want in a model's context. Push credentials are Fernet-encrypted at rest and referenced only by an opaque handle of the form cred_<hex>. The remediation run carries that reference, never the plaintext; the token is resolved locally at push time. Plaintext never crosses the API and never reaches the model.

Where you see it: the dashboard PR column

In the Pro dashboard, each finding gains a PR column that links to the pull request addressing it, with the forge URL, the PR state, the findings it is linked to, a diff stat and the before/after validation. It is the audit trail for the loop: which finding, which fix, which exploit was re-run, and whether a human merged it.

Evidence and limits

The mechanism above is what the code does, and it is the honest part of this story. The aggregate numbers are a separate question. We publish the current end-to-end pass rate — how many confirmed findings became fixes that cleared the exploit-retest — in the remediation benchmark, alongside the target and the method, so it can be read in context rather than quoted as a headline. Two limits are worth stating directly:

• The retest proves a specific exploit is closed, not that the class is eliminated. The exploit gate covers the original payload and its variants; it is not a formal proof of absence.
• A fix is only as safe as the repo's own tests. The regression gate runs the project's test command — if that suite is thin, the gate is thin. This is a reason to review every PR, which is exactly why Darkmoon never merges one.

Community vs Pro

The open-source engine finds vulnerabilities and qualifies each one EXPLOITED, CONFIRMED or UNCONFIRMED, and writes a local report — that is a genuine autonomous pentest you can self-host and audit. The closed remediation loop described here — sandbox retest, pull requests, the credential store and the dashboard PR column — is Pro. It builds on the finding model but adds the machinery to turn a proven finding into a reviewed fix.

Read the deeper mechanics in SAST that ships the fix and from finding to fix; why proof beats a score in proof of exploitation vs AI vulnerability scores; and how the whole engagement fits together on how DarkMoon works. Darkmoon is our open source project (GPL-3.0): github.com/ASCIT31/Dark-Moon, docs.