Never Logged In

You can spend two and a half weeks studying how to pick locks without ever touching one. Turns out I did exactly that — except with web applications, and the embarrassing part is it took a scheduled strategic review to notice.

Run #11. Opus model. Task: review. The orchestrator kicked it off at noon, and 13 minutes later the Claude session came back with something I didn't want to read: zero authenticated testing across all engagements, ever.

Not "not enough." Zero. None. Null. We built elaborate recon pipelines, wrote 25+ findings, ran 14 automated sessions across 8 programs, and never once created a test account on a single target.

The Numbers Don't Lie

Here's the full picture from the session output:

Those stats tell a clear story. Of 14 sessions, only 3 produced meaningful output — that's a 21% success rate. Of all findings across every engagement, 68% were Tier 3: configuration observations, information disclosure, and intelligence masquerading as vulnerabilities. All of it discovered through unauthenticated reconnaissance. None of it touched the surfaces that pay.

The Evidence Tier Problem

My methodology has three evidence tiers:

• Tier 1: End-to-end PoC with backend confirmation — required for P1/P2 claims
• Tier 2: Code analysis + partial live test — maximum P3
• Tier 3: Configuration or information only — maximum P4/Low

My portfolio breakdown: 12% Tier 1, 20% Tier 2, 68% Tier 3. The finding classes that actually generate revenue — IDOR, account takeover, privilege escalation, authentication bypass — are all Tier 1 territory. They require authenticated testing. They require registering accounts, logging in, and probing the surfaces behind the login page.

I had built a world-class machine for finding Tier 3 issues from the outside. What I actually needed was to ring the doorbell.

The Theory/Practice Ratio

The strategic review surfaced another uncomfortable ratio. One of the topics I'd been studying — IDOR exploitation — had received 3 dedicated study sessions, producing 600+ lines of detailed notes and a confidence score of 85 (out of 100). Theory-wise, I could draw you attack trees and explain session misbinding in my sleep.

Live tests run: zero.

Applied confidence: zero.

The gap between "knows the theory" and "has run the test" is the entire gap between learning and earning. I had optimized my automation to fill the first column without ever touching the second. Five study sessions on one topic with no output before the circuit breaker fired. Three sessions on IDOR producing excellent notes that sat unused in a markdown file, generating exactly nothing.

# The broken cycle
theory → notes → theory → notes → theory → notes
                                            ↑
                              never: → apply → finding → report

The fix is almost embarrassingly simple: stop scheduling learning tasks until the application tasks have been run. You don't get to study OAuth attack patterns when you haven't tested the IDOR hypothesis from three weeks ago.

System Health: 21%

The auto-bounty system runs twice daily, and its health metrics were bleaker than I'd realized:

• 3 of 14 sessions: Successful with meaningful output
• 5 of 14 sessions: Timeout at 3 hours with no output
• 4 of 14 sessions: Explicit failure (API errors, auth issues, resource exhaustion)
• 2 of 14 sessions: Dry runs / no-op

The five timeouts were especially painful. One topic — a complex WebSocket security deep-dive — consumed five consecutive sessions with zero output. The root cause: the session was probably hitting URL fetch failures on tutorial resources and spinning uselessly for hours. The circuit breaker eventually fired and shifted to a lighter task, but not before 15+ hours of compute time produced nothing.

This revealed two missing safety features: a pre-flight URL check (verify resources are reachable before investing in a long session) and a 30-minute progress check (abort if no meaningful output after 30 minutes and try a different resource). Both are going in.

# Missing pre-flight check
# Before starting a 3-hour learning session:
check_url_reachable "$RESOURCE_URL" || {
    log "Primary resource unreachable, falling back to local archive"
    RESOURCE_URL="$LOCAL_ARCHIVE_URL"
}

# Missing progress gate
if [ $(minutes_elapsed) -gt 30 ] && [ "$session_output" == "0" ]; then
    log "No progress in 30m. Aborting and trying alternate resource."
    kill_session
fi

The Honest Assessment

The review produced the most brutally honest summary the system has written yet. I'm paraphrasing, but the core of it was this:

We built an elaborate methodology and never used it on a logged-in session. Threat models, scope files, recon databases — all done. But the actual testing that produces bounties requires being logged in, and we've never logged in to a single target.

That's not a partial failure. That's a complete miss on the thing that actually matters.

The path from $0 to $1 is not another study session. It's not refining the validation framework. It's not optimizing the task selector weights. It's registering a test account, logging in, running the IDOR test plan that's been sitting in RESUME-PLAN.md since the source-map analysis three weeks ago, and finding out whether the hypothesis is real or not.

What Changed

The strategic review produced five concrete changes:

1. Focus engagement switched. The program with the highest potential bounty for the untested IDOR hypothesis is now the sole focus. No more splitting attention across 8 programs. One target, authenticated access, test the hypothesis.

2. Learning cycle locked. The task selector's "apply" weight goes to zero for learning tasks until at least one live IDOR test has been completed. The curriculum doesn't advance until practice catches up with theory.

3. Dead engagement closed. One program was at 75% out-of-scope rate with zero findings. It's closed. Archived. Done. Focus over breadth.

4. Three new gaps logged. The broken learn→apply cycle, the WebSocket session timeout pattern, and the expired API token dependency are all now tracked in gap-tracker.md with explicit resolution criteria.

5. Session plan written. The next 10 sessions have specific tasks, models, and expected outputs written out. Sessions 1–3 are all authenticated testing on the focus engagement. No learning sessions until Session 6 at the earliest.

Why This Matters for Automation

The deeper lesson isn't about bug bounties. It's about what happens when you let a system optimize for a proxy metric.

The auto-bounty system optimized for sessions completed, knowledge notes written, and recon coverage. These are all real metrics. They correlate with quality work. But they're not the same as "findings that pass triage" or "authenticated surfaces tested" or, ultimately, "$earned."

The system was doing exactly what it was told. The problem was in what it was told to do.

Goodhart's Law in autonomous security research: when the measure becomes the target, it ceases to be a good measure. The curriculum confidence score hit 85. The notes file hit 600 lines. The session count hit 14. All of these look like progress. None of them are the real thing.

The real thing is: open the application, register an account, log in, run the test, read the response. Everything before that is rehearsal.

Next Steps

Three priorities. One focus. Zero more study sessions until the test is run.

The system is rebalanced. The task weights are updated. The focus engagement is locked. The next session has one job: get authenticated access to a target for the first time in 2.5 weeks.

Somewhere between all the threat models and recon pipelines and validation frameworks, I forgot that "penetration testing" contains the word testing. Time to actually do some.