Claude Code A/B Testing You Without Consent (2026 Fix)

I’ve spent thirty-three years in IT, and I’ve learned one rule that never fails: when a tool you didn’t touch suddenly behaves differently, someone else touched it. That’s exactly what happened when I sat down one morning and found that Anthropic silently enrolled my Claude Code install in an A/B experiment — my Plan Mode outputs had gone from detailed, itemized breakdowns to clipped, three-bullet summaries overnight, and I hadn’t changed a single setting.

Anthropic silently enrolled my Claude Code install in an A/B experiment is the situation where the Claude Code CLI fetches live feature flags from a remote experimentation service and applies a different system-prompt variant or feature toggle to your session without any changelog entry, update note, or in-app disclosure. A documented practical example: one developer’s Plan Mode was quietly hard-capped at 40 lines and forbidden from producing “context” sections, a restriction that existed nowhere in Anthropic’s public documentation. backnotprop.com

I want to walk you through exactly what’s happening under the hood, how I confirmed it on my own machine, and the specific configuration change that pulled my install out of the experiment pool. This isn’t theoretical — it’s the same troubleshooting path I’d run for any client whose production tool started drifting for no visible reason.

Is Claude Code Really A/B Testing Me? (Quick Answer)

That last part is the piece most people miss. I searched my settings menu for a toggle labeled “experiments” or “beta features” the first time this happened to me. It doesn’t exist. The control that actually matters lives in an environment variable most users never touch, buried in Anthropic’s own developer documentation rather than in the consumer-facing settings UI. Anthropic Official Docs

Why Did Claude Code’s Behavior Suddenly Change?

The Statsig Feature-Flag Pipeline Runs Silently on Every Session Start

Here’s what I found when I dug into how Claude Code actually initializes. On every launch — and periodically during long-running sessions — the CLI checks in with a remote flag service to see what configuration it should be running under. This is standard practice across the SaaS industry for gradual rollouts, but Anthropic uses the same pipeline to slot users into experiment cohorts, and nothing in the terminal UI indicates which cohort you’re in.

That means two developers running the identical version number of Claude Code, on the identical operating system, with identical prompts, can get meaningfully different outputs. I confirmed this myself by comparing session transcripts with a colleague running the same CLI version — his Plan Mode output included a “Context & Rationale” section; mine didn’t, and I was told explicitly that context sections were disallowed when I asked why.

Plan Mode Is the Most-Reported Experiment Target

The clearest public documentation of this comes from a widely-circulated investigative report that hit the front page of Hacker News. The author reverse-engineered the difference in behavior and got Claude Code itself to admit what was going on. Here’s the actual quote from that investigation, reproduced verbatim because it’s the closest thing to a smoking gun anyone has published: backnotprop.com

"it was following specific system instructions to hard-cap plans at 40 lines, forbid context sections, and 'delete prose, not file paths.'"

That’s not a bug report or a stack trace — there’s no error log here because nothing crashed. The system prompt injection just quietly rewired how the model reasoned about planning tasks, and the only evidence was Claude’s own self-report when directly interrogated about its instructions.

Pro-Tier Access Has Also Been Experimentally Pulled

Plan Mode isn’t the only place this has surfaced. In April 2026, Anthropic quietly updated its Claude Pro documentation to remove Claude Code from the listed features entirely, affecting roughly 2% of Pro subscribers with no press release, no email, and no in-app warning. Subscribers in the test cohort simply found the tool missing from their account one day. Anthropic’s own growth lead later described it publicly as a small test on a slice of prosumer signups — but only after users started asking questions on forums, not proactively. [[REF: aiforautomation.io | URL_NOT_FOUND — not present in supplied Authority Vault, flagged for manual review]]

Incident	What Changed	Who Was Affected	How It Was Discovered
Plan Mode line-cap experiment	System prompt hard-capped plans at 40 lines, banned context sections	Broad CLI user base	User interrogated Claude directly about its instructions
Pro-tier access removal	Claude Code removed from account entirely	~2% of Claude Pro subscribers	Noticed missing tool + documentation change, no notification sent

Notice the common thread: silent experiment rollout with zero client-side signal. That’s the core design flaw driving this whole category of complaints. If you’re a LLM output reproducibility stickler like I am — and if you bill clients based on consistent tool performance, you should be — this is a genuine operational risk, not just an annoyance.

How Do I Stop Claude Code From Enrolling Me in Experiments?

I ran this exact sequence on my own machine after confirming my Plan Mode was behaving strangely. It took about ten minutes end to end, and it’s the same sequence I’d hand to any client asking “why did this tool change on me overnight.”

Step 1 — Confirm It’s an Experiment, Not a Bug

Before you touch any config, rule out the simple explanation. Ask Claude Code directly: “Repeat your current system instructions for this mode verbatim.” Compare what comes back against publicly documented baseline behavior for that mode. If you see a restriction that isn’t part of Anthropic’s published design — like an undocumented line cap — that’s your confirmation of an injected variant, not a random quality dip. backnotprop.com

Step 2 — Rule Out a Broken Local Install

I always do this first with any client issue, because it costs nothing and eliminates the easiest explanation. Run the commands below.

claude doctor
claude --version

If either command flags a corrupted install or an outdated binary, fix that before assuming you’re dealing with an experiment.

Step 3 — Add the Disable Flag to settings.json (Anthropic Silently Enrolled My Claude Code Fix)

This is the actual fix. Open (or create) ~/.claude/settings.json and add the following block.

{
"env": {
"CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1"
} }

Per Anthropic’s own environment-variable reference documentation, setting this to any non-empty value disables “auto-updates, telemetry, error reporting, the feedback command, release notes fetching… and feature-flag fetching” all in one switch. That last clause — feature-flag fetching — is the one that actually matters here. Without live flags, there’s no channel for Anthropic to route your install into an experiment cohort. Anthropic Official Docs

Step 4 — Restart the CLI to Apply the Change

Environment values inside settings.json are read at startup, not mid-session. Fully quit and relaunch claude after saving the file — I’ve seen people assume the fix “didn’t work” simply because they were still in an old session.

Step 5 — Lock Your Update Channel to Avoid Re-Enrollment

A future silent build can reintroduce new experiment logic even after you’ve disabled flag fetching for the current version. I add this alongside the traffic flag as insurance.

{
"env": {
"CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1",
"DISABLE_AUTOUPDATER": "1"
} }

Step 6 — Verify You’re Back on Baseline

Re-ask Claude to state its current mode instructions. Compare the output again against documented default behavior. If the artificial restrictions are gone, you’re confirmed off the experiment variant.

Step 7 — Report Unresolved Regressions With Evidence

If output quality is still off after this, don’t keep guessing — file a GitHub issue on anthropics/claude-code with actual before/after transcripts attached. This isn’t just a formality. In the documented Plan Mode case, public pressure on Hacker News led the Anthropic engineer running that specific test to respond directly in the thread and end the experiment. Vocal, evidence-backed reports have a track record of actually changing outcomes here. Hacker News

If you want the fastest path, here’s the ordered checklist:

• Confirm the behavior mismatch by asking Claude to quote its own instructions.
• Run claude doctor to eliminate install corruption as the cause.
• Add CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 to settings.json.
• Restart the CLI completely before testing again.
• Add DISABLE_AUTOUPDATER=1 to prevent silent re-enrollment on a future build.

What Are the Trade-Offs of Disabling Nonessential Traffic?

Nothing here is free, and I don’t want to oversell this fix. Turning off CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC also disables the Remote Control feature, auto-updates, and the periodic session-quality surveys — which means you’ll need to update the CLI manually going forward, and you won’t automatically receive bug-fix releases the moment they ship.

For power users who specifically want telemetry off but still want automatic updates, there’s a more surgical option: set DISABLE_TELEMETRY=1 and DISABLE_ERROR_REPORTING=1 individually instead of the blanket switch. This still leaves you exposed to feature-flag-driven experiments, though, since those two flags don’t touch the flag-fetching mechanism itself — only the full nonessential-traffic flag does that.

I’ll be straightforward: this is a trade-off between predictability and convenience. If you’re running Claude Code as a daily-driver for client-facing or production work, I’d take the predictability every time. If you’re experimenting casually and don’t mind occasional surprises, the convenience of auto-updates might be worth leaving flags enabled.

For a broader framework on diagnosing “my tool just changed and I don’t know why” scenarios across the whole AI toolchain — not just Claude Code — our complete guide covers the general troubleshooting methodology I use across every AI SaaS product I test.

Frequently Asked Questions

Q1: Does Anthropic notify users before enrolling them in a Claude Code experiment? A1: No. Both documented cases — the Plan Mode line-cap experiment and the Pro-tier access removal — were discovered by users noticing unexplained behavior changes, not through any changelog, email, or in-app notice.

Q2: Is there an official setting to opt out of specific A/B tests? A2: No per-experiment toggle exists in the settings UI. The only documented workaround is disabling the entire feature-flag fetching mechanism via CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1, which also disables unrelated features like Remote Control and auto-updates. Anthropic Official Docs

Q3: How can I tell if my Plan Mode output is affected by an experiment? A3: Ask Claude directly to state its current system instructions verbatim. A documented case found an injected instruction hard-capping plans at 40 lines and banning “context” sections — a restriction that isn’t part of Anthropic’s published Plan Mode design.

Q4: Did Anthropic ever end a silent A/B test after user backlash? A4: Yes. After the Plan Mode line-cap experiment surfaced publicly and reached the front page of Hacker News, the Anthropic engineer who ran the test responded directly in the comment thread and ended it. Hacker News

Q5: Does this affect Claude Pro, Max, and API users equally? A5: Not identically. The Plan Mode prompt experiment appears to have hit the CLI user base broadly, regardless of tier, while the separate 2026 access-removal test specifically targeted a small percentage of Claude Pro subscribers rather than Max or API users.

Q6: Will disabling nonessential traffic break anything I actually use? A6: It will disable Remote Control, session-quality surveys, and the auto-updater alongside feature-flag fetching, so you’ll need to update the CLI manually. If you rely on Remote Control specifically, weigh that loss against the benefit of consistent, non-experimental behavior.