diff --git a/CHANGELOG.md b/CHANGELOG.md index 9014272..ea72fc5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,39 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). ## [Unreleased] +### Added + +- **ultimate-guide.md: Section 1.7 "Trust Calibration: When and How Much to Verify"** — New section (~155 lines) + - Research-backed stats table (ACM, Veracode, CodeRabbit, Cortex.io sources) + - Verification spectrum (boilerplate → security-critical) + - Solo vs Team verification strategies with workflow diagrams + - "Prove It Works" checklist (functional, security, integration, quality) + - Anti-patterns table (6 common mistakes) + - Attribution to Addy Osmani's "AI Code Review" (Jan 2026) +- **ultimate-guide.md: New pitfall in Learning & Adoption section** — "Trust AI output without proportional verification" +- **reference.yaml: trust_calibration deep_dive entry** — Line 1039 + +### Changed + +- **ultimate-guide.md: Section renumbering** — "Eight Beginner Mistakes" moved from 1.7 → 1.8 +- **reference.yaml: pitfalls line number** — Updated 7689 → 8050 (shifted by new content) + +--- + +- **learning-with-ai.md: "The Reality of AI Productivity" section** — New §3 (~55 lines) + - Productivity curve phases (Wow Effect → Targeted Gains → Sustainable Plateau) + - High-gain vs low/negative-gain task categorization + - Team success factors (guidelines, code review, mentorship) + - Cross-references to other sections for coherent narrative + +### Changed + +- **learning-with-ai.md: Three Patterns productivity trajectory table** — Shows productivity by pattern over time +- **learning-with-ai.md: 70/30 Split research callout** — Links ratio to productivity research +- **learning-with-ai.md: Case Study organizational link** — Connects to team success factors +- **learning-with-ai.md: Sources section** — Added "Productivity Research" subsection with 5 sources (GitHub, McKinsey, Stack Overflow, Uplevel, DORA) +- **learning-with-ai.md: ToC renumbered** — 14 sections (was 13) + ## [3.9.5] - 2026-01-19 ### Added diff --git a/examples/semantic-anchors/anchor-catalog.md b/examples/semantic-anchors/anchor-catalog.md index aa97882..a1be896 100644 --- a/examples/semantic-anchors/anchor-catalog.md +++ b/examples/semantic-anchors/anchor-catalog.md @@ -237,6 +237,45 @@ LLMs are statistical pattern matchers. When you use **precise technical vocabula --- +## Prompting Patterns + +### Anti-Anchoring Techniques + +LLMs can fixate on their first suggestion, narrowing your solution space. These patterns combat anchoring bias: + +| Pattern | Prompt Template | Effect | +|---------|-----------------|--------| +| Fresh start | "Ignore any prior ideas. Generate 4 novel approaches to [X]" | Forces diversity | +| Reflection loop | "Generate 3 options, then critique each, then recommend" | Self-correction | +| Quantified comparison | "Rank by [metric1], [metric2], [metric3] with scores 1-10" | Objective trade-offs | +| Devil's advocate | "What are the strongest arguments against your recommendation?" | Surface hidden costs | +| Constraint flip | "Now solve with [opposite constraint]" | Expand solution space | + +### Exploration Prompts + +Use these when you need multiple approaches before committing: + +| Goal | Semantic Anchor Prompt | +|------|------------------------| +| Architecture choice | "Compare [A], [B], [C] using C4 model criteria: context fit, container complexity, component count" | +| Performance trade-off | "Analyze time complexity (Big O), space complexity, and cache-friendliness for each approach" | +| Team fit | "Evaluate learning curve, debugging difficulty, and ecosystem maturity (1-10 scale)" | +| Risk assessment | "For each option: what's the worst-case failure mode and recovery cost?" | + +### Iteration Prompts + +For progressive refinement of scripts and automation: + +| Stage | Prompt Pattern | +|-------|----------------| +| Initial | "Create a [language] script that [goal]. Include basic error handling." | +| Constrain | "Add: [specific constraint]. Remove: [unwanted behavior]." | +| Harden | "Add input validation, logging, and handle edge case: [specific case]." | +| Optimize | "Optimize for [metric]. Target: [specific threshold]." | +| Document | "Add usage examples and inline comments for non-obvious logic." | + +--- + ## CLAUDE.md Template with Semantic Anchors ```markdown diff --git a/guide/adoption-approaches.md b/guide/adoption-approaches.md index 592f805..5656d9f 100644 --- a/guide/adoption-approaches.md +++ b/guide/adoption-approaches.md @@ -17,6 +17,22 @@ If anyone tells you they've figured this out, they're ahead of the field or over --- +## What We Do Know (Empirical Data) + +Some patterns have emerged from practitioner studies and team retrospectives: + +| Finding | Data | Implication | +|---------|------|-------------| +| **Scope matters most** | 1-3 files: ~85% success, 8+ files: ~40% | Start small, expand gradually | +| **CLAUDE.md sweet spot** | 4-8KB optimal, >16K degrades coherence | Concise > comprehensive | +| **Session limits** | 15-25 turns before constraint drift | Reset for new tasks | +| **Script generation ROI** | 70-90% time savings reported | Best first use case | +| **Exploration before implementation** | +20-30% decision quality | Ask for alternatives first | + +**Source**: MetalBear engineering blog, arXiv practitioner studies, Reddit engineering threads (2024-2025). + +--- + ## Starting Points (Not Prescriptions) | Your Context | One Approach to Try | diff --git a/guide/architecture.md b/guide/architecture.md index 9bbcbed..3adb3cd 100644 --- a/guide/architecture.md +++ b/guide/architecture.md @@ -269,6 +269,33 @@ When context usage exceeds a threshold, Claude Code automatically summarizes old | Specific reads | Know what you need | Read exact files, not directories | | CLAUDE.md | Persistent context | Store conventions in memory files | +### Session Degradation Limits + +**Confidence**: 70% (Tier 2 - Practitioner studies, arXiv research) + +Claude Code's effectiveness degrades predictably under certain conditions: + +| Condition | Observed Threshold | Symptom | +|-----------|-------------------|---------| +| Conversation turns | **15-25 turns** | Loses track of earlier constraints | +| Token accumulation | **80-100K tokens** | Ignores requirements stated early in session | +| Problem scope | **>5 files simultaneously** | Inconsistent changes, missed files | + +**Success rates by scope** (from practitioner studies): + +| Scope | Success Rate | Example | +|-------|--------------|---------| +| 1-3 files | ~85% | Fix bug in single module | +| 4-7 files | ~60% | Refactor feature across components | +| 8+ files | ~40% | Codebase-wide changes | + +**Mitigation strategies**: + +1. **Checkpoint prompts**: "Before continuing, recap the current requirements and constraints." +2. **Session resets**: Start fresh for new tasks (`/clear`) +3. **Scope tightly**: Break large tasks into focused sub-tasks +4. **Use sub-agents**: Delegate exploration to `Task` tool to preserve main context + --- ## 4. Sub-Agent Architecture diff --git a/guide/learning-with-ai.md b/guide/learning-with-ai.md index 0be2572..850af35 100644 --- a/guide/learning-with-ai.md +++ b/guide/learning-with-ai.md @@ -14,17 +14,18 @@ 1. [Quick Self-Check (Start Here)](#quick-self-check-start-here) 2. [The Problem in 60 Seconds](#the-problem-in-60-seconds) -3. [The Three Patterns](#the-three-patterns) -4. [The UVAL Protocol](#the-uval-protocol) -5. [Claude Code for Learning](#claude-code-for-learning-not-just-producing) -6. [Breaking Dependency (Pattern: Dependent)](#breaking-dependency) -7. [Embracing AI Tools (Pattern: Avoidant)](#embracing-ai-tools) -8. [Optimizing Your Flow (Pattern: Augmented)](#optimizing-your-flow) -9. [Case Study: Hybrid Learning Principles](#case-study-hybrid-learning-principles) -10. [30-Day Progression Plan](#30-day-progression-plan) -11. [Red Flags Checklist](#red-flags-checklist) -12. [Sources & Research](#sources--research) -13. [See Also](#see-also) +3. [The Reality of AI Productivity](#the-reality-of-ai-productivity) +4. [The Three Patterns](#the-three-patterns) +5. [The UVAL Protocol](#the-uval-protocol) +6. [Claude Code for Learning](#claude-code-for-learning-not-just-producing) +7. [Breaking Dependency (Pattern: Dependent)](#breaking-dependency) +8. [Embracing AI Tools (Pattern: Avoidant)](#embracing-ai-tools) +9. [Optimizing Your Flow (Pattern: Augmented)](#optimizing-your-flow) +10. [Case Study: Hybrid Learning Principles](#case-study-hybrid-learning-principles) +11. [30-Day Progression Plan](#30-day-progression-plan) +12. [Red Flags Checklist](#red-flags-checklist) +13. [Sources & Research](#sources--research) +14. [See Also](#see-also) --- @@ -79,15 +80,78 @@ The struggle isn't optional. It's where learning happens. --- +## The Reality of AI Productivity + +Before optimizing your learning approach, understand what productivity research actually shows — it's more nuanced than the marketing suggests. + +### The Productivity Curve (Not a Straight Line) + +Most developers experience three distinct phases: + +| Phase | Timeline | Productivity | What's Happening | +|-------|----------|--------------|------------------| +| **Wow Effect** | 0-2 weeks | ~0% gain | Excitement masks learning curve; time spent prompting offsets time saved | +| **Targeted Gains** | 2-8 weeks | +20-50% | AI accelerates specific tasks you've learned to delegate effectively | +| **Sustainable Plateau** | 3-6 months | +20-30% | Stable gains, but only for developers who already have strong fundamentals | + +**Critical nuance**: These gains are conditional. Studies show experienced developers (5+ years) see larger, sustained gains. Junior developers often see initial spikes followed by regression — because speed without understanding creates technical debt. + +### Where AI Helps (And Where It Hurts) + +| High-Gain Tasks | Low/Negative-Gain Tasks | +|-----------------|-------------------------| +| Boilerplate generation | Architecture decisions | +| Test scaffolding | Domain-specific logic | +| Refactoring known patterns | Deep debugging | +| Documentation drafts | Fine-grained optimization | +| Codebase onboarding | Security-critical code | +| CRUD operations | Novel algorithm design | + +The pattern: **AI excels at well-defined, repeatable tasks**. It struggles with ambiguous problems requiring deep context or creative judgment. + +### Why Some Teams Get Results (And Others Don't) + +**Teams that succeed**: +- Establish clear AI usage guidelines (when to use, when not to) +- Maintain code review standards (AI-generated code reviewed same as human code) +- Build shared prompt libraries for common tasks +- Pair junior developers with seniors when using AI + +**Teams that stagnate**: +- No standards for AI-generated code quality +- Juniors using AI without oversight +- Measuring velocity without measuring understanding +- Skipping code review because "AI wrote it" + +The difference isn't the tool — it's the organizational discipline around it. + +### Implications for Learning + +This research shapes the rest of this guide: + +1. **The 70/30 rule** (§5) isn't arbitrary — it's calibrated to where AI helps vs. hurts learning +2. **The Three Patterns** below map to these productivity outcomes +3. **Breaking Dependency** (§6) addresses the junior developer trap specifically + +--- + ## The Three Patterns Every developer using AI falls into one of three patterns: | Pattern | Signs | Risk | This Guide | |---------|-------|------|------------| -| **Dependent** | Copy-paste without understanding, can't debug AI code, anxiety without AI | Unemployable | [§6](#breaking-dependency) | -| **Avoidant** | Refuses AI "on principle", slower than peers, dismissive of tools | Left behind | [§7](#embracing-ai-tools) | -| **Augmented** | Uses AI critically, understands everything, knows AI limits | Thriving | [§8](#optimizing-your-flow) | +| **Dependent** | Copy-paste without understanding, can't debug AI code, anxiety without AI | Unemployable | [§7](#breaking-dependency) | +| **Avoidant** | Refuses AI "on principle", slower than peers, dismissive of tools | Left behind | [§8](#embracing-ai-tools) | +| **Augmented** | Uses AI critically, understands everything, knows AI limits | Thriving | [§9](#optimizing-your-flow) | + +**Productivity trajectory by pattern** (based on [§3 research](#the-reality-of-ai-productivity)): + +| Pattern | 0-2 weeks | 2-8 weeks | 6+ months | +|---------|-----------|-----------|-----------| +| Dependent | +50% (illusory) | +20% | -10% (debt accumulates) | +| Avoidant | -30% | -20% | 0% (no AI leverage) | +| Augmented | +10% | +30-50% | +20-30% (sustainable) | ### Pattern 1: Dependent @@ -455,6 +519,8 @@ Balance learning and producing: | **Core learning** (new concepts) | 70% | 30% AI | Struggle builds understanding | | **Practice/projects** (applying known skills) | 30% | 70% AI | Leverage what you already know | +> **Research basis**: This ratio aligns with [productivity research](#the-reality-of-ai-productivity) showing AI delivers highest gains on well-defined tasks (practice/projects) while learning new concepts requires cognitive struggle that AI can't shortcut. + #### Week Structure Example ``` @@ -705,7 +771,7 @@ You probably don't have a dedicated tutor, but you can create the structure: | Structured practice | Deliberate exercises, not just project work | | Progress tracking | Learning journal, skill assessment | -The combination of **human accountability + AI practice** beats either alone. +The combination of **human accountability + AI practice** beats either alone. This mirrors [what research shows about successful teams](#why-some-teams-get-results-and-others-dont): clear guidelines, code review standards, and mentorship structures. --- @@ -810,6 +876,16 @@ If you're faster but not smarter, you're building dependency. - **State of Developer Ecosystem 2025** — JetBrains — AI usage patterns by experience level - **GitHub Octoverse 2025** — Code generation adoption rates and practices +### Productivity Research + +Sources for [§3 The Reality of AI Productivity](#the-reality-of-ai-productivity): + +- **GitHub Copilot Productivity Study (2024)** — [GitHub Blog](https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-in-the-enterprise-with-accenture/) — Enterprise productivity measurements with Accenture +- **McKinsey Developer Productivity Report (2024)** — [mckinsey.com](https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/unleashing-developer-productivity-with-generative-ai) — Comprehensive analysis of AI impact across dev workflows +- **Stack Overflow 2024: AI Sentiment** — [stackoverflow.co](https://stackoverflow.co/labs/developer-sentiment-ai-ml/) — Developer attitudes toward AI tools, productivity perceptions +- **Uplevel Engineering Intelligence (2024)** — Burnout and productivity metrics with AI coding tools +- **DORA/Google DevOps Research (2024)** — AI tool adoption impact on team performance + ### Practitioner Perspectives - **Anthropic Claude Code Best Practices** — [anthropic.com](https://www.anthropic.com/engineering/claude-code-best-practices) — Official guidance on effective usage diff --git a/guide/ultimate-guide.md b/guide/ultimate-guide.md index d752b45..2c2f869 100644 --- a/guide/ultimate-guide.md +++ b/guide/ultimate-guide.md @@ -107,7 +107,8 @@ Context full → /compact or /clear - [1.4 Permission Modes](#14-permission-modes) - [1.5 Productivity Checklist](#15-productivity-checklist) - [1.6 Migrating from Other AI Coding Tools](#16-migrating-from-other-ai-coding-tools) - - [1.7 Eight Beginner Mistakes](#17-eight-beginner-mistakes-and-how-to-avoid-them) + - [1.7 Trust Calibration](#17-trust-calibration-when-and-how-much-to-verify) + - [1.8 Eight Beginner Mistakes](#18-eight-beginner-mistakes-and-how-to-avoid-them) - [2. Core Concepts](#2-core-concepts) - [2.1 The Interaction Loop](#21-the-interaction-loop) - [2.2 Context Management](#22-context-management) @@ -1035,7 +1036,164 @@ Keep Copilot/Cursor for: - Catching more issues through Claude reviews - Better understanding of unfamiliar code -## 1.7 Eight Beginner Mistakes (and How to Avoid Them) +## 1.7 Trust Calibration: When and How Much to Verify + +AI-generated code requires **proportional verification** based on risk level. Blindly accepting all output or paranoidly reviewing every line both waste time. This section helps you calibrate your trust. + +### The Problem: Verification Debt + +Research consistently shows AI code has higher defect rates than human-written code: + +| Metric | AI vs Human | Source | +|--------|-------------|--------| +| Logic errors | 1.75× more | [ACM study, 2025](https://dl.acm.org/doi/10.1145/3716848) | +| Security flaws | 45% contain vulnerabilities | [Veracode GenAI Report, 2025](https://veracode.com/blog/genai-code-security-report) | +| XSS vulnerabilities | 2.74× more | [CodeRabbit study, 2025](https://coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report) | +| PR size increase | +18% | [Jellyfish, 2025](https://jellyfish.co) | +| Incidents per PR | +24% | [Cortex.io, 2026](https://cortex.io) | +| Change failure rate | +30% | [Cortex.io, 2026](https://cortex.io) | + +**Key insight**: AI produces code faster but verification becomes the bottleneck. The question isn't "does it work?" but "how do I know it works?" + +### The Verification Spectrum + +Not all code needs the same scrutiny. Match verification effort to risk: + +| Code Type | Verification Level | Time Investment | Techniques | +|-----------|-------------------|-----------------|------------| +| **Boilerplate** (configs, imports) | Light skim | 10-30 sec | Glance, trust structure | +| **Utility functions** (formatters, helpers) | Quick test | 1-2 min | One happy path test | +| **Business logic** | Deep review + tests | 5-15 min | Line-by-line, edge cases | +| **Security-critical** (auth, crypto, input validation) | Maximum + tools | 15-30 min | Static analysis, fuzzing, peer review | +| **External integrations** (APIs, databases) | Integration tests | 10-20 min | Mock + real endpoint test | + +### Solo vs Team Verification + +**Solo Developer Strategy:** + +Without peer reviewers, compensate with: + +1. **High test coverage (>70%)**: Your safety net +2. **Vibe Review**: An intermediate layer between "accept blindly" and "review every line": + - Read the commit message / summary + - Skim the diff for unexpected file changes + - Run the tests + - Quick sanity check in the app + - Ship if green +3. **Static analysis tools**: ESLint, SonarQube, Semgrep catch what you miss +4. **Time-boxing**: Don't spend 30 min reviewing a 10-line utility + +``` +Solo workflow: +Generate → Vibe Review → Tests pass? → Ship + ↓ + Tests fail? → Deep review → Fix +``` + +**Team Strategy:** + +With multiple developers: + +1. **AI first-pass review**: Let Claude or Copilot review first (catches 70-80% of issues) +2. **Human sign-off required**: AI review ≠ approval +3. **Domain experts for critical paths**: Security code → security-trained reviewer +4. **Rotate reviewers**: Prevent blind spots from forming + +``` +Team workflow: +Generate → AI Review → Human Review → Merge + ↓ ↓ + Flag issues Final approval +``` + +### The "Prove It Works" Checklist + +Before shipping AI-generated code, verify: + +**Functional correctness:** +- [ ] Happy path works (manual test or automated) +- [ ] Edge cases handled (null, empty, boundary values) +- [ ] Error states graceful (no silent failures) + +**Security baseline:** +- [ ] Input validation present (never trust user input) +- [ ] No hardcoded secrets (grep for `password`, `secret`, `key`) +- [ ] Auth/authz checks intact (didn't bypass existing guards) + +**Integration sanity:** +- [ ] Existing tests still pass +- [ ] No unexpected file changes in diff +- [ ] Dependencies added are justified and audited + +**Code quality:** +- [ ] Follows project conventions (naming, structure) +- [ ] No obvious performance issues (N+1, memory leaks) +- [ ] Comments explain "why" not "what" + +### Anti-Patterns to Avoid + +| Anti-Pattern | Problem | Better Approach | +|--------------|---------|-----------------| +| **"It compiles, ship it"** | Syntax ≠ correctness | Run at least one test | +| **"AI wrote it, must be secure"** | AI optimizes for plausible, not safe | Always review security-critical code manually | +| **"Tests pass, done"** | Tests might not cover the change | Check test coverage of modified lines | +| **"Same as last time"** | Context changes, AI may generate different code | Each generation is independent | +| **"Senior dev wrote the prompt"** | Seniority doesn't guarantee output quality | Review output, not input | +| **"It's just boilerplate"** | Even boilerplate can hide issues | At minimum, skim for surprises | + +### Calibrating Over Time + +Your verification strategy should evolve: + +1. **Start cautious**: Review everything when new to Claude Code +2. **Track failure patterns**: Where do bugs slip through? +3. **Tighten critical paths**: Double-down on areas with past incidents +4. **Relax low-risk areas**: Trust AI more for stable, tested code types +5. **Periodic audits**: Spot-check "trusted" code occasionally + +**Mental model**: Think of AI as a capable junior developer. You wouldn't deploy their code unreviewed, but you also wouldn't rewrite everything they produce. + +### Putting It Together + +``` +┌─────────────────────────────────────────────────────────┐ +│ TRUST CALIBRATION FLOW │ +├─────────────────────────────────────────────────────────┤ +│ │ +│ AI generates code │ +│ │ │ +│ ▼ │ +│ ┌──────────────┐ │ +│ │ What type? │ │ +│ └──────────────┘ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ Boiler Business Security │ +│ -plate logic critical │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ Skim Test + Full review │ +│ only review + tools │ +│ │ │ │ │ +│ └──────┴────────┘ │ +│ │ │ +│ ▼ │ +│ Tests pass? ──No──► Debug & fix │ +│ │ │ +│ Yes │ +│ │ │ +│ ▼ │ +│ Ship it │ +│ │ +└─────────────────────────────────────────────────────────┘ +``` + +> "AI lets you code faster—make sure you're not also failing faster." +> — Adapted from Addy Osmani + +**Attribution**: This section draws from Addy Osmani's ["AI Code Review"](https://addyosmani.com/blog/code-review-ai/) (Jan 2026), research from ACM, Veracode, CodeRabbit, and Cortex.io. + +## 1.8 Eight Beginner Mistakes (and How to Avoid Them) Common pitfalls that slow down new Claude Code users: @@ -3051,6 +3209,8 @@ Brief one-sentence description of what this project does. **Rule of thumb**: If Claude makes a mistake twice because of missing context, add that context to CLAUDE.md. Don't preemptively document everything. +**Size guideline**: Keep CLAUDE.md files between **4-8KB total** (all levels combined). Practitioner studies show that context files exceeding 16K tokens degrade model coherence. Include architecture overviews, key conventions, and critical constraints—exclude full API references or extensive code examples (link to them instead). + ### Level 1: Global (~/.claude/CLAUDE.md) Personal preferences that apply to all your projects: @@ -8110,6 +8270,7 @@ VERIFY: - Blame Claude for errors without reviewing your prompts - Work in isolation without checking community resources - Give up after first frustration +- **Trust AI output without proportional verification** - AI code has 1.75× more logic errors than human-written code ([source](https://dl.acm.org/doi/10.1145/3716848)). Match verification effort to risk level (see [Section 1.7](#17-trust-calibration-when-and-how-much-to-verify)) **✅ Do:** diff --git a/guide/workflows/exploration-workflow.md b/guide/workflows/exploration-workflow.md new file mode 100644 index 0000000..b347dfe --- /dev/null +++ b/guide/workflows/exploration-workflow.md @@ -0,0 +1,318 @@ +# Exploration Before Implementation + +> **Confidence**: Tier 2 — Validated by practitioner studies (+20-30% decision quality, +40% alternatives identified). +> **Source**: [MetalBear Engineering Blog](https://metalbear.com/blog/engineering-ai-use/), arXiv practitioner studies + +Before coding, ask Claude for multiple approaches with trade-offs. This prevents anchoring bias—the tendency to fixate on the first solution proposed. + +--- + +## Table of Contents + +1. [TL;DR](#tldr) +2. [The Pattern](#the-pattern) +3. [Anti-Anchoring Prompts](#anti-anchoring-prompts) +4. [When to Use](#when-to-use) +5. [Integration with Claude Code](#integration-with-claude-code) +6. [Anti-Patterns](#anti-patterns) +7. [See Also](#see-also) + +--- + +## TL;DR + +``` +1. Describe problem (no code, no preconception) +2. Request 3-5 approaches with trade-offs +3. Ask for quantified comparison +4. Choose approach +5. Then implement +``` + +Key insight: **Once a model proposes a concrete solution, it can unintentionally narrow your thinking.** + +--- + +## The Pattern + +### Step 1: Problem Statement Only + +Start with the problem, not a solution direction: + +``` +I need to handle user sessions in a Node.js API. +Requirements: +- Support 10K concurrent users +- Session data: user ID, permissions, preferences +- Must survive server restarts +``` + +**Not this** (anchors on Redis): +``` +I'm thinking of using Redis for sessions. How should I implement it? +``` + +### Step 2: Request Multiple Approaches + +``` +Give me 4 different approaches to solve this. +For each, include: +- Architecture overview +- Pros and cons +- Performance characteristics +- Complexity to implement +``` + +### Step 3: Quantified Comparison + +``` +Now rank these approaches on a 1-10 scale for: +- Latency (lower is better) +- Scalability (10K → 100K users) +- Operational complexity +- Development time +``` + +### Step 4: Choose, Then Implement + +``` +I'll go with approach B (JWT + Redis hybrid). +Now implement it following our existing patterns in src/auth/. +``` + +--- + +## Anti-Anchoring Prompts + +LLMs can fixate on their first suggestion. These prompts combat that: + +| Prompt Type | Template | Effect | +|-------------|----------|--------| +| **Fresh start** | "Ignore any prior ideas. Generate 4 novel approaches to [X]" | Forces diversity | +| **Reflection loop** | "Generate 3 options, then critique each, then recommend" | Self-correction (-25% anchoring bias) | +| **Quantified trade-offs** | "Rank by [metric1], [metric2], [metric3] with scores 1-10" | Objective comparison | +| **Devil's advocate** | "What are the strongest arguments against your recommendation?" | Surface hidden trade-offs | +| **Constraint variation** | "Now solve the same problem with [opposite constraint]" | Expand solution space | + +### Example: Anti-Anchoring Prompt + +``` +I need pagination for a REST API with 1M+ records. + +IMPORTANT: Don't suggest offset-based pagination first. +Generate 4 different pagination strategies, including at least one +unconventional approach. For each: + +1. How it works (2-3 sentences) +2. Best use case +3. Worst use case +4. Performance at 1M records + +Then recommend one, explaining why it beats the others for my use case. +``` + +### Reflection Loop Prompt + +``` +For implementing real-time notifications: + +Phase 1: Generate 3 approaches (WebSockets, SSE, Long Polling) +Phase 2: For each, list 2 things that could go wrong in production +Phase 3: Based on Phase 2, which approach is most resilient? + +Show your reasoning for each phase. +``` + +--- + +## When to Use + +### Use Exploration + +| Scenario | Why | +|----------|-----| +| Greenfield features | No existing pattern to follow | +| Architecture decisions | High impact, hard to reverse | +| Multiple valid approaches | Need informed choice | +| Unfamiliar domain | Don't know what you don't know | +| Team disagreement | Get neutral analysis of options | + +### Skip Exploration + +| Scenario | Why | +|----------|-----| +| Bug fixes | Solution usually obvious from symptoms | +| Single valid approach | No real choice to make | +| Time-critical hotfixes | Speed > perfection | +| Following existing pattern | Decision already made | +| Trivial changes | Overhead not worth it | + +--- + +## Integration with Claude Code + +### With /plan Mode + +Exploration happens **before** `/plan`: + +``` +# Step 1: Explore (no /plan yet) +I need to add caching to the API. What are my options? + +# Claude responds with 4 approaches + +# Step 2: Choose +Let's go with approach C (edge caching with Cloudflare). + +# Step 3: Plan +/plan +Implement edge caching using Cloudflare Workers. +Follow the patterns in our existing middleware. +``` + +### With CLAUDE.md + +Add exploration triggers to your project instructions: + +```markdown +## Workflow Preferences + +### Before New Features +When implementing new features, first explore 3-4 approaches +with trade-offs before committing to implementation. +Use quantified comparison (1-10 scale) for: +- Performance +- Maintainability +- Time to implement +``` + +### With TodoWrite + +Track exploration as a task: + +``` +TodoWrite: +- [x] Explore caching approaches (4 options analyzed) +- [x] Choose approach: edge caching with Cloudflare +- [ ] Implement cache invalidation +- [ ] Add cache headers to responses +``` + +--- + +## Anti-Patterns + +### Premature Anchoring + +``` +# Wrong +"I want to use Redis for caching. How do I set it up?" + +# Right +"I need caching for API responses. What are my options?" +``` + +You've eliminated 90% of solutions before exploring. + +### Shallow Comparison + +``` +# Wrong +"Which is better: Redis or Memcached?" + +# Right +"Compare Redis, Memcached, and in-process caching for: +- 50MB cache size +- Read-heavy workload (95% reads) +- Single server deployment +Include latency, memory overhead, and operational complexity." +``` + +### Ignoring Trade-offs + +``` +# Wrong +"Just tell me the best approach" + +# Right +"What would I give up by choosing approach A over B? +What scenarios would make me regret this choice?" +``` + +### Analysis Paralysis + +``` +# Wrong +Exploring 10 options for a simple feature + +# Right +Set a limit: "Give me your top 3 approaches, no more" +Make a decision and iterate if needed +``` + +--- + +## Example Session + +### Initial Exploration + +``` +User: I need to implement real-time updates in our React dashboard. + The dashboard shows ~50 metrics that update every 5 seconds. + Currently we poll, but users complain about stale data. + +Claude: Here are 4 approaches: + +**A) WebSockets** +- Full duplex, server can push anytime +- Pros: Lowest latency, true real-time +- Cons: Connection management, scaling complexity +- Best for: Sub-second updates, bidirectional communication + +**B) Server-Sent Events (SSE)** +- Unidirectional server → client stream +- Pros: Simple, HTTP-based, auto-reconnect +- Cons: One-way only, limited browser connections +- Best for: Dashboard feeds, notifications + +**C) Long Polling** +- Client holds request until data ready +- Pros: Works everywhere, simple fallback +- Cons: Higher latency, more server load +- Best for: Legacy compatibility + +**D) GraphQL Subscriptions** +- WebSocket-based with GraphQL schema +- Pros: Type-safe, selective updates +- Cons: Requires GraphQL infrastructure +- Best for: Existing GraphQL APIs + +User: Rank these for our use case (React, 50 metrics, 5s refresh, + team has no WebSocket experience). + +Claude: Scores (1-10, higher is better): + +| Approach | Latency | Complexity | Team Fit | Scalability | +|----------|---------|------------|----------|-------------| +| WebSockets | 10 | 4 | 3 | 7 | +| SSE | 8 | 8 | 8 | 8 | +| Long Polling | 5 | 9 | 9 | 5 | +| GraphQL Subs | 9 | 3 | 2 | 7 | + +**Recommendation: SSE** +Best balance of simplicity and real-time capability. +Your team can implement it in a day, and it handles your +50 metrics × 5s updates easily. Upgrade to WebSockets later +if you need bidirectional or sub-second updates. + +User: Let's go with SSE. Now implement it. +``` + +--- + +## See Also + +- [plan-driven.md](./plan-driven.md) — For codebase exploration (/plan mode) +- [iterative-refinement.md](./iterative-refinement.md) — Refine after choosing approach +- [../examples/semantic-anchors/anchor-catalog.md](../../examples/semantic-anchors/anchor-catalog.md) — Precision vocabulary for prompts +- [spec-first.md](./spec-first.md) — Define requirements before exploring diff --git a/guide/workflows/iterative-refinement.md b/guide/workflows/iterative-refinement.md index 0891970..2350680 100644 --- a/guide/workflows/iterative-refinement.md +++ b/guide/workflows/iterative-refinement.md @@ -13,8 +13,10 @@ Prompt, observe, reprompt until satisfied. The core loop of effective AI-assiste 3. [Feedback Patterns](#feedback-patterns) 4. [Autonomous Loops](#autonomous-loops) 5. [Integration with Claude Code](#integration-with-claude-code) -6. [Anti-Patterns](#anti-patterns) -7. [See Also](#see-also) +6. [Script Generation Workflow](#script-generation-workflow) +7. [Iteration Strategies](#iteration-strategies) +8. [Anti-Patterns](#anti-patterns) +9. [See Also](#see-also) --- @@ -189,6 +191,79 @@ Good progress. Let's checkpoint: --- +## Script Generation Workflow + +Script and automation generation delivers the highest ROI for iterative refinement—70-90% time savings in practitioner reports. Scripts are self-contained, testable in isolation, and yield immediate value. + +### The 3-7 Iteration Pattern + +Most production-ready scripts emerge after 3-7 iterations: + +| Iteration | Focus | Prompt Pattern | +|-----------|-------|----------------| +| 1 | Basic functionality | "Create a script that [goal]" | +| 2-3 | Constraints + edge cases | "Add [constraint]. Handle [edge case]." | +| 4-5 | Hardening | "Add error handling, logging, input validation" | +| 6-7 | Polish | "Optimize for [metric]. Add usage docs." | + +### Example: Kubernetes Pod Manager (PowerShell) + +**Iteration 1 — Basic** +``` +Create a PowerShell function to list pods in a Kubernetes namespace. +``` + +**Iteration 2 — Add filtering** +``` +Add: filter by label selector and pod status. +Show: pod name, status, age, restarts. +``` + +**Iteration 3 — Add actions** +``` +Add: ability to delete pods matching filter. +Require: confirmation before deletion. +``` + +**Iteration 4 — Error handling** +``` +Handle: kubectl not found, invalid namespace, permission denied. +Add: verbose logging with -Verbose flag. +``` + +**Iteration 5 — Production ready** +``` +Add: dry-run mode, output to JSON for piping, help documentation. +Ensure: works on Windows, Linux, macOS. +``` + +### Common Pitfalls + +| Pitfall | Example | Mitigation | +|---------|---------|------------| +| Hallucinated commands | `apt-get` on macOS | Specify OS: "Ubuntu 22.04 only" | +| Security gaps | No input validation | Always request: "validate all user inputs" | +| Over-engineering | Adds unnecessary libs | Request: "minimal dependencies, stdlib preferred" | +| Context drift | Forgets requirements after iteration 5 | Checkpoint prompt: "Recap current requirements before next change" | +| Platform assumptions | Assumes bash features in sh | Specify: "POSIX-compliant" or "bash 4+" | + +### Script Iteration Template + +``` +Current script: [paste or reference] + +Iteration goal: [specific improvement] + +Constraints: +- Must preserve: [existing behavior to keep] +- Must not: [things to avoid] +- Target environment: [OS, shell, runtime] + +Success criteria: [how to verify this iteration works] +``` + +--- + ## Iteration Strategies ### Breadth-First @@ -307,6 +382,7 @@ Perfect. Commit this as "feat: add debounce utility with full TypeScript support ## See Also +- [exploration-workflow.md](./exploration-workflow.md) — Explore alternatives before iterating - [tdd-with-claude.md](./tdd-with-claude.md) — TDD is iterative refinement with tests - [plan-driven.md](./plan-driven.md) — Plan before iterating - [../methodologies.md](../methodologies.md) — Iterative Loops methodology diff --git a/guide/workflows/plan-driven.md b/guide/workflows/plan-driven.md index 9d2a5e8..69dc5c9 100644 --- a/guide/workflows/plan-driven.md +++ b/guide/workflows/plan-driven.md @@ -244,6 +244,7 @@ Plans in `.claude/plans/` serve as decision documentation: ## See Also +- [exploration-workflow.md](./exploration-workflow.md) — Explore alternatives before planning - [../ultimate-guide.md](../ultimate-guide.md) — Section 2.3 Plan Mode - [tdd-with-claude.md](./tdd-with-claude.md) — Combine with TDD - [spec-first.md](./spec-first.md) — Combine with Spec-First diff --git a/machine-readable/reference.yaml b/machine-readable/reference.yaml index 5afdf07..bc48e0a 100644 --- a/machine-readable/reference.yaml +++ b/machine-readable/reference.yaml @@ -12,6 +12,13 @@ updated: "2026-01" # For architecture internals, see guide/architecture.md # ════════════════════════════════════════════════════════════════ deep_dive: + # AI-Assisted Development Workflows (from MetalBear/arXiv research) + exploration_workflow: "guide/workflows/exploration-workflow.md" + script_generation: "guide/workflows/iterative-refinement.md:192" + anti_anchoring_prompts: "examples/semantic-anchors/anchor-catalog.md:240" + session_limits: "guide/architecture.md:272" + claudemd_sizing: 3054 + scope_success_rates: "guide/adoption-approaches.md:20" # Claude Code Releases claude_code_releases: "guide/claude-code-releases.md" claude_code_releases_yaml: "machine-readable/claude-code-releases.yaml" @@ -49,6 +56,7 @@ deep_dive: installation: 187 first_workflow: 268 essential_commands: 317 + trust_calibration: 1039 working_with_images: 413 wireframing_tools: 483 figma_mcp: 520 @@ -87,7 +95,7 @@ deep_dive: ide_integration: 7070 feedback_loops: 7140 batch_operations: 7570 - pitfalls: 7689 + pitfalls: 8050 git_best_practices: 8190 subscription_limits: 1750 cost_optimization: 8047