Chain-of-Thought Prompting: When It Helps and When It Wastes Tokens

Chain-of-thought prompting gets treated as a free upgrade in most tutorials: add "think step by step" and watch accuracy climb. The reality is more conditional. A 2025 meta-analysis from researchers at UT Austin, Johns Hopkins, and Princeton covered over 100 papers and 20 datasets across 14 models, and the conclusion is uncomfortable for anyone billing clients by token: outside math and symbolic reasoning, CoT frequently adds cost without adding correctness. Knowing when that threshold applies is what separates disciplined prompt engineering from cargo-cult technique.

CoT Has a Documented Domain, and It Is Narrower Than the Hype Suggests

The Sprague et al. paper published at ICLR 2025 is the most rigorous quantitative treatment of this question to date. Their finding is specific: CoT reliably lifts accuracy on tasks with compositional or symbolic structure, things like multi-step arithmetic, formal logic, and algorithmic reasoning where the correct answer depends on an explicit sequence of intermediate states. In those cases, forcing the model to articulate the steps constrains the generation toward internally consistent outputs.

Outside that narrow domain, the picture collapses. On knowledge retrieval, commonsense reasoning, classification, and natural language generation tasks, CoT produces comparable or worse results than direct-answer prompting. The study is careful to note that much of the literature reporting CoT gains was concentrated precisely on the task types where it works, which inflated the perceived generality of the technique.

CoT as a method is sound. The scope condition attached to it is what the method's popularizers rarely state clearly. If you are running a model on multi-step financial calculations, code debugging, or formal constraint satisfaction, the token cost buys real accuracy. If you are classifying support tickets, writing product descriptions, or summarizing reviews, the research suggests you are likely paying for a confidence performance rather than a capability gain.

A parallel finding from a Wharton Generative AI Lab report in June 2025 adds a model-type dimension worth noting: for reasoning-native models like OpenAI's o-series or Claude with extended thinking enabled, bolting on explicit CoT instructions can inflate latency by 20-80% with marginal accuracy returns, because the model is already doing internal reasoning before surfacing a response. Stacking explicit CoT on top of a model that reasons natively is, my read of the evidence suggests, roughly equivalent to asking someone to narrate their thinking while they are already thinking.

The Customer Support Ticket Problem Illustrates the Faithfulness Gap

Here is a concrete, reproducible scenario that makes the Sprague findings tangible. Take a five-category support ticket classifier: billing, shipping, returns, technical issues, account access. Run it with direct-answer prompting and with full CoT prompting across a few hundred tickets. Accuracy between the two approaches tends to be negligible for this task class, but the CoT responses run 4-6x longer.

The more consequential issue is what Li, Cao, Chen et al. call the "faithfulness" problem in their 2025 paper on CoT effectiveness and faithfulness: the reasoning trace a model produces does not necessarily reflect the computational process that produced the answer. In classification tasks especially, a model will sometimes generate a confident, detailed chain of reasoning that describes the wrong decision path, then land on the correct label anyway. The stated reasoning and the actual output are partially decoupled.

For a product team automating ticket routing, this is mostly harmless. For any workflow where a human reviewer is auditing the logic, it is a significant problem. The reviewer reads a plausible-sounding rationale, assumes the model reasoned correctly, and trusts the output more than they should. The CoT output is functioning as a persuasion artifact rather than a reasoning trace. You are paying token costs for text that misleads your QA process while the model's actual classification behavior remains unchanged.

This is not a fringe edge case. My read of the faithfulness literature is that this decoupling is especially likely on tasks where the model has strong pattern-matching priors, which is exactly where direct prompting already works well and CoT adds nothing.

Truncated CoT Is Worse Than No CoT

One failure mode that rarely appears in CoT tutorials: truncating a reasoning trace mid-way through is not equivalent to removing CoT from the prompt. Research covered in a February 2026 paper on incomplete reasoning (arxiv.org paper 2602.14444) found that comparing no explicit reasoning against CoT truncated at 10% of its natural budget produces worse outcomes than either complete CoT or no CoT at all. A partial chain breaks the internal structure the model relies on when reasoning is actually load-bearing.

The practical implications are specific. If you are working within tight token budgets or context windows and cannot guarantee a complete reasoning trace, you are better off switching to direct-answer prompting than capping CoT output mid-thought. This matters for production environments where max_tokens settings interact with reasoning prompts, and for any system where CoT is enabled by default but responses get truncated by downstream processing.

A more elegant solution exists. The Focused Chain-of-Thought approach described in a November 2025 paper (arxiv.org, 2511.22176) separates information extraction from reasoning by feeding the model structured input, a table or tagged fields rather than a raw paragraph, before invoking any reasoning step. The reported result is a 2-3x reduction in generated tokens at equivalent accuracy, because structured inputs reduce filler reasoning steps and the model's tendency to restate context before engaging with the problem. If you are paying for reasoning, structuring your inputs before invoking it is the most actionable cost lever available right now.

The Decision Framework Before Enabling CoT

The practical question before adding CoT to any prompt is whether the task has the structural properties that make it load-bearing. A few diagnostic questions that narrow this down:

• Does solving the task require explicit intermediate states where an error at step three would cascade to a wrong final answer?
• Is the correct answer compositionally dependent on a sequence of sub-answers, or can pattern matching on the input directly produce the correct output?
• Are you using a reasoning-native model that already performs internal chain-of-thought before outputting a response?
• Do you have token budget constraints that could result in a truncated trace?

If the task is compositionally sequential (multi-step math, code that requires maintaining variable state, formal logic), CoT is earning its cost. If the task is classification, extraction, summarization, or generation where the model can pattern-match from the input to a correct output directly, the evidence points toward direct prompting with a well-structured input as the more efficient path. The structured input approach from the Focused CoT research gives you the discipline of explicit context without the token overhead of a full reasoning trace.

The default assumption in most prompt engineering advice runs the other way: add CoT unless you have a reason not to. Given the current research, that default should be inverted for any task class outside formal reasoning.

From My Experience

In catalog operations at a high-volume retailer, a significant share of daily AI tasks fall into the classification and extraction category: mapping supplier attributes to internal taxonomy fields, flagging inconsistent product descriptions, routing content issues to the right team. These tasks are structurally similar to the support ticket classification scenario above. My experience matches what the research describes qualitatively: responses with explicit step-by-step prompting tend to be longer and, if anything, slightly less consistent than direct-answer prompts with well-structured inputs. Switching to structured field inputs and dropping the CoT instruction produced noticeably cleaner outputs in catalog classification workflows, with faster turnaround and lower API costs, without any observable accuracy trade-off.

FAQ

Does "think step by step" still work in 2026?

On math, coding, and formal reasoning tasks, yes. The Sprague et al. ICLR 2025 meta-analysis confirms CoT reliably helps where tasks have compositional or symbolic structure. On classification, retrieval, and generation tasks, direct-answer prompting with structured inputs performs comparably at a fraction of the token cost.

Should I use CoT with Claude's extended thinking mode?

Extended thinking in Claude is a reasoning-native feature that already performs internal CoT before generating a response. Adding explicit CoT instructions on top of it is likely redundant and may inflate latency. Anthropic's documentation on extended thinking outlines when the feature is appropriate and how it interacts with prompt structure.

What happens if my CoT response gets cut off by a token limit?

Based on the 2026 "Broken Chains" paper, truncated CoT performs worse than either complete CoT or no CoT at all. If your token budget cannot guarantee a full reasoning trace, drop the CoT instruction and use structured direct prompting instead.

Is there a cheaper alternative to full CoT for complex tasks?

Focused Chain-of-Thought, described in a November 2025 arXiv paper (2511.22176), structures the input context before reasoning rather than asking the model to reason over raw text. It reports 2-3x token reductions at equivalent accuracy on reasoning tasks. Worth testing before committing to full CoT on any high-volume workflow.

The most expensive prompt engineering mistake is applying the right technique to the wrong task class at scale. CoT is genuinely useful in a specific domain, and that domain is smaller than most tutorials imply.