< Research

Why the Same Prompt Doesn’t Work the Same Way on Claude, GPT, and Gemini

I ran a total of 36 API calls across four different prompt categories using the same parameters. I sent the same six-rule prompt to Claude Sonnet 4.6, GPT-5.4, and Gemini 3.1 Pro.

The results were surprising.

GPT followed all six rules flawlessly.
Claude violated the same rules in all three attempts.
Gemini followed the rules but consumed 5x more tokens.

So, is this a bug?
Not really. It’s a direct consequence of each model’s architectural DNA.

In this article, I’ll use that data to answer a key question: Why does the same prompt behave differently across models? And more importantly, what are you losing if you write prompts without understanding this?

‍

1. WHY DO THEY BEHAVE DIFFERENTLY?

1.1 Instruction Hierarchy Differences

Modern large language models (LLMs) generally interact through system prompts, developer prompts, and user prompts.

The system prompt informs the model about its operating behavior and general rule set. Developer prompts contain specific instructions added by the application developer, such as output format or security policies. User prompts are direct requests and questions coming from the end user. In general, models accept system/developer messages as the highest-level directives and process user messages within this framework.

For example, while the system prompt says “Write in French” and the developer prompt says “Return responses in JSON format”, the user might ask “Tell me about today’s weather.” In this case, the model first applies the constraints coming from the system and developer messages. That is, even if the user says “respond only as plain text”, if the developer has said “return JSON”, the model maintains the JSON format.

‍

1.2 Tokenization Differences

System, developer and user messages are split into tokens before being sent to the model, and each model has its own tokenization schema. This difference produces much deeper consequences than one might think.

For example, GPT models generally use tiktoken-based subword (byte pair encoding) vocabularies such as cl100k. This allows commonly used word groups to be expressed as a single token with a vocabulary of 200,000 tokens. Claude's tokenizer is generally considered BPE-based, using byte-level segmentation rather than alphabet-based segmentation. It works with a smaller vocabulary of approximately 65,000 tokens. Google uses SentencePiece-based tokenization methods in the majority of its models, including the Gemma and Gemini series. The most critical difference of the SentencePiece algorithm from other traditional algorithms such as BPE or WordPiece is its approach to whitespace characters in text. It treats raw text as a continuous data stream and works language-independently.

In practice, token boundaries may vary according to Claude’s vocabulary (for example, “Merhaba” could be a single token). Nevertheless, these examples demonstrate the logic of BPE’s subword segmentation and the character-to-token ratio (approximately 3–4 characters per token).

So why does this difference matters?

One of the 4 prompt categories I mentioned at the beginning was Structured Output. In this test, despite giving Gemini the input “Return ONLY a valid JSON object, no markdown, no formatting”, I got the same result in all 3 attempts: indented, line-broken, pretty-print JSON. At the same time, while Claude used 182 tokens for the same prompt, Gemini used 858–882 tokens. Almost a 5x difference.

In other words, even when you tell Gemini to “return plain JSON”, the model’s architectural structure forces it toward pretty-print. However, this tendency of Gemini is not a bug but a direct consequence of its training pipeline and tokenization architecture.

At the root of this tendency lies the Reinforcement Learning from Human Feedback (RLHF) process used during the model’s alignment phase.

‍

1.3 Training & Alignment Differences

After being trained on massive text corpora, language models go through a specific alignment process to produce outputs that humans prefer. In this process, human evaluators are presented with different model responses given to the same prompt and are asked to score these responses according to criteria such as “usefulness”, “clarity” and “readability”.

While a compressed JSON string consisting of thousands of characters side by side is nothing more than a meaningless pile of characters for the human eye, a JSON table in “pretty-print” format is easily readable and verifiable. Over time, the model’s reward function weights this human tendency and the model’s internal probability distribution (logit distribution) shifts toward always using line breaks (\n), tabs (\t) and whitespace characters when producing data. As a result, Gemini internalizes producing “good-looking JSON” as the most probable and most rewarded behavior. This can only be partially overcome by adding schema constraints via the response_schema parameter or by writing very specific directives in the system prompt, but cannot be completely eliminated.

Compared to Gemini, Claude’s training uses the Constitutional AI approach. The model first produces uncontrolled outputs, then rewrites harmful content by critiquing itself. That is, instead of RLHF, RLAIF (Reinforcement Learning from AI Feedback) is applied: instead of human evaluations, a preference model based on a “penalty-response” model (the AI itself) is used. As a result, Claude internalizes ethical principles and instead of completely avoiding a harmful request, tries to explain why it cannot respond negatively.

To test what we discussed, I gave Claude 6 explicit rules (I will also explain these rules below). GPT managed to comply with all 6 rules in all 3 attempts. Claude, violated the same rules in all 3 attempts. According to TAU-bench (τ-bench), which tests the model’s adherence to assigned roles and decision-making according to given guidelines, the Claude model is placed in a specific role such as customer service and is restricted with many strict rule-based instructions. According to Anthropic’s data, Claude 3.5 Sonnet managed to solve 69.2% of cases in a relatively simple domain such as retail. However, when moved to a more challenging domain such as aviation, which contains much more complex cancellation, refund and routing instructions, the model’s success rate dropped significantly to 46.0%.

This situation shows that when the instructions or constraints Claude is guided by become more complex, it forgets the guidelines or violates the rules by misinterpreting them.

‍

2. TEST METHODOLOGY

To collect the data that forms the basis of this analysis, I tested Claude Sonnet 4.6, GPT-5.4 and Gemini 3.1 Pro with the same prompt set. I accessed each model through the OpenRouter API, with the goal of eliminating interface differences to isolate model behavior.

I proceeded through four prompt categories during the testing process: structured output, instruction-heavy, ambiguous and role-based.

With the structured output category, I measured how strictly models adhered to format instructions. In the instruction-heavy category, I tested whether compliance with all 6 simultaneous rules could be achieved. With the ambiguous prompt, I conducted a test of how models interpret ambiguity. Finally, with the role-based category, I measured persona adoption quality and technical accuracy.

Each category was repeated 3 times and a total of 36 API calls were made. Model settings were kept at temperature 0.7 and max_tokens1000.

As for the reasoning behind these settings, the natural default temperature setting at the API level for all three major ecosystems is 1.0. This is the balance point at which the model is tested and optimized by its producer. However, while Anthropic aims to preserve determinism by limiting the maximum temperature in the Claude API to 1.0, Google and OpenAI allow the temperature to be stretched up to 2.0 for areas requiring creativity.

A temperature value of 0.7 was used in these tests. The main purpose of this is to ensure that all models are compared fairly without hallucinating, but also without being trapped in mechanical language. However, it should be noted that this situation leads to a critical methodological bias: using 0.7 means intentionally moving each model away from its own natural default setting. Google’s official documentation explicitly warns about this situation:

“For all Gemini 3 models, we strongly recommend keeping the temperature parameter at its default value of 1.0. Changing the temperature (setting it below 1.0) may lead to unexpected behavior, such as looping or degraded performance, particularly in complex mathematical or reasoning tasks.”

The results should be evaluated in this context.

At the same time, max_tokenswas kept at 1000 across all tests. This was a critical isolation decision: if each model had produced output at its own natural length, we would not have been able to distinguish whether the token difference originated from model behavior or simply from output length. By setting a fixed upper limit, we were able to verify that token consumption differences genuinely stemmed from tokenization architecture and formatting preferences.

Finally, the purpose of 3 repetitions was to filter out random variation and to see whether the model was truly behaving deterministically or inconsistently.

‍

3. FINDINGS

I want to state upfront that this analysis measures behavioral differences across specific prompt categories, not the general capability of the three models. The results cannot be generalized to every use case, but the patterns are reproducible and can be explained by architectural reasoning.

3.1 Structured Output

The prompt for this category was straightforward:

Extract the following information from the text below and return ONLY a valid JSON object with these exact fields: 
"topic", "sentiment" (positive/negative/neutral), "key_points" (array of max 3 strings), "confidence_score" (float between 0 and 1). 
Do not include any explanation or markdown formatting.

Text: "The new transformer architecture significantly reduces inference time by 40% compared to previous models, though it requires more memory during training. 
Early adopters report mixed results in production environments.

What I wanted was for the model to extract specific fields from the given text and return only a valid JSON object. Claude produced clean, inline JSON using 182 tokens across all 3 attempts. The only variation was in the confidence_score, a small float variance from 0.85 to 0.82. GPT paraphrased the topic field differently in each attempt but the format remained consistent, staying within the 174–176 token range. Gemini, despite the “ONLY valid JSON” instruction, produced pretty-print formatted output in all 3 attempts and consumed 858–882 tokens.

‍

// Claude & GPT output
{"topic": "transformer architecture", "sentiment": "neutral", "key_points": [...], "confidence_score": 0.85}

// Gemini output
{
  "topic": "transformer architecture",
  "sentiment": "neutral",
  "key_points": [
    "Reduces inference time by 40%",
    ...
  ],
  "confidence_score": 0.95
}

The SentencePiece + RLHF combination I explained in the tokenization section becomes directly visible here. Gemini has internalized producing pretty-print as the highest probability token selection, and a format instruction is not enough to override this. The token difference is not an abstract number either: given that output tokens are billed at 5–6 times the cost of input tokens, Gemini’s cost for the same JSON content reaches approximately 25 times that of Claude.

What this means: All three models produced correct JSON structure. Claude delivered the most deterministic output with the fewest tokens, GPT paraphrased the content differently in each attempt but the format remained consistent. The most important observation was this: if you want compressed JSON from Gemini in production, a plain instruction is not enough and it also consumes significantly more tokens.

3.2 Ambiguous

In this category, I intentionally used an ambiguous prompt:

Summarize this for a technical audience.

Content: "Our system processes requests, validates inputs, and returns structured responses. 
It handles errors gracefully and logs all activity for monitoring purposes."

No length, no format, no level of detail was specified. The goal was to see how each model interprets ambiguity.

GPT made the most minimal interpretation: 78–80 tokens, a single paragraph, zero assumptions. It summarized the content as-is and added no extra commentary. Claude interpreted the ambiguity as a “structural breakdown”, adding markdown headers and bullet points to break the system down into its components. Even though the prompt did not explicitly ask for this, Claude’s Constitutional AI training pushes it to maximize the “being helpful” criterion. Gemini exhibited the same behavior across all 3 attempts: instead of a single summary, it offered 3 different summary options with labels such as “Ultra-concise”, “standard” and “feature-focused”, consuming 800–950 tokens.

This difference is not coincidental. GPT’s developer role hierarchy draws a clear boundary, which is why the model does not go beyond the instruction. Gemini’s RLHF process, trained to maximize the “usefulness” criterion, interprets ambiguity as “offer more options”. Both are the logical outcome of their own architecture.

As an additional factor, the biggest contributor to Gemini’s excess token consumption is the Thinking Tokens mechanism that activates in models like Gemini 3.1 Pro. Standard LLMs begin predicting words directly when a question arrives.
However, advanced reasoning models initiate a massive internal thinking process before reflecting the final answer on screen. When the model receives a user’s question, it opens a hidden workspace where it examines the context of the question, evaluates possible assumptions, builds logical chains (Chain-of-Thought), self-corrects and reaches a final decision. Thousands of tokens are generated throughout this process. Once the final decision is reached, only a summarized final answer of perhaps 50 words is printed on screen. While the user sees only 50 words on screen, 898 output tokens may be billed in the usage_metadata statistics behind the API response.

Developers must use the thinking_level parameter in API requests to control this inflation. Leaving the parameter at HIGH causes 20,000 tokens to be wasted even on simple tasks, while pulling it down to LOW reduces token consumption by 50 to 80 percent.

What this means: Based on the ambiguous prompt output, GPT interprets minimally while Gemini interprets maximally. The model can be chosen based on which behavior is desired, but it should be known that changing this behavior at the prompt level is difficult for both.

3.3 Role-Based

In the final category, I gave the models a persona. The prompt also contained SQL injection, with the goal of having the model review the code and identify the issues.

You are a senior backend engineer with 10 years of experience who is known for being extremely opinionated and direct. 
You do not sugarcoat feedback.

Review the following code and tell me what's wrong with it:
def get_user(id):
    db = connect_to_database()
    user = db.query("SELECT * FROM users WHERE id = " + str(id))
    return user

The things I paid attention to here were:

  • How well did it adopt the role?
  • Did the tone genuinely change?
  • Did it catch a critical issue like SQL injection?
  • Is there a “personality” difference between models?

All three models detected the SQL injection, but persona quality and technical depth varied from model to model. Claude delivered the most consistent technical review: it opened with a “Dumpster Fire” header, referenced OWASP, and the level of detail increased with each attempt. GPT adopted the persona but the tone remained more measured, using expressions like “bad in all predictable ways”. Gemini adopted the most dramatic persona and gave the following response: “If you submitted this in a pull request, I would close it immediately, decline to merge, and we’d be having a very uncomfortable 1-on-1 about backend fundamentals.” What is notable here is that despite GPT giving the longest response, it used a maximum of 700 total tokens, while Gemini used 1085 total tokens across all three attempts for such a short response.

What this means: In terms of persona adoption quality, Claude gives the most consistent, Gemini the most dramatic, and GPT the most measured response. But all three detected the critical security vulnerability, which means that in role-based prompts, technical accuracy is independent of model choice.

3.4 Instruction-Heavy

This category contains the most striking finding of the article. The prompt I gave to the models was as follows:

You are a technical documentation writer. Follow ALL of the rules below strictly:
1. Use only simple present tense
2. Never use the word "utilize" — use "use" instead
3. Every sentence must be under 15 words
4. Do not use bullet points or numbered lists in your output
5. Start every paragraph with a transition word
6. Output must be exactly 3 paragraphs

Task: Explain what a REST API is.

GPT complied with all 6 rules in all 3 attempts. Gemini also largely managed to comply with the rules but used exactly 1092 tokens in every attempt. Claude, on the other hand, violated the same rule in all 3 attempts: it used a markdown bold header (Understanding REST APIs). There were no bullet points, but the plain text format was broken.

In the OSWorld benchmark, the industry standard for testing a model’s instruction-following ability in a computer operating system environment, Claude 3.5 Sonnet achieved a success rate of only 14.9% when limited to 15 interaction steps. In a task where human performance is 72.36%, the 14.9% rate is Anthropic’s own admission of how inadequate the model is at end-to-end instruction execution. The main reason for the failure is the model losing context while following instructions step by step or making wrong decisions on the interface. Anthropic states that when the interaction limit is increased from 15 to 50 steps in the same test, the success rate rises to 22%.

What this data means is: the model cannot directly apply the first instruction given to it, but tries to reach the result by attempting roundabout paths as it makes errors within its own limits. Rather than pure instruction-following perfection, what we see is a trial-and-error process with high error rates.

What this means: The general consensus in the market is that Claude is strong at instruction following, and broadly speaking this is true. But “instruction following” and “complying with multiple simultaneous formatting rules” are not the same thing. Choosing Claude without knowing this distinction means leaving the door open to surprises.

‍

4. WHAT THE DATA ACTUALLY TELLS YOU ABOUT PROMPTING

Looking at the test results, one thing becomes clear: model-agnostic prompting does not exist. Every model is built on a different architectural philosophy, and that philosophy shapes how it responds to every instruction you give. Based on this, here are 5 directives I can offer:

  1. Give Gemini extra directives for JSON output:
    Saying “ONLY valid JSON” is not enough. You need to add “no newlines, no indentation, no whitespace” to the system prompt or use the response_schema parameter. Otherwise, the SentencePiece + RLHF combination forces the output into pretty-print every time.
  2. Prefer GPT when there are multiple simultaneous rules:
    Claude’s success was inversely proportional to the number of rules. In my 6-rule test, GPT achieved 3/3 full compliance while Claude violated the same rule 3/3. Anthropic’s own τ-bench data confirms this: as the rule set becomes more complex, Claude’s success rate drops from 69% to 46%.
  3. Model choice plays a critical role when writing ambiguous prompts: GPT interprets ambiguity minimally, doing exactly what is asked without adding extra commentary. Gemini, on the other hand, interprets ambiguity as “offer more options”. In my tests, GPT used 80 tokens and Gemini used 900 tokens for the same input. Changing this behavior at the prompt level is quite difficult.
  4. Test at the default temperature first, then optimize:
    0.7 is a common standard in comparative tests, but every model’s native default is 1.0. It is important to test each model at its own default temperature before moving to production. Dropping below 1.0 especially for Gemini can lead to unexpected behaviors in complex tasks.
  5. Use token cost as a factor in model selection:
    Gemini can consume 5 times more tokens than Claude for the same content. Given that output tokens are billed at 5–6 times the cost of input tokens, this difference significantly impacts production costs. If Gemini is going to be used, the thinking_level parameter should be checked. Leaving it at HIGH can waste 20K+ extra tokens even on simple tasks.

The common denominator of all these rules is this: model-agnostic prompting does not exist. That is why writing the prompt according to the model is the right approach.

‍

5. RESULT

36 API calls, 4 prompt categories, 3 models. The result:

GPT achieved the highest compliance with rules by strictly applying its instruction hierarchy. Claude systematically failed at complex formatting rules as a side effect of its ethics-focused Constitutional AI training. Gemini ignored format instructions through the combined effect of SentencePiece tokenization and RLHF, consuming 5 times more tokens for the same content. None of these are bugs, all of them are the outcomes of deliberate engineering decisions.

This finding raises a question that goes beyond prompt engineering: which model you choose is just as determining as how you write your prompt. Understanding a model’s architecture before moving a system to production is no longer optional.

So which model do you use for which task, and when making that decision, do you start from architecture, benchmarks, or habit?

‍

RESOURCES

‍

Author:

‍