Skip to content

README.md

Latest commit

 

History

History
81 lines (54 loc) · 4.3 KB

README.md

File metadata and controls

81 lines (54 loc) · 4.3 KB
Raw

How Whitespaces Affect LLM Performance in Arithmetic Calculations?

We conducted an experiment to determine whether whitespace modifications in arithmetic expressions influence how LLMs generate answers.

Dataset

We used the arithmetic_1dc subset from EleutherAI/arithmetic, which contains pairs like:

context: Question: What is (9 + 8) * 2? Answer:
completion: 34

The complexity level of these problems is appropriate for modern ~8B parameter LLMs. Short prompts also avoid context window limitations, allowing us to isolate the effect of minor prompt modifications. This dataset is also included in the lm-evaluation-harness benchmark.

The dataset is loaded via datasets library from huggingface hub.

Code

To run the analysis:

  1. Open arithmetic_robustness.py

  2. Set the model_name variable to your target model

  3. Execute the script

Methodology

Whitespace Modifications

We added or removed spaces (ASCII 0x20) in positions that don't alter the mathematical meaning:

  • Never inside words or numbers
  • Up to 3 consecutive spaces could be added
  • Spaces before brackets and operators could be removed

Example modification:

Original: What is (9 + 8) * 2?
Modified: What is ( 9+8) * 2 ?

Experimental Setup

  • Prompt Format: Fixed across all tested models (no model-specific optimization)
  • Generation Paradigm: Greedy decoding with answer extraction via <start>number</start> pattern
  • Chat Template: Standard template applied via tokenizer's apply_chat_template with enable_thinking=False
  • Statistical Test: McNemar's test (p-value<0.05 threshold) for significance testing

Note: McNemar's test becomes unreliable when failure rates are very low (e.g., for Qwen3-32B's near-perfect accuracy).

Results

Model Instruction-following Acc. (↑) Population Original Acc. (↑) Distorted Acc. (↑) Significant Difference
Qwen/Qwen3-14B 1.0 2000 0.946 0.964 True
Qwen/Qwen3-8B 1.0 2000 0.897 0.939 True
t-tech/T-lite-it-1.0 1.0 2000 0.762 0.676 True
ai-sage/GigaChat-20B-A3B-instruct 0.99625 1985 0.498 0.507 False
mistralai/Mistral-7B-Instruct-v0.3 0.93275 1749 0.207 0.189 True
yandex/YandexGPT-5-Lite-8B-instruct 0.981 1926 0.201 0.088 True
HuggingFaceTB/SmolLM2-1.7B-Instruct 0.972 1890 0.0 0.0 True

Column Descriptions:

  • Instruction-following acc.: Ratio of responses containing the <start>number</start> pattern; higher is better
  • Population: Valid response pairs (both original/modified prompts contained extractable answers)
  • Original Acc.: Accuracy with unmodified prompts; higher is better
  • Distorted Acc.: Accuracy with whitespace-modified prompts; higher is better
  • Significant Difference: McNemar's test result (p<0.05)

Key Findings

  1. Whitespaces significantly affect LLM performance in arithmetic calculations
  2. Qwen3 models show improved accuracy with added whitespaces, while other models degrade
  3. GigaChat-20B-A3B-instruct performance is not affected by whitespace noise (difference of mean accuracies is not significant)
  4. Instruction-following capability:
    • Qwen3 achieves perfect compliance
    • Other models occasionally deviate from response formatting requirements
  5. Russian-language LLMs underperform likely due to English prompt mismatch (we maintained consistency by not using Russian prompts)