This September 2025 paper introduces Inverse IFEval, a novel benchmark designed to evaluate Large Language Models (LLMs) for their Counter-intuitive Ability. This refers to an LLM's capacity to override its ingrained training patterns and comply with instructions that conflict with conventional norms or standardized formats. The benchmark includes eight distinct categories of such challenging instructions, like "Code without Comments" or "Deliberately Incorrect Answers," to expose the cognitive inertia and overfitting that current LLMs exhibit. The study underscores the need for future LLM development to prioritize adaptability in unconventional contexts beyond mere fluency and factual accuracy. Findings demonstrate that while some models perform well on traditional instruction-following tasks, their performance significantly declines when faced with these inverse instructions.
Source:
https://arxiv.org/pdf/2509.04292
