BLEU has been the default automated metric for judging machine translation quality for two decades - it's fast, cheap, and gives you a single number. It was also never built to tell you whether a creative rewrite is good, and the research saying so predates most of today's LLM-based translation tools by more than fifteen years.
What BLEU actually measures
BLEU scores a candidate translation by how many of its word sequences overlap with one or more human reference translations. Callison-Burch, Osborne, and Koehn's 2006 EACL paper, Re-evaluating the Role of BLEU in Machine Translation Research, is the paper most often cited for what that leaves out: it argues, with direct evidence, that an improved BLEU score is "neither necessary nor sufficient" for an actual improvement in translation quality as judged by a human. A translation that rewords a sentence correctly, but differently from the reference, can score worse than a translation that's closer to the reference wording but reads more awkwardly.
That's a fundamental mismatch for creative adaptation specifically: the entire point of a creative rewrite is to depart from the most literal wording on purpose. A metric that rewards closeness to one reference string will always penalize the thing a good adaptation is trying to do.
Using a model to judge a model
One alternative that's gotten real research attention is "LLM-as-a-judge" - having a language model score or rank other outputs against a rubric, instead of a fixed formula like BLEU. Zheng et al.'s 2023 NeurIPS paper, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, found that a strong LLM judge (GPT-4, in their study) can match both controlled and crowdsourced human preference well - the paper reports over 80% agreement, comparable to the agreement level between two human raters.
The same paper is just as clear about the catch: LLM judges carry real, documented biases - favoring longer answers regardless of quality, favoring whichever position an answer appears in, and favoring a model's own outputs over a competitor's. A judge stage that ignores those failure modes isn't actually solving BLEU's problem, just moving it somewhere less visible.
Why a Judge needs an anchor
Castia's Judge stage is built around that specific caveat: it doesn't rank rewrites purely on open-ended preference, which is where position and verbosity bias creep in unchecked - it scores each rewrite against a literal anchor translation produced by a separate Translator stage first. That anchor is the fixed reference point BLEU never had a creative-translation-appropriate version of, and it's what lets the Judge state a specific reason a rewrite earns its departure from a literal reading, rather than just a preference score. See the full pipeline on how it works.