Art of FACELESS — Independent R&D Protocol
AoF-RD-2026-088: Pathological Confidence, Out-of-Distribution Density Collapse, and NMT Artefacts in Discriminative AI Classifiers
Lead Investigator: Retired PhD Pharmacologist (AoF Independent R&D)
Methodology: Cross-Lingual Metric Audit & Epistemic Calibration Assessment
Date: November 2026
1. Abstract
Commercial transformer-based synthetic text classifiers (specifically Pangram architectures) report near-zero false-positive rates (FPR < 10⁻⁴) and robust cross-lingual generalisation. Empirical evaluation of a human-authored control text in literary Welsh (Cymraeg) and its subsequent deterministic Neural Machine Translation (NMT) via Google Translate into English yielded degenerate classification states:
P(Synthetic | Text_NMT_English) = 1.000 (100% AI)
This paper documents how commercial detection pipelines suffer from severe epistemic overconfidence induced by:
- Out-of-Distribution (OOD) density collapse in minority/regional languages;
- Structural metric compression under poetic verse constraints;
- Low-entropy path regularisation inherent to sequence-to-sequence beam-search decoding in NMT.
2. Mathematical Architecture of the Detector
Pangram and comparable discriminative classifiers deploy fine-tuned bidirectional transformer backbones (e.g., RoBERTa-large derivations) coupled to a pooling layer and a multi-layer perceptron (MLP) classification head.
For an input token sequence x = [t₁, t₂, ..., t_N]:
h = Encoder(x)
z = W₂ · GeLU(W₁ · h + b₁) + b₂
ŷ = σ(z) = 1 / (1 + e⁻ᶻ)
Where:
- h is the pooled semantic embedding representation.
- z represents the unbounded scalar logit output for the synthetic class.
- ŷ ∈ [0, 1] represents the model's raw posterior estimate: P(Synthetic | x).
2.1 The Asymptotic Saturation Fallacy (The "100%" Artefact)
In binary cross-entropy loss, assigning an empirical posterior probability of exactly 1.000 requires infinite loss if the ground truth is negative (human-authored):
lim [ŷ → 1.0] Loss_BCE(y = 0, ŷ) = -lim [ŷ → 1.0] log(1 - ŷ) = +∞
In rigorous analytical assays, absolute certainty is unachievable due to intrinsic measurement variance and Bayesian uncertainty bounds. Outputting 100% indicates that the logit z exceeded an internal clipping threshold (z ≥ z_clip, where σ(z) ≈ 1 - ε, with ε < 10⁻⁶). The interface rounds this extreme asymptotic tail to an integer (100%), replacing calibrated Bayesian uncertainty with unearned marketing authority.
3. Mechanisms of Failure
3.1 Out-of-Distribution (OOD) Density Collapse in Welsh
Let D_train represent the multi-lingual training distribution. Standard training corpora exhibit vast linguistic asymmetry:
Ratio(Tokens_English / Tokens_Welsh) > 1,000 : 1
Discriminative classifiers do not examine semantic intent; they estimate whether the input x resides within the expected density manifold of human prose. In sparse, low-resource feature spaces:
∇_x log p(x | Human) → undefined / erratic
Because generative LLMs have limited Welsh pre-training data, synthetic Welsh text is restricted to highly regular, textbook grammatical morphology and canonical mutation rules. When a fluent human author writes structured, disciplined Welsh verse, the token transition probabilities mirror this low-variance profile. The detector's severely under-parameterised manifold for Welsh mistakenly maps genuine human stylistic discipline directly into the synthetic class boundary.
3.2 Verse-Induced Metric Distortion
Classifiers rely on two core statistical proxies across running text:
- Perplexity (PPL): The geometric mean of inverse conditional token likelihoods:
PPL(x) = exp( - (1/N) · Σ log P(t_i | t₁, ..., t_{i-1}) ) - Burstiness: The standard deviation of sentence-level perplexity divided by the mean perplexity:
Burstiness = σ(PPL) / μ(PPL)
Poetry breaks these assumptions:
- Truncated Context: Short poetic lineation fractures standard 512-token sliding self-attention contexts.
- Metric / Rhyme Regularisation: Poetic cadences create cyclical, low-entropy token transitions rather than the wide, erratic variance typical of human prose.
The detector reads formal poetic restraint not as human craft, but as mechanical generation.
3.3 Neural Machine Translation (NMT) as an Entropy Funnel
Subjecting the Welsh original to Google Translate generated an English output that also scored 100% synthetic. This is an unavoidable mathematical outcome of NMT sequence decoding.
Google Translate uses an encoder-decoder transformer operating under Beam Search decoding with length penalisation:
ŷ_NMT = argmax_y Σ log P(y_t | y_<t, x_Welsh)
Human literary translation introduces idiosyncratic, high-perplexity lexical choices. In contrast, Beam Search actively prunes low-probability paths, optimising exclusively for:
- High collocation frequency (standardised multi-word expressions);
- Minimum conditional entropy paths;
- Canonical syntactic structures.
[Human Thought in Welsh]
│
▼ (High structural density / low training corpus representation)
[Google NMT Engine]
│
▼ (Beam-search pruning strips lexical entropy and personal voice)
[Smoothed English Output]
│
▼ (Near-zero burstiness / uniform low-perplexity token chains)
[Pangram Detector]
│
▼
Result: 100% AI Flag (Model identifies the NMT conduit, not provenance)
4. Diagnostic Conclusions
- Epistemic Overconfidence: Publishing a binary 100% certainty metric on out-of-distribution, minority-language, or short-form texts violates standard diagnostic error budgeting.
- Translation Invalidation: Subjecting translated literature to statistical detectors penalises international and minority-language writers. The tool cannot distinguish between an LLM hallucinating text and an NMT pipeline translating human thought.
- Protocol Verdict: Commercial detection scores on regional languages and translated verse do not reflect authorship provenance; they merely register the presence of non-English syntactic structures and algorithmic sequence-smoothing.
Empirical evaluations demonstrate that commercial AI text detectors consistently produce false positives when applied to constrained linguistic structures, exhibiting significant error rates against non-native writing and non-standard prose (Liang et al., 2023; Weber-Wulff et al., 2023).
References
Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023). GPT detectors are biased against non-native English writers. Patterns, 4(7), 100779. https://doi.org/10.1016/j.patter.2023.100779 Cited by: 1465
Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19(1). https://doi.org/10.1007/s40979-023-00146-z Cited by: 1358