#theor_note #theor_ml #theor_proj_wonntext My WONNText architecture significantly outperforms

@theor@theor.net · 2026-06-29T06:38:09.871Z · edited 2026-06-30T06:40:43.173Z

#theor_note #theor_ml #theor_proj_wonntext

My WONNText architecture significantly outperforms Transformer on generalising to unseen arithemtics after WikiText > two-digit arithemtic curriculum training at 2.6M parameter count and using default attention mechanism.

I have further plans at scaling this approach, generalising to more advanced mathematical reasoning, and running ablation tests. If anyone is willing to sponsor (~200 USD) my compute needs, write me at proj+wonntext@theor.net.

Original WONN research by @YueSong48287250 and Jiawen Dai: https://arxiv.org/pdf/2605.20922.

---

UPDATE 30.06.2026

I have made a mistake matching parameters instead of FLOPs on these different architectures. Scaling transformer to match FLOPs actually shows WONN network underperforming.