Publications

2026

OLMo Hybrid: From Theory to Practice and Back

William Merrill, Yanhong Li, Tyler Romero, Anej Svete, Caia Costello, Pradeep Dasigi, Dirk Groeneveld, David Heineman, Bailey Kuehl, Nathan Lambert, Chuan Li, Kyle Lo, Saumya Malik, DJ Matusz, Benjamin Minixhofer, Jacob Morrison, Luca Soldaini, Finbarr Timbers, Pete Walsh, Noah A. Smith, Hannaneh Hajishirzi, Ashish Sabharwal · arXiv · [arXiv]

Probability Distributions Computed by Hard-Attention Transformers

Andy Yang, Anej Svete, Jiaoda Li, Anthony Widjaja Lin, Jonathan Rawski, Ryan Cotterell, David Chiang · ICLR 2026 · [arXiv]

On the Reasoning Abilities of Masked Diffusion Language Models ★ Oral at ICLR 2026

Anej Svete, Ashish Sabharwal · ICLR 2026 · [arXiv]

Context-Free Recognition with Transformers

Selim Jerad, Anej Svete, Sophie Hao, Ryan Cotterell, William Merrill · arXiv · [arXiv]

2025

The Exact Expressive Power of Fixed-Precision Looped Padded Transformers

Anej Svete, William Merrill, Ashish Sabharwal ·

The Transformer Cookbook

Andy Yang, Christopher Watson, Anton Xue, Satwik Bhattamishra, Jose Llarena, William Merrill, Emile Dos Santos Ferreira, Anej Svete, David Chiang · TMLR · [arXiv]

Information Locality as an Inductive Bias for Neural Language Models ★ Oral at ACL 2025 · Panel: Role of Linguistics in LLM Research

Taiga Someya, Anej Svete, Brian DuSell, Timothy J. O'Donnell, Mario Giulianelli, Ryan Cotterell · ACL 2025 · [arXiv]

Unique Hard Attention: A Tale of Two Sides ★ Oral at ACL 2025

Selim Jerad, Anej Svete, Jiaoda Li, Ryan Cotterell · ACL 2025 · [arXiv]

Gumbel Counterfactual Generation From Language Models

Shauli Ravfogel, Anej Svete, Vésteinn Snæbjarnarson, Ryan Cotterell · ICLR 2025 · [arXiv]

Training Neural Networks as Recognizers of Formal Languages

Alexandra Butoi, Ghazal Khalighinejad, Anej Svete, Josef Valvoda, Ryan Cotterell, Brian DuSell · ICLR 2025 · [arXiv]

2024

A Probability-Quality Trade-off in Aligned Language Models and its Relation to Sampling Adaptors

Naaman Tan, Josef Valvoda, Tianyu Liu, Anej Svete, Yanxia Qin, Min-Yen Kan, Ryan Cotterell · EMNLP 2024 · [arXiv]

Can Transformers Learn n-gram Language Models?

Anej Svete, Nadav Borenstein, Mike Zhou, Isabelle Augenstein, Ryan Cotterell · ACL 2024 · [arXiv]

On Efficiently Representing Regular Languages as RNNs

Anej Svete, Robin Shing Moon Chan, Ryan Cotterell · ACL 2024 Findings · [arXiv]

An L* Algorithm for Deterministic Weighted Regular Languages

Clemente Pasti, Talu Karagöz, Franz Nowak, Anej Svete, Reda Boumasmoud, Ryan Cotterell · ACL 2024 · [arXiv]

On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning

Franz Nowak, Anej Svete, Alexandra Butoi, Ryan Cotterell · ACL 2024 · [arXiv]

On Affine Homotopy between Language Encoders

Robin SM Chan, Reda Boumasmoud, Anej Svete, Yuxin Ren, Qipeng Guo, Zhijing Jin, Shauli Ravfogel, Mrinmaya Sachan, Bernhard Schölkopf, Mennatallah El-Assady, Ryan Cotterell · NeurIPS 2024 · [arXiv]

What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages ★ Oral at ACL 2024

Nadav Borenstein, Anej Svete, Robin Chan, Josef Valvoda, Franz Nowak, Isabelle Augenstein, Eleanor Chodroff, Ryan Cotterell · ACL 2024 · [arXiv]

Transformers Can Represent n-gram Language Models

Anej Svete, Ryan Cotterell · NAACL 2024

Lower Bounds on the Expressivity of Recurrent Neural Language Models

Anej Svete, Franz Nowak, Anisha Sahabdeen, Ryan Cotterell · NAACL 2024

The Role of n-gram Smoothing in the Age of Neural Networks

Luca Malagutti, Andrius Buinovskij, Anej Svete, Clara Meister, Afra Amini, Ryan Cotterell · NAACL 2024

2023

Recurrent Neural Language Models as Probabilistic Finite-state Automata

Anej Svete, Ryan Cotterell · EMNLP 2023

On the Representational Capacity of Recurrent Neural Language Models

Franz Nowak, Anej Svete, Li Du, Ryan Cotterell · EMNLP 2023

Formal Aspects of Language Modeling

Ryan Cotterell, Anej Svete, Clara Meister, Tianyu Liu, Li Du · arXiv · [arXiv]

Efficiently Representing Finite-state Automata With Recurrent Neural Networks

Anej Svete, Ryan Cotterell · arXiv · [arXiv]

A Geometric Notion of Causal Probing

Clément Guerner, Anej Svete, Tianyu Liu, Alexander Warstadt, Ryan Cotterell · arXiv · [arXiv]

2022

Algorithms for Acyclic Weighted Finite-State Automata with Failure Arcs

Anej Svete, Benjamin Dayan, Tim Vieira, Ryan Cotterell, Jason Eisner · EMNLP 2022