Recurrent Neural Language Models as Probabilistic Finite-state Automata
Published in EMNLP 2023, 2023
We study what classes of such probability distributions RNN LMs can represent and show that simple RNNs are equivalent to a subclass of probabilistic finite-state automata, and can thus model a strict subset of probability distributions expressible by finite-state models.