Appears in collection : Combinatorics and Arithmetic for Physics - 2024

Large Language Models are neural networks which are trained to produce a probability distribution on the possible next words to given texts in a corpus, in such a way that the most likely word predicted, is the actual word in the training text. We will explain what is the mathematical structure defined by such conditional probability distributions of text extensions. Changing the viewpoint from probabilities to -log probabilities, we observe that the data of text extensions are encoded in a directed (non-symmetric) metric structure defined on the space of texts ${\mathcal L}$. We then consider the space $P ({\mathcal L})$, of non-expansive functions on ${\mathcal L}$ which turns out to be a directed metric, alcoved polytope, in which ${\mathcal L}$ is isometrically embedded as generators of certain special extremal rays. Each such generator encodes extensions of a text along with the corresponding probabilities. Moreover $P ({\mathcal L})$ is (min, +) (i.e. tropically) generated by the text extremal rays. $P ({\mathcal L})$ encodes semantic information about the language. We study this space and in particular explain a duality theorem relating the space generated by text extensions and that generated by text restrictions. The metric space ${\mathcal L}$ can equivalently be considered as an enriched category and then the embedding into $P ({\mathcal L})$ is the Yoneda embedding into its category of presheaves. In fact all constructions have categorical meaning (in particular generalizing the familiar view of language as a monoid or as a poset with the subtext order). This is joint work with Stéphane Gaubert.