The Extended Lexicon: Language Processing as Lexical Description

In this paper we introduce an approach to lexical description which is sufficiently powerful to support language processing tasks such as part-of-speech tagging or sentence recognition, traditionally considered the province of external algorithmic components. We show how this approach can be implemented in the lexical description language, DATR, and provide examples of modelling extended lexical phenomena. We argue that applying a modelling approach originally designed for lexicons to a wider range of language phenomena brings a new perspective to the relationship between theory-based and empirically-based approaches to language processing. 1 The Extended Lexicon A lexicon is essentially a structured description of a set of lexical entries. One of the first tasks when developing a lexicon is to decide what the lexical entries are. This task has two dimensions: what kind of linguistic object does a lexical entry describe, and what does it say about it. So for example, one might decide to produce a lexicon which describes individual word instances, and provides the orthographic form and part-of-speech tag for each form. It is the first of these dimensions that is most relevant to the idea of the Extended Lexicon. Conventionally, there are two main candidates for the type of linguistic object described by a lexicon: word forms (such as sings, singing, sang1), corresponding to actual words in a text and lexemes (such as SING, WALK, MAN), describing abstract words, from which word forms are somehow derived. Choosing between these two candidates Typographical conventions for object types: ABSTRACT, LEXEME, wordform, instance, code. Figure 1: A simple inheritance-based lexicon might be a matter of theoretical disposition, or a practical consideration of how the lexicon is populated or used. In the Extended Lexicon, we introduce a third kind of linguistic object, called word instances (or just instances), consisting of word forms as they occur in strings (sequences of words, typically sentences). For example, a string such as the cats sat on the mat contains two distinct instances of the word the. the cats slept contains further (distinct) instances of the and cats. However the instances in a repetition of the cats sat on the mat are the same as those in the original (because instances are defined relative to strings, that is, string types not string tokens). So in an extended lexicon, the lexical entries are word instances, and the lexicon itself is a structured description of a set of word instances. In order to explore this notion in more detail, it is helpful to introduce a more specific notion of a ‘structured description’. We shall use an inheritancebased lexicon, in which there are internal abstract ‘nodes’ representing information that is shared by several lexical entries and inherited by them. Figure 1 shows the structure of a simple inheritancebased lexicon with some abstract high-level structure (CATEGORY, VERB, NOUN), then a layer of lexemes (WALK, TALK, HOUSE, BANK), and below that a layer of word forms (walks, walking,