Is English Segmentation Trivial?

1. What is the problem?

2. English Morphology

Languages differ in the processes by which they form new words. An important unit of word formation is the morpheme. A morpheme is a minimal unit of meaning -- a distinctive linguistic unit of relatively stable meaning that cannot be divided into smaller meaningful parts.

First, let us take a quick look at the principal morphological processes in English:

1. Compounding: combining two or more base forms, e.g. headwaiter
2. Inflection: adding endings to indicate grammatical functions, e.g. walk + ed gives walked
3. Affixation: adding affixes to a base form, e.g. great + ness gives greatness)
4. Conversion: a shift of one word class to another, e.g. the noun chemical is derived from the adjective chemical
5. Blending: merging two words, like breakfast + lunch to get brunch)
6. Clipping: truncating a word into one syllable, e.g. laboratory into lab)
7. Acronyms: combining initial letters, e.g. UNESCO)

3. Principal Morphological Processes

Of particular importance are the three morphological processes described below:

1. Compounding consists of combining two or more words having their own lexical meaning (having a substantial meaning of their own) to produce a new unit that functions as a single word. Traditionally, a compound word is considered to be a combination of two or more free words, such as headwaiter, which consists of head and waiter, though in practice the elements are not necessarily free.
   
   
2. Affixation (also called derivation) refers to creating a new word by adding to a base form an affix (prefix, infix or suffix) that expresses grammatical meaning but has no lexical meaning. For example, in English the adverb-forming suffix ly is attached to the base form great to form the adverb greatly. It is important to note that units formed through affixation are distinct words in their own right, not merely variants of the word from which they are derived.
   
   
3. Inflection consists of adding word endings or modifying the form of a word in order to indicate various grammatical functions, such as tense (called conjugation,) or number and case (called declension). The resulting word is another form of the original word, not a new word in itself. For example, worked is the past tense of work and books is the plural form of book.

4. Some Issues in English Segmentation

Here are a few reasons why breaking an English text stream into meaningful semantic units is difficult. It of utmost importance to keep in mind that the traditional notion of a space-delimited "word" is often not a useful unit.

For a search engine to be effective, the most important units that must be identified are lexemes (also called lexical units), which can be roughly defined as basic semantic units of vocabulary. Some lexemes, like run, can have several variants (inflected forms) like ran and running. Others, which we will refer to as multi-word lexemes, can consist of several traditional words, e.g. information retrieval.

1. Space-Delimited Compound Words
   

   "Take off your jacket!"

   This is an instance of a verbal compound, a single lexeme, take off. An intelligent search engine should identify it as a single unit, and not break it up. The status of take off as a lexeme in English is equivalent to single-word lexemes like remove and take.

2. Compound Verbs with Inserted Elements
   

   "Take your jacket off!"
   "Send that letter off before he comes"

   Here, there are words inserted between the verb and off, so that even an intelligent segmenter that can do (1) will fail to identify the lexemes take off and send off.

3. Stemming

   An intelligent segmenter must be able to identify inflected forms like went, took and took off and determine their stem or root form. This is referred to as stemming or sometimes deinflecting.

   Combining the above three problems, a sophisticated segmenter should be able to identify take off in the sentence below:

   "He took his shirt off yesterday"

4. Hyphenated/Solid/Open

   Take the following as an example:

   1. word processor
   2. word-processor
   3. wordprocessor

   A compound word often first appears in English as free forms in juxtaposition, like (1) above. As it becomes established ("petrified"), forms such as (1) are considered to be noun phrases and begin to appear in glossaries and terminology dictionaries. As they further evolve they become "real" words (become lexicalized) and they make their way into ordinary dictionaries and the layperson's vocabulary.

   Some compounds go through a stage of hyphenation, like (2), and then eventually may become solid, like (3). Some lexemes, like school bus and high school, always remain open.

   What is very important to understand is that the degree of solidity often has nothing to do with the status of a string as a lexeme. School bus is just as legitimate a lexeme as is headwaiter or word-processor. The presence or absence of spaces or hyphens, that is, the orthography, does not determine the lexemic status of a string.

5. Proper Nouns

   Proper Nouns, like AT&T and New York, must be treated as single units. Republic of China is just as legitimate a proper noun as China is, and there is not the slightest justification for treating them differently.