Words and Lemmatization

2024-03-20

Tagging lexical words in the text and/or linking to lemmata for purposes of indexing or search.

Explicit markup of words (tokenization) and identification of their dictionary headwords (lemmatization) are both optional. Many projects simply leave these features unmarked, or rely on automated processes in search software to detect word-breaks and link to lemmatizing tools such as Morpheus. (The Papyrological Navigator, Perseus, and the TLG all use methods similar to this.)

To explicitly mark-up lexical words in a papyrological or epigraphic text, however, each word in the text should be enclosed in a w element. (For ease of processing, it is advised that inter-word spacing, punctuation and other features be left outside of this element, and if possible there should be no spaces or carriage returns within the w element.)

<w>maximo</w>
<w>tribunicia</w>
<w>potestate</w>
<num>XXIIII</num>
<w>imperatori</w>
(IRT: 22)

To record the lemma or dictionary headword of the word in question, the simplest solution is to enter the uninflected form in a lemma attribute, which may be used, for example, to generate the entries in a lexical index to the corpus.

<w lemma="ἵστημι">ἕστηκα</w>

A slightly more sophisticated solution is to enter in a lemmaRef attribute a URL or other URI that points to the entry for the word in question in a database or online dictionary. This solution allows for better disambiguation of homonymous words, for example, or linking to morphological and statistical information about the word.

<w lemmaRef="http://www.perseus.tufts.edu/hopper/morph?l=fero&amp;la=la">tulisti</w>

Multiple lemmata

If a single word tag contains more than one indexable word (e.g. "populusque" in Latin, "κἀκ" in Greek or "gudrotæn" and "hlkuþrs" in Old Danish), or an incomplete word is ambiguous and might correspond to either of two or more lemmata (e.g. ΕΚΦΕΥΞ), then all lemmata that should be indexed may be included in the lemma attribute, separated by spaces:

<w lemma="suus que">suisque</w>
<w lemma="καὶ ἐκ">κἀκ</w>
<w lemma="ἔκφευξις ἐκφεύγωpart="I">ΕΚΦΕΥΞ</w>
<gap reason="lostextent="unknown"
 unit="character"/>
<w lemma="Guð dróttin">gud drotæn</w>
<w lemma="helgu Guðs">hlku kuþrs</w>

Interaction with editorial markup

The w element may contain further editorial tags, such as expansions of abbreviations. Wherever possible the w element should remain intact (cf. Editorial restoration: Segmented or adjacent lacunae).

<w lemma="Κύριος"
 lemmaRef="http://www.perseus.tufts.edu/hopper/morph?l=kurie&amp;la=greek">

 <expan>
  <abbr>Κ</abbr>
  <ex>ύρι</ex>ε</expan>
</w>
<w lemma="βοηθέω"
 lemmaRef="http://www.perseus.tufts.edu/hopper/morph?l=bohqei&amp;la=greek">

 <expan>
  <abbr>β</abbr>
  <ex>οή</ex>
  <abbr>θ</abbr>
  <ex>ει</ex>
 </expan>
</w>

displayed in the edition as: Κ(ύρι)ε β(οή)θ(ει); or some restored text:

<w lemma="σός"
 lemmaRef="http://www.perseus.tufts.edu/hopper/morph?l=sw&amp;la=greek">

 <supplied reason="lost">σ</supplied>

</w>

displayed in the edition as: [σ]ῷ; a slighly more complicated situation could be the following, with a lemma written across two lines:

<w lemma="φέρω"
 lemmaRef="http://www.perseus.tufts.edu/hopper/morph?l=ferousa&amp;la=greek">
φ<supplied reason="lost">έ
 <lb n="6break="no"/>ρ</supplied>ουσα
</w>

displayed as:
φ[έ-
6 ρ]ουσα

It should be noticed that in all these examples the w element overlaps the other tags.

Responsibility for this section

  1. Simona Stoyanova, author
  2. Gabriel Bodard, author
  3. Alessio Sopracasa, author
  4. Martina Filosa, author

EpiDoc version: 9.6

Date: 2024-03-20