df12 Productions — Streaming parser (experimental)

For processing very large TEI documents without loading them entirely into memory, the streaming feature enables incremental parsing via a pull-parser interface.

Enabling the feature

Add the streaming feature to the tei-xml dependency:

[dependencies]
tei-xml = { version = "0.1", features = ["streaming"] }

Usage

The TeiPullParser implements Iterator, yielding TeiEvent values as it processes the document:

use std::io::BufReader;
use std::fs::File;
use tei_xml::streaming::{TeiPullParser, TeiEvent};

fn process_tei(path: &str) -> Result<(), Box<dyn std::error::Error>> {
    let file = File::open(path)?;
    let reader = BufReader::new(file);
    let parser = TeiPullParser::new(reader);

    for event in parser {
        match event? {
            TeiEvent::DocumentStart => println!("Parsing started"),
            TeiEvent::Header(header) => {
                println!("Title: {}", header.file_desc().title().as_str());
            }
            TeiEvent::BodyBlock(block) => println!("Received block: {block:?}"),
            TeiEvent::DocumentEnd => println!("Parsing complete"),
        }
    }
    Ok(())
}

For string slices, use the convenience constructor:

let parser = TeiPullParser::from_str(xml_string);

Event types

The parser yields four high-level event types:

DocumentStart: Emitted once at the beginning of parsing
Header(TeiHeader): The complete header metadata, emitted once after the header section is fully parsed
BodyBlock(BodyBlock): A paragraph, utterance, or division from the body, emitted one at a time as each block is parsed. Division blocks are accumulated with their full child content (lists, items, nested paragraphs and utterances) before being yielded as a single BodyBlock::Div event
DocumentEnd: Emitted once after all content has been successfully parsed

Parser state overview

This diagram shows the main InBody to InDiv flow and the return paths from InHead, InParagraph, and InUtterance.

Full state coverage, including list, item, and label states, is documented in the design document.

The streaming parser currently streams the header and body only. Root-level <standOff> markup is supported by full-document parsing and emission, but it is not yet emitted as a streaming event.

Memory efficiency

The streaming parser yields body blocks one at a time, allowing processing of documents larger than available RAM. The header is fully parsed before body blocks begin, ensuring speaker declarations are available for validation. After the Header event is yielded, the header is also accessible via the parser.header() method.

Error handling

Errors are returned through the iterator's Result type. If an error occurs (malformed XML, unexpected structure, validation failure), the parser yields an Err value and subsequent calls to next() return None.

Python usage

The tei_rapporteur Python module exposes the same streaming iterator via iter_parse(xml: str). Events are returned as tagged dictionaries that decode directly into the tei_rapporteur.structs.Event union:

import msgspec
import tei_rapporteur as tr
from tei_rapporteur.structs import Event

xml = (
    "<TEI><teiHeader><fileDesc><title>Wolf 359</title></fileDesc></teiHeader>"
    "<text><body><p>Hello <hi rend='stress'>there</hi></p></body></text></TEI>"
)

for event in tr.iter_parse(xml):
    typed = msgspec.convert(event, type=Event)
    print(typed)

Events use internal tagging (type), covering:

document_start
header (with a structured header field)
paragraph / utterance – carrying content: list[Inline] as tagged Inline values (text, hi, pause)
div (carries div_type, content: list[DivContent], and optional subtype, head, and xml_id; decodes into DivEvent)
document_end

Inline content is also tagged (text, hi, pause), so Python callers can type-check inline nodes without falling back to Any. Streamed utterance events include the same local provenance fields as full Utterance structs: n, source, resp, cert, corresp, and ana.

Limitations

The header is accumulated in memory before being deserialized, so documents with unusually large headers may still consume significant memory during that phase.