benchmarkstt.segmentation.core module

Core segmenters, each segmenter must be Iterable returning a Item

class benchmarkstt.segmentation.core.Simple(text: str, pattern='[\n\t\s]+', normalizer=None)

Bases: benchmarkstt.segmentation.Base

Simplest case, split into words by white space