benchmarkstt.metrics.core module

class benchmarkstt.metrics.core.DiffCounts(differ_class=None)

Bases: benchmarkstt.metrics.Base

Get the amount of differences between reference and hypothesis

compare(ref: benchmarkstt.schema.Schema, hyp: benchmarkstt.schema.Schema)

class benchmarkstt.metrics.core.OpcodeCounts(equal, replace, insert, delete)

Bases: tuple

delete

Alias for field number 3

equal

Alias for field number 0

insert

Alias for field number 2

replace

Alias for field number 1

class benchmarkstt.metrics.core.WER(mode=None, differ_class=None)

Bases: benchmarkstt.metrics.Base

Word Error Rate, basically defined as:

insertions + deletions + substitions
------------------------------------
     number of reference words

See: https://en.wikipedia.org/wiki/Word_error_rate

Calculates the WER using one of two algorithms:

[Mode: 'strict' or 'hunt'] Insertions, deletions and substitutions are identified using the Hunt–McIlroy diff algorithm. The 'hunt' mode applies 0.5 weight to insertions and deletions. This algorithm is the one used internally by Python. See https://docs.python.org/3/library/difflib.html

[Mode: 'levenshtein'] In the context of WER, Levenshtein distance is the minimum edit distance computed at the word level. This implementation uses the Editdistance c++ implementation by Hiroyuki Tanaka: https://github.com/aflc/editdistance. See: https://en.wikipedia.org/wiki/Levenshtein_distance

Parameters:

• mode -- 'strict' (default), 'hunt' or 'levenshtein'.
• differ_class -- For future use.

DEL_PENALTY = 1

INS_PENALTY = 1

MODE_HUNT = 'hunt'

MODE_LEVENSHTEIN = 'levenshtein'

MODE_STRICT = 'strict'

SUB_PENALTY = 1

compare(ref: benchmarkstt.schema.Schema, hyp: benchmarkstt.schema.Schema)

class benchmarkstt.metrics.core.WordDiffs(dialect=None, differ_class=None)

Bases: benchmarkstt.metrics.Base

Present differences on a per-word basis

Parameters:

• dialect -- Presentation format. Default is 'cli'.
• differ_class -- For future use.

Example dialect:  

'html'

compare(ref: benchmarkstt.schema.Schema, hyp: benchmarkstt.schema.Schema)

benchmarkstt.metrics.core.get_differ(a, b, differ_class)

benchmarkstt.metrics.core.get_opcode_counts(opcodes)

benchmarkstt.metrics.core.traversible(schema, key=None)