SentencePiece
SentencePiece: A subword tokenisation algorithm and implementation operating directly on text without requiring prior word segmentation. It supports BPE and unigram vocabularies and helps languages without spaces. Normalisation choices affect the resulting sequence. This perspective connects the definition with practical behaviour in artificial intelligence systems.
In italiano: SentencePieceLegge I · Embedding, feature e spazi vettorialiIt supports BPE and unigram vocabularies and helps languages without spaces. Normalisation choices affect the resulting sequence.