rasa.nlu.tokenizers.lm_tokenizer

LanguageModelTokenizer Objects

This tokenizer is deprecated and will be removed in the future.

Use the LanguageModelFeaturizer with any other Tokenizer instead.

init

__init__(component_config: Dict[Text, Any]=None) -> None

Initializes LanguageModelTokenizer for tokenization.

Arguments:

• component_config - Configuration for the component.