rasa.nlu.tokenizers.jieba_tokenizer

You are viewing documentation for our open source project which is maintained by the community. If you want to get started building assistants with Rasa please check out our latest documentation here.

JiebaTokenizer Objects #

class JiebaTokenizer(Tokenizer)

This tokenizer is a wrapper for Jieba ( https://github.com/fxsjy/jieba).

init #

| init(component_config: Dict[Text, Any]=None)->None

Construct a new intent classifier using the MITIE framework.

load_custom_dictionary #

| @staticmethod

| load_custom_dictionary(path: Text)->None

Load all the custom dictionaries stored in the path.

More information about the dictionaries file format can be found in the documentation of jieba. https://github.com/fxsjy/jieba#load-dictionary

load #

| @classmethod

| load(cls, meta: Dict[Text, Any], model_dir: Text, model_metadata: Optional["Metadata"] = None, cached_component: Optional[Component] = None, **kwargs: Any)->"JiebaTokenizer"

Loads trained component (see parent class for full docstring).

persist #

| persist(file_name: Text, model_dir: Text)-> Optional[Dict[Text, Any]]

Persist this model into the passed directory.