OOV token not working with conveRT - Getting Started with Rasa / Tutorials, Resources & Videos - Rasa Community Forum

👋 Introducing the Rasa Playground

OOV token not working with conveRT

post by Akhil on Sep 26, 2020

rasa version: 1.10.14

Hi @Tanja @akelad.

Using the OOV token of the CountVectorsFeaturizer is not compatible with ConveRT?

I am getting these warnings with ConveRT and not with Spacy

auto

2020-09-26 16:27:04 INFO     rasa.nlu.model  - Finished training component.
2020-09-26 16:27:04 INFO     rasa.nlu.model  - Starting to train component RegexFeaturizer
2020-09-26 16:27:04 INFO     rasa.nlu.model  - Finished training component.
2020-09-26 16:27:04 INFO     rasa.nlu.model  - Starting to train component LexicalSyntacticFeaturizer
2020-09-26 16:27:05 INFO     rasa.nlu.model  - Finished training component.
2020-09-26 16:27:05 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
/home/akhil/office/chatbot/dev/venv/lib/python3.6/site-packages/rasa/utils/common.py:363: UserWarning: The out of vocabulary token 'oov' was configured, but could not be found in any one of the NLU training examples. All unseen words will be ignored during prediction.
  More info at https://rasa.com/docs/rasa/nlu/components/#countvectorsfeaturizer
/home/akhil/office/chatbot/dev/venv/lib/python3.6/site-packages/rasa/utils/common.py:363: UserWarning: The out of vocabulary token 'oov' was configured, but could not be found in any one of the ResponseSelector training examples. All unseen words will be ignored during prediction.
  More info at https://rasa.com/docs/rasa/nlu/components/#countvectorsfeaturizer
2020-09-26 16:27:07 INFO     rasa.nlu.model  - Finished training component.
2020-09-26 16:27:07 INFO     rasa.nlu.model  - Starting to train component CountVectorsFeaturizer
2020-09-26 16:27:10 INFO     rasa.nlu.model  - Finished training component.
2020-09-26 16:27:10 INFO     rasa.nlu.model  - Starting to train component DIETClassifier
/home/akhil/office/chatbot/dev/venv/lib/python3.6/site-packages/rasa/nlu/classifiers/diet_classifier.py:589: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  X_dense = np.array(X_dense)
2020-09-26 16:27:10 DEBUG    rasa.utils.tensorflow.models  - Building tensorflow train graph...

post by Tanja on Sep 28, 2020

This is due to the fact that ConveRT is splitting words into sub-words. So the OOV word is split into sub-words and the individual sub-words do not match your OOV tokens anymore. The issue is fixed in Rasa 2.0.0rc2.