👋 Introducing the Rasa Playground

Tokens None from previous component(RASA custom component)

post by sairam on Feb 1, 2021

I created a custom component. I want to use tokens from the previous component, but I am getting "tokens none ". Please help me to solve this. Below is my “custom_component.py” code

from rasa.nlu.components import Component
from rasa.shared.nlu.training_data.message import Message
from fuzzywuzzy import process

class FuzzyExtractor(Component):
    name = "FuzzyExtractor"
    provides = ["entities"]
    requires = ["tokens"]
    defaults = {}
    language_list = ["en"]
    threshold = 90

def __init__(self, component_config=None, *args):
        super(FuzzyExtractor, self).__init__(component_config)

def train(self, training_data, cfg, **kwargs):
        pass

def process(self, message:Message, **kwargs):
        print('message', message)
        entities = list(message.get('entities'))
        print('entities', entities)
        tokens = message.get("tokens")
        print('tokens', tokens)
        message.set("entities", entities, add_to_output=True)

config.yml code

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en

pipeline:
  - name: WhitespaceTokenizer
  - name: custom_component.FuzzyExtractor
  - name: RegexFeaturizer
    "case_sensitive": false
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: char_wb
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    entity_recognition: False
    epochs: 100
  - name: "CRFEntityExtractor"
  - name: EntitySynonymMapper
  - name: ResponseSelector
    epochs: 100
  - name: DucklingEntityExtractor
    url: http://localhost:8000
    dimensions:
    - time
    - number
    timezone: "Asia/Mumbai"
  - name: FallbackClassifier
    threshold: 0.5

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
#   - name: MemoizationPolicy
#   - name: TEDPolicy
#     max_history: 5
#     epochs: 100
#   - name: RulePolicy

Rasa shell --debug logs

Next message:
hi
message <rasa.shared.nlu.training_data.message.Message object at 0x0000019811BE4DC8>
entities []
tokens None

post by souvikg10 on Feb 3, 2021

Try to replace it with tokens = [t.text for t in message.get(“tokens”)]

post by minlik on Oct 3, 2021

Have you ever solved this? I occured the same problem when I was trying to create my custom component.

post by souvikg10 on Oct 4, 2021

in rasa 2.6, I access tokens like this

tokens = [t.text for t in ex.get("text_tokens")]

where is ex is message object btw. this will give you a list of tokens for every example