👋 Introducing the Rasa Playground

Intent MissClassification on Rasa Upgrade

post by surya7592 on Nov 7, 2019

Hi Team,

We have currently migrated to Rasa 1.3.9 from Rasa 0.13.8. It has a total of 136 intents, nearly 90 small talk intents. It seems to work fine before migration, and the confidence of prediction was always above 75 percentage.

But after rasa upgrade the confidence of prediction of intents went low, mainly for the small talk intents and for inform. We tried possible hyper parameter optimization. But was not able to replicate the previous results. Can you please help.

Please find the config used.

language: "en"

pipeline:
- name: "WhitespaceTokenizer"
- name: "RegexFeaturizer"
- name: "CRFEntityExtractor"
- name: "EntitySynonymMapper"
- name: "CountVectorsFeaturizer"
- name: "EmbeddingIntentClassifier"

# Configuration for Rasa Core.
# https://rasa.com/docs/rasa/core/policies/
policies:
  - name: KerasPolicy
    nlu_threshold: 0.6
    core_threshold: 0.6
    epochs: 300
    max_history: 3
  - name: MemoizationPolicy
    max_history: 3
  - name: FormPolicy
  - name: FallbackPolicy
    nlu_threshold: 0.70
    core_threshold: 0.75
    fallback_action_name: 'action_fallback'

Regards, Surya

post by KarthiAru on Nov 30, 2019

Can you try the following config?

language: en
pipeline:
- name: WhitespaceTokenizer
- name: CRFEntityExtractor
- name: EntitySynonymMapper
- name: CountVectorsFeaturizer
  stop_words: {'english'}
  analyzer: word
  token_pattern: r'(?u)\b\w\w+\b'
  lowercase: true
  max_ngram: 5
  min_ngram: 1
- name: CountVectorsFeaturizer
  analyzer: char_wb
  lowercase: true
  max_ngram: 5
  min_ngram: 2
- name: EmbeddingIntentClassifier
  random_seed: 12345
  epochs: 100

policies:
- name: FormPolicy
- name: MemoizationPolicy
  max_history: 6
- name: MappingPolicy
- name: KerasPolicy
  rnn_size: 64
  epochs: 100
  batch_size: 32
  validation_split: 0.1
  max_history: 6
  random_seed: 12345
- name: FallbackPolicy
  nlu_threshold: 0.70
  core_threshold: 0.75
  fallback_action_name: 'action_fallback'

Can you share a few intents & utterances where the confidence scores are low? Can you also check this on version 1.4.5?