Command Generation based on NLU prediction - Rasa CALM - Rasa Community Forum

Command Generation based on NLU prediction

Post by cicciob95 on Dec 3, 2024

Has anyone developed a command generator that queries an LLM and also provides NLU module predictions, especially intent and entities?

Is it possible to achieve something like this by modifying the prompt of a generator such as SingleStepLLM or MultiStepLLM?

Any suggestions?

Thank you.

Post by souvikg10 on Dec 3, 2024

What would be the difference in comparison to how the command generator works? You can add few shot examples of your intents similar to flows in the flow description to improve the prediction of StartFlow command and add a description in the collect step to improve slot predictions.

I think you can get much more out of the conversations with the command generator instead of making an LLM call to predict intents.

Post by cicciob95 on Dec 4, 2024

Hi @souvikg10, Thank you for your reply. I have noticed that the LLM command generator tends not to fill several slots at the same time during a flow that aims to collect more user data. For example, for :

  diving_info:
    description: Collect information about the diving site
    if : False #  flow which should exclusively be started via a link or call step
    steps:
      - collect: name
        reset_after_flow_ends: False
      - collect: age
        reset_after_flow_ends: False
        next:
          - if: slots.age < 3 OR slots.age > 150
            then:
              - action: utter_invalid_age
                next: complete_briefing
          - else:
              - collect: content_type
                reset_after_flow_ends: False
              - collect: site
                reset_after_flow_ends: False
                next: complete_briefing
      - id: complete_briefing
        action: action_complete_briefing
        next: END

I obtain the following behaviour:

===
Here is the current conversation:
AI: Come posso chiamarti?
USER: Mi chiamo Francesco e ho 59 anni
===

You are currently in the flow "diving_info".
This flow has the following slots:
- name
- age
- content_type
- site

You have just asked the user for the slot "name" (text).

The user answered "Mi chiamo Francesco e ho 59 anni".
===

Based on this information, generate a list of actions. These are the available actions:
1. Setting or correcting slots, described by "SetSlot(slot_name, slot_value)". An example would be "SetSlot(recipient, Freddy)". DO NOT set a slot with an arbitrary value!
2. Indicating that the users intent goes beyond responding to a question from the AI and setting a slot, described by "ChangeFlow()". Add this action, for example, if the user might want to start a different flow, cancel the current one, skip a question, ask a question or engages in chitchat. DO NOT add the name of the flow inside "ChangeFlow" command.
===

Summarize the last user message in the context of the conversation. Then generate a final list of actions.
===

The user saying "Mi chiamo Francesco e ho 59 anni" after being asked for the slot "name" means that they might 2024-12-04 10:56:53 [debug ] base_litellm_client.formatted_response formatted_response={'id': 'chatcmpl-010df699-f93c-42d9-9517-339c63fd2d12', 'choices': ["## Summarize the last user message in the context of the conversation.\n\nThe user gave their name and age in response to a request for their name - it's possible that they misunderstood the question and are giving additional information.\n\n## Generate a final list of actions.\n\n1.  SetSlot(name, Francesco)"], 'created': 1733306211, 'model': 'gemini-pro', 'usage': {'prompt_tokens': 418, 'completion_tokens': 66, 'total_tokens': 484}, 'additional_info': None}

2024-12-04 10:56:53 [debug ] multi_step_llm_command_generator.predict_commands_for_active_flow.actions_generated action_list=## Summarize the last user message in the context of the conversation.

The user gave their name and age in response to a request for their name - it's possible that they misunderstood the question and are giving additional information.

## Generate a final list of actions.

1.  SetSlot(name, Francesco)
2024-12-04 10:56:53 [debug ] command_processor.check_commands_against_slot_mappings.active_flow active_flow=diving_info
2024-12-04 10:56:53 [debug ] graph.node.running_component clazz=RegexMessageHandler fn=process node_name=run_RegexMessageHandler
2024-12-04 10:56:53 [debug ] processor.message.parse parse_data_entities=[{'entity': 'name', 'start': 10, 'end': 19, 'confidence_entity': 0.9904364139874045, 'value': 'Francesco', 'extractor': 'CRFEntityExtractor', 'processors': ['EntitySynonymMapper']}, {'entity': 'age', 'start': 25, 'end': 27, 'confidence_entity': 0.8171495188004679, 'value': '59', 'extractor': 'CRFEntityExtractor'}] parse_data_intent={'name': 'introduce_user', 'confidence': 0.8893842101097107} parse_data_text=Mi chiamo Francesco e ho 59 anni

Note that the NLU module correctly extracts both name and age from me while the LLM model only sets the required slot. What I wonder is why the LLM model does not also set the age slot, which is part of the flow anyway?

Post by souvikg10 on Dec 4, 2024

which LLM are you using?

you can also simply map certain slots onto entities if that performs well using slot_mappings and add in your entity extractor in the pipeline prior to the LLMCommandGenerator . there is a component called NLUCommandAdapter which can take the entities extracted from your NLU pipeline and map them onto slots within a flow in complement with some other slots which you map onto with LLMs.

Post by cicciob95 on Dec 4, 2024

Sorry @souvikg10 , I had accidentally posted the incomplete post. I only added an example log to better explain my query, even if this particular case could actually be solved with your suggestion.

It is clear to me that I can use the entity value as slot_mapping and make use of the NLUCommandAdapter. However, I would need to have a somewhat more particular inference, such as setting a slot on the basis of other complementary information. I give a trivial example that ties in with the previous one. Let us assume that the user is a child I could also set the content_type to a value such as “simplified_text” or “sweetened_text”. Or suppose the bot asks the user if he has any special needs and the user replies that he does not see very well, then content_type could be set to “audio”.