👋 Introducing the Rasa Playground

LLM set’s slots without user input

post by Peveld on May 29, 2024

Hi, I’ve created a pretty simple example trying to set slots for a product search. My problem is that the LLM sets a slot without any user input. When asked the LLM sees its error and corrects but of course in rasa it wont’t get corrected. The problem is (most times) reproducable in Chat-GPT. Model is GPT 4, 3.5 Turbe seems to be not able to fill multiple slots from one response. So here’s a dialog that describes my problem:

generated rasa prompt:

Your task is to analyze the current conversation context and generate a list of actions to start new business processes that we call flows, to extract slots, or respond to small talk and knowledge requests.

These are the flows that can be started, with their description and slots:

collect_product: this flow lets the user search a product
    slot: modelname (the modelname of the product)
    slot: color (the color of the product)
    slot: length (the length of the product)
    slot: width (the width of the product)
    slot: antislip (product should have antislip coating), allowed values: [True, False]
    slot: perleffect (product should have perleffect coating), allowed values: [True, False]

get_delivery_time: this flow gets the delivery time by backend query.

===
Here is what happened previously in the conversation:
USER: find product
AI: what is the modelname of the product?
USER: saniform white 1700x700
AI: should the product be coated with antislip?
USER: no

===

You are currently in the flow "collect_product".
You have just asked the user for the slot "antislip" (product should have antislip coating).

Here are the slots of the currently active flow:
- name: modelname, value: saniform white, type: text, description: the modelname of the product of Kaldewei
- name: color, value: white, type: text, description: the color of the product
- name: length, value: 1700.0, type: float, description: the length of the product
- name: width, value: 700.0, type: float, description: the width of the product
- name: antislip, value: undefined, type: bool, description: product should have antislip coating, allowed values: [True, False]
- name: perleffect, value: undefined, type: bool, description: product should have perleffect coating, allowed values: [True, False]

If you start a flow, first start the flow and then optionally fill that flow's slots with information the user provided in their message.

Your action list:

Model answer:

SetSlot(antislip, False)
SetSlot(perleffect, False)

Asking it why:

why did you set slot perleffect

Answer:

It looks like I made an error in setting the "perleffect" slot without your instruction. I should have only set the "antislip" slot to False based on your response and then asked you about the "perleffect" coating. Let's correct that:

SetSlot(perleffect, undefined)
Should the product have a perleffect coating?

Asking why:

yes, you first made an error. Is there a reason in the prompt? Could I have better prompted?

Answer:

Your prompt was clear and well-structured for the task you wanted to achieve. The mistake was on my part; I misinterpreted your last message and preemptively set a slot value for "perleffect" without having a directive from you to do so. Your direction in the prompt was appropriate, focusing on sequentially filling each slot as required by the user's input in the context of the "collect_product" flow.

In future interactions, to prevent similar mistakes, ensuring each user input is specifically directed towards the request could help. For instance, asking direct questions about each slot sequentially, as you have been doing, is a good practice. This helps maintain clarity and ensures each response from the user directly corresponds to a specific request or question.

post by abboudtest on May 29, 2024

what should the correct answer be in this case, setting antislip to False only ?

post by Peveld on May 29, 2024

yes, correct answer would be setting antislip false and asking for perleffect

post by jtrasa on May 29, 2024

In the collect step in your flow, you can set ask_before_filling: true to force the collect step even if the slot is filled.

From Rasa Docs :

By default, a collect step is skipped if the corresponding slot is already filled. If a collect step should always be asked, no matter if the underlying slot is already filled, you can set ask_before_filling: true on the collect step …

post by Peveld on May 29, 2024

Hi John, I have read the docs there. But in my case the Slot was not already filled. You can see that in the prompt. The LLM fills it wrongly and when asked it says it made a mistake… Please see this passage in the prompt: