Minimal viable LLM for Command Generation? - Rasa CALM - Rasa Community Forum
Minimal viable LLM for Command Generation?
Post by tozo on Aug 8, 2024
Hi there,
can anyone share their experiences on what they found to be the smallest (ideally locally runnable) model that can still reliably perform the command generation task? I tried LLama3.1 8B (Q8), but it was not really working well.
Post by Arjaan on Aug 8, 2024
Hi @tozo,
You can find some LLMs on huggingface that are fine-tuned for use with Rasa Pro CALM.
You run them locally like any llama model.
These LLMs are fine-tuned on rasa-calm-demo, but will likely work quite nice for your bot as well.
I recommend you try out:
Please let us know how it works out.
We will publish instructions soon how you can further fine-tune these LLMs on your own model. Stay tuned!
Post by tozo on Aug 9, 2024
Thank you! I will try it out as soon as I have time!
Post by Elliot94 on Aug 9, 2024
I personally used phi3:14b-instruct from Ollama and it worked decently well. However a point I noticed is that if you have multiple similar flows with only minor changes, the smaller models fail to capture the differences. But if its for entirely different tasks or conversations, Phi3 worked well for me.
Post by sk1382 on Aug 21, 2024
Hi @Arjaan Could you provide more models like these that supports Rasa CALM well or the list/link where can we find them?
Post by Arjaan on Oct 1, 2024
@here,
I want to let you know that fine-tuning your own LLM is possible with Rasa Pro 3.10
I recommend you start by reading this blog post: https://rasa.com/blog/reliable-agentic-bots-with-llama-8b
And then follow the proposed next steps at the end of that blog post.
Post by vkaabunga on Oct 7, 2024
Can I do the fine-tuning of the LLM in a language other than English?
I am thinking of using this approach to give my bot multi-lingual capability.