Learn how to make BERT smaller and faster - Getting Started with Rasa / Tutorials, Resources & Videos - Rasa Community Forum
Learn how to make BERT smaller and faster
Hey everyone,
We’ve released a new blog post about compressing huge neural language models such as BERT: Learn how to make BERT smaller and faster. Edit: there is now another one about pruning BERT in particular.
Even though the blog post is aimed primarily at ML researchers and practitioners, the topic is very much relevant to everyone who wants to use today’s best-performing language models for tasks like intent classification.
If you’ve got any questions, ideas, comments, post them here!
post by SamS on Aug 12, 2019
Will your next post be on pruning etc, I would love to see how I can implement this directly into my Rasa tests. I have been learning a lot from directly using Rasa, fixing issues, learning as I go. So it would be cool to see how I can utilize this
post by FelixKing on Aug 13, 2019
post by SamS on Aug 14, 2019
Hey @FelixKing, thanks!
I have tried weight pruning and am currently exploring neuron pruning. The results look promising so far, though any inference time improvements are still to be measured. If you’re not afraid of dirty code, you can just watch my branch of the rasa repo here for live updates.
What is your motivation when it comes to model compression? Do you just want to see if big models can be made faster, or is it something else that’s driving your interest?
post by cyrilthank on Aug 29, 2019
Is there a way I can incorporate/configure/enable compression in Rasa (I am thinking config.yml or a combination of such)?
Thank you
post by JulianGerhard on Aug 29, 2019
Hi @cyrilthank, maybe you might want to think about this article from spacy in which distillation is properly explained. I’ve made really good experiences with packaging a e.g. finetuned BERT model to a spaCy model which could problem-free be imported into Rasa, e.g. by using the following pipeline:
language: de
pipeline:
- name: SpacyNLP
case_sensitive: 1
model: de_pytt_bertbasecased_lg_jg
- name: SpacyTokenizer
- name: SpacyFeaturizer
- name: SklearnIntentClassifier
The results are really good and there is currently no lack of performance.
post by SamS on Aug 29, 2019
In addition to @JulianGerhard’s advice, I should say that a model pruning blog post is coming soon. I am also making the code for a BERT-based intent classifier (which supports weight/neuron pruning) significantly easier to use.
That being said, Rasa probably won’t officially ship model compression techniques in the near future, but the code I wrote should serve as a good, reusable example in case you want to apply quantisation, weight pruning or neuron pruning to a model.
What kind of model are you trying to compress, @cyrilthank?
post by JulianGerhard on Sep 3, 2019
I have created a bert_spacy_rasa repo in which I describe a quick dive into the matter. The distillation part however needs to be added and maybe we could collaborate at this point.
I keep this repo pretty much up-to-date so if you want to share the code @SamS I can use it.
My idea behind this repo was to make life easier for those, who want to follow the track and don’t want to or don’t have the time to puzzle everything together.
Feel free to share your thoughts!
post by cyrilthank on Sep 4, 2019
Thanks much @JulianGerhard! Can you please let me know where I may reach out to you for ‘embarrassingly silly questions’?
post by JulianGerhard on Sep 4, 2019
Hi @cyrilthank, I think silly questions don’t exist - in contrary - if you have them, others might have them too - so I suggest to post them here!
Usually, I am rather quick!
post by cyrilthank on Sep 4, 2019
Can you please share about using two models in the same pipeline for Rasa?
post by JulianGerhard on Sep 4, 2019
Correct - this dataset is not optimal since it cannot directly be seen as a conversational AI related dataset but it fitted my needs:
- It is large enough to be representative for a finetuning evaluation task
- It has only a few classes which means that if a class is seen as an ‘intent’ in Rasa, it can easily be integrated since there is only minor manual effort necessary
- Imagine the bot as a classifier… you want to know the domain for a given article and the bot utters its domain - seems to be fair enough for the moment - doesn’t it?
post by cyrilthank on Sep 4, 2019
How can I leverage BERT in some form for my domain-specific Entity/Intent recognition scheme?
post by JulianGerhard on Sep 4, 2019
You would be able to finetune BERT on a domain-specific news-data set, if there is enough data to let BERT learn from it. This can be done either only by finetuning BERT or by doing it with the spaCy-pytorch-transformers library.
If you have any specific questions about your use case, please describe a bit more detailed what you want to achieve!
post by cyrilthank on Sep 5, 2019
I want to fine-tune BERT using domain-specific news data so that Rasa can correctly pick up entities like ‘it is hot’ as ‘weather_intent’ and not as ‘spice_intent’. What are the steps to achieve that?