Feedback: Upgrading to Tensorflow 2.6 - Rasa Open Source - Rasa Community Forum

Feedback: Upgrading to Tensorflow 2.6

Rasa will upgrade from Tensorflow 2.3 to Tensorflow 2.6 as of Rasa Open Source version 2.8.9. The upgrade is required due to security concerns but it may have consequences on training times in the short term.

You can read more details about the upgrade on our blog and we’d like to use this thread to collect metrics/feedback from the community.

post by koaning on Oct 8, 2021

I’ve just upgraded and I’m experiencing a huge increase in training time.

Current setup:

Rasa Version      :         2.8.9
Minimum Compatible Version: 2.8.9
Rasa SDK Version  :         2.8.2
Rasa X Version    :         0.42.3
Python Version    :         3.7.11
Operating System  :         Linux-5.13.19-2-MANJARO-x86_64-with-arch-Manjaro-Linux
Python Path       :         /home/joan/Desktop/rasa-upgrade/VihrtualApp/venv/bin/python3.7

Project repo: https://github.com/joancipria/VihrtualApp

post by joancipria on Oct 14, 2021

Training time before upgrading was about 35 minutes, now is about 3 hours. I moved from Rasa 2.5.2 to 2.8.9.

post by fkoerner on Oct 15, 2021

Thank you for sharing! These workarounds may be an option for you:

  1. turn entity_recognition for DIETClassifier off, and use CRFEntityExtractor instead
  2. downgrade to Rasa version 2.8.8

post by joancipria on Oct 15, 2021

Thanks for the advice! I’ve followed your instructions and now training times are back to normal. Will using CRFEntityExtractor instead of DIETClassifier for entity recognition produce a bad impact on the model?

post by fkoerner on Oct 15, 2021

It won’t be exactly the same behaviour – I’d recommend that you try it out with a test set to see if the entities are still extracted as expected. If performance has degraded, you can counteract this by adding more examples for those entities.

post by nonola on Oct 22, 2021

With the configuration:

Rasa Version      :         2.8.11
Minimum Compatible Version: 2.8.9
Rasa SDK Version  :         2.8.2
Rasa X Version    :         0.42.4
Python Version    :         3.8.10
Operating System  :         Linux-4.15.0-1026-gcp-x86_64-with-glibc2.29
Python Path       :         /opt/venv/bin/python

When training the DIET classifier, this appears:

/opt/venv/lib/python3.8/site-packages/rasa/utils/tensorflow/model_data.py:750: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  np.concatenate(np.array(f)),

Is it “normal”?

post by koaning on Oct 22, 2021

If it’s a mere warning I wouldn’t be too concerned.

post by nonola on Oct 22, 2021

The problem is that this happened at 40%/epochs:

... (long error trace omitted for readability) ...

And it stopped…

post by fkoerner on Oct 29, 2021

Hi @nonola this warning is indeed “normal” (we are aware of it, and plan to address it in the future). I also don’t think it is related to your training stopping, that seems to be an OOM (out of memory) issue. If you were able to train this model on rasa<2.8.9 and are now running into OOM it’s possible the TF upgrade is the culprit. You have three options:

  1. train on a machine with more memory
  2. reduce the memory requirements of training your model
  3. downgrade your rasa version back to <2.8.9

post by koaning on Nov 5, 2021

@nonola does you have maybe three examples from a nlu.yml that I can copy to reproduce the error?