Failed to get convolution algorithm.

Tensorflow | allow_growth| cuDNN failed to initialize| Image classification

We get this error for image classification using TensorFlow on Windows PC with Nvidia . The solution led us to explore some options regarding GPU memory utilization.

Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node sequential_4/conv2d_12/Relu (defined at <ipython-input-98–471bb77471a2>:2) ]] [Op:__inference_train_function_3889]

The TensorFlow by default allocates full GPU memory even for a small network. One can set a fraction of GPU memory or allow growing memory usage based on need when constructing the session

Setting fraction of memory

# Assume that you have 12GB of GPU memory and want to allocate ~4GB:
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.333)
sess = tf.Session(config=tf.ConfigProto(gpu_options=gpu_options))

The per_process_gpu_memory_fraction acts as a hard upper bound on the amount of GPU memory

Allowing growth of memory usage

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config, ...)

(link)[https://stackoverflow.com/questions/34199233/how-to-prevent-tensorflow-from-allocating-the-totality-of-a-gpu-memory#:~:text=The%20first%20is%20the%20allow_growth,needed%20by%20the%20TensorFlow%20process.]

Packages list

Python
Python 3.8.8 64 bit

List
tensorboard 2.4.1
tensorboard-plugin-wit 1.8.0
tensorflow-estimator 2.4.0
tensorflow-gpu 2.4.1
termcolor 1.1.0

CUDA
CUDA 11.0 (cuda_11.0.3_451.82_win10) and cudnn-11.0-windows-x64-v8.0.4.30

Solution

Set the up allow_growth = True. I paste the following snippet in the begenning of the code.

from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession
config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)

References

  1. https://github.com/tensorflow/tensorflow/issues/24828#issuecomment-464910864
  2. https://kobkrit.com/using-allow-growth-memory-option-in-tensorflow-and-keras-dc8c8081bc96
  3. https://starriet.medium.com/tensorflow-2-0-wanna-limit-gpu-memory-10ad474e2528

Imran is a computer vision and AI enthusiast with a PhD in computer vision. Imran loves to share his experience with self-improvement, technology and travel.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store