De Wiki de Calcul Québec
Aller à : Navigation, rechercher
Cette page est une traduction de la page Tensorflow et la traduction est complétée à 100 % et à jour.

Autres langues :anglais 100% • ‎français 100%

TensorFlow is an open source machine-learning tool developed by Google

This page contains the information necessary for the compilation and use of TensorFlow on the GPU clusters of Calcul Québec


Installation and Use on Guillimin

Listing the Available Versions

To see the available versions, enter the following command:

 [name@lg-1r14-n04 ~]$ module spider Tensorflow

Loading the Required Modules

These modules should be loaded before any use of TensorFlow:

 [name@lg-1r14-n04 ~]$ module load foss/2015b

Python 2

 [name@lg-1r14-n04 ~]$ module load Tensorflow/1.0.0-Python-2.7.12

Python 3

 [name@lg-1r14-n04 ~]$ module load Tensorflow/1.0.0-Python-3.5.2

Installation and Use on Helios

Listing the Available Versions

The most recent version of Tensorflow is made available in the weeks following its introduction. To see the available versions, enter the following command

 [name@helios ]$ ls /software-gpu/apps/python/wheelhouse/tensorflow-*

Loading the Required Modules

These modules should be loaded before any use of TensorFlow.

 [name@helios ]$ module load compilers/gcc/4.8.5 cuda/8.0.44 libs/cuDNN/6

Python 2

 [name@helios ]$ module load apps/python/2.7.10

Python 3

 [name@helios ]$ module load apps/python/3.5.0

Installing with pip

To install the latest available version, create and activate a virtual environment. Then enter the following command:

 [name@helios venv]$ pip install tensorflow

Testing the Tensorflow installation

 [name@helios venv]$ python -m tensorflow.models.image.mnist.convolutional

Compiling TensorFlow (Advanced)

This section explains how to compile Tensorflow on Calcul Québec servers and should only be consulted if you want to modify the Tensorflow source code and create your own installation package. If you simply want to use Tensorflow you should refer to one of the preceding sections.

Detailed compilation instructions for Helios (v1.0.1)

Load the Necessary Modules

 [name@helios ]$ module purge -f
 [name@helios ]$ module load compilers/gcc/4.8.5 compilers/java/1.8 apps/buildtools \
                           cuda/7.5 libs/cuDNN/5 compilers/swig apps/git apps/bazel/0.4.3

Obtain the TensorFlow source and move to the source directory.

 [name@helios ]$ git clone https://github.com/tensorflow/tensorflow.git
 [name@helios ]$ cd tensorflow

3. Select the version of the code (checkout)

 [name@helios tensorflow]$ git checkout 1.0.1

Configuring the Compiler Options

 [nom@helios tensorflow]$ cp third_party/gpus/crosstool/CROSSTOOL.tpl{,.orig}
 [nom@helios tensorflow]$ GCC_PREFIX=$(dirname $(dirname $(which gcc)))
 [nom@helios tensorflow]$ sed -i "\;linker_flag: \"-B/usr/bin/\";a \
 > \ \ linker_flag: \"-Wl,-rpath=$GCC_PREFIX/lib64\"" third_party/gpus/crosstool/CROSSTOOL.tpl
 [nom@helios tensorflow]$ sed -i "\;linker_flag: \"-B/usr/bin/\";a \
 > \ \ linker_flag: \"-Wl,-rpath=$GCC_PREFIX/lib\"" third_party/gpus/crosstool/CROSSTOOL.tpl
 [nom@helios tensorflow]$ sed -i "s;-B/usr/bin;-B$(dirname $(which ar));g" third_party/gpus/crosstool/CROSSTOOL.tpl

Create a Python virtual environment for the compilation.

Python 2
 [name@helios tensorflow]$ module load apps/python/2
 [name@helios tensorflow]$ virtualenv buildenv

Python 3
 [name@helios tensorflow]$ module load apps/python/3
 [name@helios tensorflow]$ pyvenv buildenv

Activate the Virtual Environment

 [name@helios tensorflow]$ source buildenv/bin/activate

Install the Necessary Python Packages

 (buildenv)[name@helios tensorflow]$ pip install numpy wheel
 Collecting numpy
 Collecting wheel
 Installing collected packages: numpy, wheel

Adapt the Crosstool Wrapper Script and correct bug for this version

 (buildenv)[name@helios tensorflow]$ sed -i "/cmd = 'PATH=' + PREFIX_DIR + ' ' + cmd/s/^/#/" third_party/gpus/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc.tpl

Configure the Compilation Options

 (buildenv)[name@helios tensorflow]$ PYTHON_BIN_PATH=$(which python) \
 GCC_HOST_COMPILER_PATH=$(which gcc) \
 CUDNN_INSTALL_PATH="/software-gpu/libs/cuDNN/5_cuda7.5" \

Compiling TensorFlow and the build_pip_package Application

 (buildenv)[name@helios tensorflow]$ MALLOC_ARENA_MAX=4 bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

Creating the wheel package for TensorFlow

 (buildenv)[name@helios tensorflow]$ mkdir dist
 (buildenv)[name@helios tensorflow]$ bazel-bin/tensorflow/tools/pip_package/build_pip_package $PWD/dist

The compiled TensorFlow wheel package will be in the dist sub-directory.

Quitting the Virtual Environment

 (buildenv)[name@helios tensorflow]$ deactivate

Stop the Bazel compilation server

 [name@helios tensorflow]$ bazel shutdown

Compilation Scripts

Guillimin-specific instructions

Using the Helios and [1] compilation notes.

Adjustments for v0.10.0

See /software/CentOS-6/tools/python/tensorflow-0.10.diff

Paths to /sb/software and /software are both necessary. For example:

  • cxx_builtin_include_directory: "/sb/software/CentOS-6/eb/software/Core/GCCcore/4.9.3/include"
  • cxx_builtin_include_directory: "/software/CentOS-6/eb/software/Core/GCCcore/4.9.3/include"
Outils personnels
Espaces de noms

Ressources de Calcul Québec