HowTo install MXNet on Debian

Share your own howto's etc. Not for support questions!

HowTo install MXNet on Debian

Postby Soul Singin' » 2019-05-28 17:29

For machine learning projects, I use MXNet, which provides a good set of installation instructions. Nonetheless, I wanted to type a few notes on how I built and installed MXNet on Debian Buster and how I built the API bindings for Python and Perl.

The purpose of this HowTo is to share those notes, but if you're interested in learning more about MXNet, the "MXNet Gluon in 60-minutes" article by Thom Lane provides a good introduction. Lane also provides a series of YouTube videos, and at the documentation page, you can find his Jupyter notebooks for the "Gluon Crash Course."

And if you enjoy programming in Perl, then check out the AI::MXNet module by Sergey Kolychev. He kept the Perl syntax as similar as possible to the Python syntax, so I used it to replicate Lane's MXNet course in Perl.

In his own writing, Kolychev also provides an introduction, some examples, a script on neural networks and a picture of his dog. In my work, I am using MXNet to develop a dictionary and a machine translator.


1. set variables and paths

After building the shared library, I wanted to store it in /home/soul/.local/mxnet/lib, so my first step was to add the following variables and paths to my $HOME/.bashrc

Code: Select all
##  local library path -- contains libmxnet.so and more
export LD_LIBRARY_PATH="/home/soul/.local/mxnet/lib"

##  python path
export PYTHONPATH="/home/soul/.local/lib/python3.7/site-packages"

##  perl path
export PERL5LIB="/home/soul/.perl/lib/perl5"


## set PATH so it includes $HOME/.perl/bin if it exists
if [ -d "$HOME/.perl/bin" ] ; then
    PATH="$HOME/.perl/bin:$PATH"
fi

## set PATH so it includes $HOME/.local/bin if it exists
if [ -d "$HOME/.local/bin" ] ; then
    PATH="$HOME/.local/bin:$PATH"
fi

## set PATH so it includes $HOME/.bin if it exists
if [ -d "$HOME/.bin" ] ; then
    PATH="$HOME/.bin:$PATH"
fi


2. install the build dependencies

Next, I installed the build dependencies for the shared library:

Code: Select all
apt-get install build-essential ca-certificates cmake git liblapack-dev libmkldnn-dev libopenblas-dev libopencv-dev ninja-build python3-dev unzip wget

Python's pip3 installer:

Code: Select all
apt-get install python3-pip

and the dependencies for the Perl module:

Code: Select all
apt-get install libmouse-perl pdl cpanminus swig libgraphviz-perl


3. build and install the shared library

Having chosen those paths and variables and having installed the build dependencies, I downloaded the apache-mxnet-src-1.4.1-incubating.tar.gz tarball from https://github.com/apache/incubator-mxnet/releases/tag/1.4.1 and began to build.

First, I extracted the tarball and entered the directory:

Code: Select all
tar -xzf apache-mxnet-src-1.4.1-incubating.tar.gz
cd apache-mxnet-src-1.4.1-incubating/
MXNET_HOME=${PWD}

Then, I built the shared library (saving a copy of the build log) using OpenCV for computer vision, OpenBLAS for linear algebra and Intel's Math Kernel Library for Deep Neural Networks (MKL-DNN):

Code: Select all
make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas USE_MKLDNN=1 | tee ../build-log_2019-05-28.txt

If you have a Nvidia graphics card, you might want to use the CUDA toolkit:

Code: Select all
USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1

After building the shared libraries, I installed them by copying them to /home/soul/.local/mxnet/lib

Code: Select all
cp lib/libiomp5.so  ${LD_LIBRARY_PATH}/.
cp lib/libmkldnn.so.0  ${LD_LIBRARY_PATH}/.
cp lib/libmklml_intel.so  ${LD_LIBRARY_PATH}/.
cp lib/libmxnet.a  ${LD_LIBRARY_PATH}/.
cp lib/libmxnet.so  ${LD_LIBRARY_PATH}/.


4. install the Python bindings

Next, I installed the Python package:

Code: Select all
cd ${MXNET_HOME}/python
pip3 install .

I also had to create a few symbolic links inside the installed package:

Code: Select all
cd /home/soul/.local/lib/python3.7/site-packages/mxnet
ln -s /home/soul/.local/mxnet/lib/libmxnet.so libmxnet.so
ln -s /home/soul/.local/mxnet/lib/libmklml_intel.so libmklml_intel.so
ln -s /home/soul/.local/mxnet/lib/libiomp5.so libiomp5.so


5. validate the installation

After that, I validated the installation by passing the following commands to the python3 interpreter:

Code: Select all
>>> import mxnet as mx
>>> a = mx.nd.ones((2, 3))
>>> b = a * 2 + 1
>>> b.asnumpy()

which returned:

Code: Select all
array([[ 3.,  3.,  3.],
       [ 3.,  3.,  3.]], dtype=float32)

To test MKL-DNN, I also performed the validation at the MKLDNN_README.


6. install the Perl bindings

Next, I installed the Perl packages:

Code: Select all
cpanm -q -L "${HOME}/.perl" Function::Parameters Hash::Ordered PDL::CCS

cd ${MXNET_HOME}/perl-package/AI-MXNetCAPI/
perl Makefile.PL INSTALL_BASE="${HOME}/.perl"
make install

cd ${MXNET_HOME}/perl-package/AI-NNVMCAPI/
perl Makefile.PL INSTALL_BASE="${HOME}/.perl"
make install

cd ${MXNET_HOME}/perl-package/AI-MXNet
perl Makefile.PL INSTALL_BASE="${HOME}/.perl"
make test TEST_VERBOSE=1 
make install

cd ${MXNET_HOME}/perl-package/AI-MXNet-Gluon-Contrib
perl Makefile.PL INSTALL_BASE="${HOME}/.perl"
make install

cd ${MXNET_HOME}/perl-package/AI-MXNet-Gluon-ModelZoo
perl Makefile.PL INSTALL_BASE="${HOME}/.perl"
make test TEST_VERBOSE=1 
make install

And I validated its installation by passing the following commands to the pdl2 interpreter:

Code: Select all
pdl> use AI::MXNet qw(mx)
pdl> $a = mx->nd->ones([2, 3])
pdl> $b = $a * 2 + 1
pdl> print $b->aspdl

which returned:
Code: Select all
[
 [3 3 3]
 [3 3 3]
]

which shows that MXNet and the Perl bindings were successfully installed.


7. have fun with MXNet

:D
User avatar
Soul Singin'
 
Posts: 1577
Joined: 2008-12-21 07:02

Re: HowTo install MXNet on Debian

Postby Soul Singin' » 2019-09-23 16:50

For anyone interested in putting a machine translator online, running MXNet on Debian makes it easy. Just set up your virtual host with suEXEC. Then drop your translation scripts into your cgi-bin.

I posted the complete set up -- from dataset assembly, to training to deployment -- at GitHub. And you can see a working version of at the Napizia translator page.
User avatar
Soul Singin'
 
Posts: 1577
Joined: 2008-12-21 07:02


Return to Docs, Howtos, Tips & Tricks

Who is online

Users browsing this forum: No registered users and 3 guests

fashionable