Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

HowTo install MXNet on Debian

Share your HowTo, Documentation, Tips and Tricks. Not for support questions!.
Post Reply
Message
Author
User avatar
Soul Singin'
Posts: 1605
Joined: 2008-12-21 07:02

HowTo install MXNet on Debian

#1 Post by Soul Singin' »

For machine learning projects, I use MXNet, which provides a good set of installation instructions. Nonetheless, I wanted to type a few notes on how I built and installed MXNet on Debian Buster and how I built the API bindings for Python and Perl.

The purpose of this HowTo is to share those notes, but if you're interested in learning more about MXNet, the "MXNet Gluon in 60-minutes" article by Thom Lane provides a good introduction. Lane also provides a series of YouTube videos, and at the documentation page, you can find his Jupyter notebooks for the "Gluon Crash Course."

And if you enjoy programming in Perl, then check out the AI::MXNet module by Sergey Kolychev. He kept the Perl syntax as similar as possible to the Python syntax, so I used it to replicate Lane's MXNet course in Perl.

In his own writing, Kolychev also provides an introduction, some examples, a script on neural networks and a picture of his dog. In my work, I am using MXNet to develop a dictionary and a machine translator.


1. set variables and paths

After building the shared library, I wanted to store it in /home/soul/.local/mxnet/lib, so my first step was to add the following variables and paths to my $HOME/.bashrc

Code: Select all

##  local library path -- contains libmxnet.so and more
export LD_LIBRARY_PATH="/home/soul/.local/mxnet/lib"

##  python path
export PYTHONPATH="/home/soul/.local/lib/python3.7/site-packages"

##  perl path
export PERL5LIB="/home/soul/.perl/lib/perl5"


## set PATH so it includes $HOME/.perl/bin if it exists
if [ -d "$HOME/.perl/bin" ] ; then
    PATH="$HOME/.perl/bin:$PATH"
fi

## set PATH so it includes $HOME/.local/bin if it exists
if [ -d "$HOME/.local/bin" ] ; then
    PATH="$HOME/.local/bin:$PATH"
fi

## set PATH so it includes $HOME/.bin if it exists
if [ -d "$HOME/.bin" ] ; then
    PATH="$HOME/.bin:$PATH"
fi
2. install the build dependencies

Next, I installed the build dependencies for the shared library:

Code: Select all

apt-get install build-essential ca-certificates cmake git liblapack-dev libmkldnn-dev libopenblas-dev libopencv-dev ninja-build python3-dev unzip wget
Python's pip3 installer:

Code: Select all

apt-get install python3-pip
and the dependencies for the Perl module:

Code: Select all

apt-get install libmouse-perl pdl cpanminus swig libgraphviz-perl
3. build and install the shared library

Having chosen those paths and variables and having installed the build dependencies, I downloaded the apache-mxnet-src-1.4.1-incubating.tar.gz tarball from https://github.com/apache/incubator-mxn ... /tag/1.4.1 and began to build.

First, I extracted the tarball and entered the directory:

Code: Select all

tar -xzf apache-mxnet-src-1.4.1-incubating.tar.gz
cd apache-mxnet-src-1.4.1-incubating/
MXNET_HOME=${PWD}
Then, I built the shared library (saving a copy of the build log) using OpenCV for computer vision, OpenBLAS for linear algebra and Intel's Math Kernel Library for Deep Neural Networks (MKL-DNN):

Code: Select all

make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas USE_MKLDNN=1 | tee ../build-log_2019-05-28.txt
If you have a Nvidia graphics card, you might want to use the CUDA toolkit:

Code: Select all

USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1
After building the shared libraries, I installed them by copying them to /home/soul/.local/mxnet/lib

Code: Select all

cp lib/libiomp5.so  ${LD_LIBRARY_PATH}/.
cp lib/libmkldnn.so.0  ${LD_LIBRARY_PATH}/.
cp lib/libmklml_intel.so  ${LD_LIBRARY_PATH}/.
cp lib/libmxnet.a  ${LD_LIBRARY_PATH}/.
cp lib/libmxnet.so  ${LD_LIBRARY_PATH}/.
4. install the Python bindings

Next, I installed the Python package:

Code: Select all

cd ${MXNET_HOME}/python
pip3 install .
I also had to create a few symbolic links inside the installed package:

Code: Select all

cd /home/soul/.local/lib/python3.7/site-packages/mxnet
ln -s /home/soul/.local/mxnet/lib/libmxnet.so libmxnet.so 
ln -s /home/soul/.local/mxnet/lib/libmklml_intel.so libmklml_intel.so 
ln -s /home/soul/.local/mxnet/lib/libiomp5.so libiomp5.so
5. validate the installation

After that, I validated the installation by passing the following commands to the python3 interpreter:

Code: Select all

>>> import mxnet as mx
>>> a = mx.nd.ones((2, 3))
>>> b = a * 2 + 1
>>> b.asnumpy()
which returned:

Code: Select all

array([[ 3.,  3.,  3.],
       [ 3.,  3.,  3.]], dtype=float32)
To test MKL-DNN, I also performed the validation at the MKLDNN_README.


6. install the Perl bindings

Next, I installed the Perl packages:

Code: Select all

cpanm -q -L "${HOME}/.perl" Function::Parameters Hash::Ordered PDL::CCS 

cd ${MXNET_HOME}/perl-package/AI-MXNetCAPI/
perl Makefile.PL INSTALL_BASE="${HOME}/.perl" 
make install

cd ${MXNET_HOME}/perl-package/AI-NNVMCAPI/ 
perl Makefile.PL INSTALL_BASE="${HOME}/.perl" 
make install

cd ${MXNET_HOME}/perl-package/AI-MXNet 
perl Makefile.PL INSTALL_BASE="${HOME}/.perl" 
make test TEST_VERBOSE=1  
make install

cd ${MXNET_HOME}/perl-package/AI-MXNet-Gluon-Contrib 
perl Makefile.PL INSTALL_BASE="${HOME}/.perl" 
make install

cd ${MXNET_HOME}/perl-package/AI-MXNet-Gluon-ModelZoo 
perl Makefile.PL INSTALL_BASE="${HOME}/.perl" 
make test TEST_VERBOSE=1  
make install
And I validated its installation by passing the following commands to the pdl2 interpreter:

Code: Select all

pdl> use AI::MXNet qw(mx)
pdl> $a = mx->nd->ones([2, 3])
pdl> $b = $a * 2 + 1
pdl> print $b->aspdl
which returned:

Code: Select all

[
 [3 3 3]
 [3 3 3]
]
which shows that MXNet and the Perl bindings were successfully installed.


7. have fun with MXNet

:D

User avatar
Soul Singin'
Posts: 1605
Joined: 2008-12-21 07:02

Re: HowTo install MXNet on Debian

#2 Post by Soul Singin' »

For anyone interested in putting a machine translator online, running MXNet on Debian makes it easy. Just set up your virtual host with suEXEC. Then drop your translation scripts into your cgi-bin.

I posted the complete set up -- from dataset assembly, to training to deployment -- at GitHub. And you can see a working version of at the Napizia translator page.

Post Reply