CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2theanodeep-learningcaffe
Prathamesh M Kulkarni
Prathamesh M Kulkarni

Reputation: 3

Library for deep learning on Amazon EC2 with CPU and GPU support for convolutional neural network

I want to train a CNN on a bunch of images. I want to run it on Amazon EC2 CPU or GPU clusters. For running deep learning on a cluster, I figured that some of the options are:

  1. h2o (with Spark)
  2. Caffee
  3. Theano

I am not sure which of these options suit my needs. I read through h2o documentation on deep learning, they do not seem to support CNNs. Any ideas on how I should proceed?

Another side question: How do I upload my images to the cluster for training the CNN? I am fairly new to cluster computing.

Upvotes: 0

Views: 734

Answers (3)

rgaut
rgaut

Reputation: 3599

AWS provides the Deep Learning AMI with various Deep Learning Frameworks installed into it, which covers your use case since it has Theano as well as Caffe. Link to Deep Learning AMI https://aws.amazon.com/machine-learning/amis/.

How do I upload my images to the cluster for training the CNN? I am fairly new to cluster computing?

There are many AWS storage services which gives you way to store your training data (images) which will be accessible to your cluster. Few of them are

  1. S3

  2. EBS

  3. EFS

    Explore them and see what works best for you.

Upvotes: 1

ted
ted

Reputation: 14764

Just came accross your question. In this tutorial you'll also find how to set up an Amazon instance with a GPU to run Deep Learning frameworks.

The AMI (~computer model) is pre-configured with:

  • Ubuntu Server 16.04 as OS
  • Anaconda 4.2.0 (scientific Python distribution)
  • Python 3.5
  • Cuda 8.0 (“parallel computing platform and programming model”, used to send code to the GPU)
  • cuDNN 5.1 (Cuda’s library for Deep Learning used by Tensorflow and Theano)
  • Tensorflow 0.12 for Python 3.5 and GPU-enabled
  • Keras 1.1.2 (use with Tensorflow backend)

I believe you can use this set-up with elastic GPUs to scale the system according to your needs or use a P2 instance

Anyway you can follow the tutorial and use another AMI like Amazon's Deep Learning AMI

Upvotes: 1

eickenberg
eickenberg

Reputation: 14377

If you follow the instructions here https://github.com/deeplearningparis/dl-machine then you can set up an AMI image with Theano and Torch. There is also a PR on the config to have caffe by default as well (if you need it, just checkout the branch and run the install script as soon as the instance is up).

Upvotes: 0

Related Questions