Trying to install keras & tensorflow on an Amazon AWS Instance

Question

I am running into some problems when trying to install keras and tensorflow on a Ubuntu EC2 Instance.

I load in the library and try to install keras.

library(keras)
library(tensorflow)
install_keras(tensorflow = "gpu")

I obtain the following error:

Error: could not find a Python environment for /usr/bin/python

I have also ran the following along with a few other terminal commands to try and install conda and python etc.

sudo apt install git libpython3.7-dev

I have also followed the keras.io website to try and install it. I have a folder on the server called /usr/bin/python3.6: however when I try to access it I get an error Directory /usr/bin/python3.6: no such file or directory. I have two user accounts at the home directory on the server (along with two folders by the same name) ubuntu and myUsername_folder.

In the /home/myUsername_folder I have the .rstudio files. (/home/myUsername_folder/.rstudio) and some datasets. In the home/ubuntu folder I have R installed /home/ubuntu/R/x86_64-pc-linux-gnu-library/3.6. In the / root directory I have folders such as usr, var, lib64, tmp, bin etc.

How I can tell R where Python is located?

EDIT:

I restarted RStudio and ran:

> library(keras)
> library(tensorflow)
> install_keras(tensorflow = "gpu")
Using virtual environment '~/.virtualenvs/r-reticulate' ...
/home/myUser/.virtualenvs/r-reticulate/bin/python: No module named pip
Error in strsplit(output, "\\s+")[[1]] : subscript out of bounds
In addition: Warning message:
In system2(python, c("-m", "pip", "--version"), stdout = TRUE) :
  running command ''/home/myUser/.virtualenvs/r-reticulate/bin/python' -m pip --version' had status 1

Trying to install pip using:

sudo apt-get install python3-pip

Reading package lists... Done
Building dependency tree
Reading state information... Done
python3-pip is already the newest version (9.0.1-2.3~ubuntu1.18.04.1).
0 upgraded, 0 newly installed, 0 to remove and 39 not upgraded.

I ran the following to install Anaconda:

wget https://repo.anaconda.com/archive/Anaconda3-2019.10-Linux-x86_64.sh bash Anaconda3-2019.10-Linux-x86_64.sh

Then ran:

> install_keras(method = c("conda"),
+               conda = "auto", version = "default", tensorflow = "gpu")
ERROR: The install method you used for conda--probably either `pip install conda`
or `easy_install conda`--is not compatible with using conda as an application.
If your intention is to install conda as a standalone application, currently
supported install methods include the Anaconda installer and the miniconda
installer.  You can download the miniconda installer from
https://conda.io/miniconda.html.

Error: Error 1 occurred creating conda environment r-reticulate

EDIT: I tried:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh

Then in RStudio

> install_keras(method = c("conda"),
+               conda = "/home/myUser/miniconda3/bin/",
+               version = "default", tensorflow = "gpu")
sh: 1: /home/myuser/miniconda3/bin/: Permission denied
Error: Error 126 occurred creating conda environment r-reticulate

I change the permissions:

sudo chmod -R 777 /home/myUser/miniconda3/bin/

and get the same error after running install_keras(...) .

EDIT:

I run in RStudio

tf_config()

Installation of TensorFlow not found.

Python environments searched for 'tensorflow' package:
 /usr/bin/python3.6
 /usr/bin/python3.6
 /usr/bin/python2.7
 /home/myuser/anaconda3/bin/python3.7
 /usr/bin/python3.6
 /home/myuser/miniconda3/bin/python3.7

You can install TensorFlow using the install_tensorflow() function.

install_tensorflow()

Error: could not find a Python environment for /usr/bin/python

I also tried use_python("/home/myuser/anaconda3/bin/python3.7")

EDIT

The problems above still persist but I can run the following R code (which "works"):

nn_dat = iris %>% as_tibble %>%
  mutate(sepal_l_feat = scale(Sepal.Length),
         sepal_w_feat = scale(Sepal.Width),
         petal_l_feat = scale(Petal.Length),
         petal_w_feat = scale(Petal.Width),          
         class_num    = as.numeric(Species) - 1, # factor, so = 0, 1, 2
         class_label  = Species) %>%
  select(contains("feat"), class_num, class_label)
nn_dat %>% head(3)

test_f = 0.20
nn_dat = nn_dat %>%
  mutate(partition = sample(c('train','test'), nrow(.), replace = TRUE, prob = c(1 - test_f, test_f)))

x_train = nn_dat %>% filter(partition == 'train') %>% select(contains("feat")) %>% as.matrix
y_train = nn_dat %>% filter(partition == 'train') %>% pull(class_num) %>% to_categorical(3)
x_test  = nn_dat %>% filter(partition == 'test')  %>% select(contains("feat")) %>% as.matrix
y_test  = nn_dat %>% filter(partition == 'test')  %>% pull(class_num) %>% to_categorical(3)

model = keras_model_sequential()
model %>% 
  layer_dense(units = 4, activation = 'relu', input_shape = 4) %>% 
  layer_dense(units = 3, activation = 'softmax')
model %>% summary

model %>% compile(
  loss      = 'categorical_crossentropy',
  optimizer = optimizer_rmsprop(),
  metrics   = c('accuracy')
)

history = model %>% fit(
  x = x_train, y = y_train,
  epochs           = 200,
  batch_size       = 20,
  validation_split = 0
)
plot(history)

Here I get this message:

/home/myuser/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/myuser/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/myuser/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/myuser/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/myuser/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/myuser/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/myuser/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/myuser/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/myuser/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/myuser/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/myuser/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/myuser/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

2020-01-21 21:49:45.344207: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-01-21 21:49:45.384211: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2400050000 Hz
2020-01-21 21:49:45.384567: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55fc4e3c43b0 executing computations on platform Host. Devices:
2020-01-21 21:49:45.384598: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2020-01-21 21:49:45.448620: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.

Then running:

perf = model %>% evaluate(x_test, y_test)
print(perf)

$loss
[1] 0.2995828

$acc
[1] 0.84375

So I am able to use keras & tensor flow but I get the error message still as above.

Answer 1

Make sure conda or virtualenv are properly installed.

Answer 2

Well I have not worked with R before, however, I had similar issues about enviorement when it cames to use such libraries as Tensorflow. I had the same issue when I was intalling Tensorflow and Keras for Deep learning work. In my case, I figured out that the hole problem was due to Tensorflow library itself. I use windows, but this may work at linux as well so what I have done was:

1. Unistalled every Conda, Python, and R (including the adtional libraries such tensorflow) from your computer.
2. I downloaded the las version of "ANACONDA" but only downloaded ANACONDA, nothing else. That is actually all what you need to run out of errors.
3. Proceed to install, and then you will have Anaconda navigator. That includes app and python by default (inside the anaconda navigator) such Jupyter, Spyder and R.
4. Go to the enviorment section, and create a dedicated enviorement in wich you will intall the Tensorflow library. You can type the name in navigator inside the Anaconda navigator, to find the libraries you are serching for.
5. Make sure you installed the hole libraries you need, inside the new enviorement you've just created.
6. Go back to home section, and by making sure that you have selected that "new created enviorement" install the apps you need, that you may have below. Among them R is aviable.

This should work, and I hope it helps you. This way is more sharp and clear that the traditional pip installing process. Note that Tensorflow is not a simple library, and by using this methode, eveything is well organized, by the ANACONDA navigator, with no future errors.