Unable to load an ipython magic extension sparkmagic: module not found

Question

I am new to loading custom magics into ipython. The instructions to load the

sparkmagics

extension include the following steps that have been completed successfully:

pip install sparkmagic
pip show sparkmagic
jupyter-kernelspec install sparkmagic/kernels/sparkkernel
jupyter-kernelspec install sparkmagic/kernels/pysparkkernel
jupyter-kernelspec install sparkmagic/kernels/pyspark3kernel
jupyter-kernelspec install sparkmagic/kernels/sparkrkernel

The command pip show sparkmagic Seems to be healthy:

$pip show sparkmagic
Name: sparkmagic
Version: 0.10.1
Summary: SparkMagic: Spark execution via Livy
Home-page: https://github.com/jupyter-incubator/sparkmagic/sparkmagic
Author: Jupyter Development Team
Author-email: jupyter@googlegroups.org
License: BSD 3-clause
Location: /usr/local/lib/python2.7/site-packages
Requires: mock, requests, tornado, nose, notebook, ipywidgets, ipykernel, pandas, hdijupyterutils, autovizwidget, ipython, numpy

Something that did not work - but listed as optional is ;

$ jupyter serverextension enable --py sparkmagic

Traceback (most recent call last):
  File "/usr/local/bin/jupyter-serverextension", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/jupyter_core/application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/usr/local/lib/python3.6/site-packages/notebook/serverextensions.py", line 300, in start
    super(ServerExtensionApp, self).start()
  File "/usr/local/lib/python3.6/site-packages/jupyter_core/application.py", line 256, in start
    self.subapp.start()
  File "/usr/local/lib/python3.6/site-packages/notebook/serverextensions.py", line 217, in start
    self.toggle_server_extension_python(arg)
  File "/usr/local/lib/python3.6/site-packages/notebook/serverextensions.py", line 206, in toggle_server_extension_python
    m, server_exts = _get_server_extension_metadata(package)
  File "/usr/local/lib/python3.6/site-packages/notebook/serverextensions.py", line 334, in _get_server_extension_metadata
    m = import_item(module)
  File "/usr/local/lib/python3.6/site-packages/traitlets/utils/importstring.py", line 42, in import_item
    return __import__(parts[0])
ModuleNotFoundError: No module named 'sparkmagic'

The error ModuleNotFoundError: No module named 'sparkmagic' seems ominous. Given the show sparkmagic were healthy then why the complaint here? What is missing still to make the sparkmagic module correctly set up ?

In any case I also tried inside ipython and predictably it failed:

%load_ext sparkmagic.magics

Here is the first portion of the (very long) stacktrace. Let me know if the entire trace were required.

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-2b579b518c56> in <module>()
----> 1 get_ipython().magic(u'load_ext sparkmagic.magics')

/usr/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in magic(self, arg_s)
   2161         magic_name, _, magic_arg_s = arg_s.partition(' ')
   2162         magic_name = magic_name.lstrip(prefilter.ESC_MAGIC)
-> 2163         return self.run_line_magic(magic_name, magic_arg_s)
   2164
   2165     #-------------------------------------------------------------------------

/usr/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in run_line_magic(self, magic_name, line)
   2082                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2083             with self.builtin_trap:
-> 2084                 result = fn(*args,**kwargs)
   2085             return result
   2086

<decorator-gen-64> in load_ext(self, module_str)

/usr/local/lib/python2.7/site-packages/IPython/core/magic.pyc in <lambda>(f, *a, **k)
    191     # but it's overkill for just that one bit of state.
    192     def magic_deco(arg):
--> 193         call = lambda f, *a, **k: f(*a, **k)
    194
    195         if callable(arg):

/usr/local/lib/python2.7/site-packages/IPython/core/magics/extension.pyc in load_ext(self, module_str)
     64         if not module_str:
     65             raise UsageError('Missing module name.')
---> 66         res = self.shell.extension_manager.load_extension(module_str)
     67
     68         if res == 'already loaded':

In summary it is likely there were a step typically required for installing/using any magics. Pointers appreciated.

Answer 1

In my case, I had this problem in Ubuntu. The problem was that the system did not know where magic was.

it was in my site-packages, inside my conda environment, so this is what I had to do:

jupyter-kernelspec install `dirname \`which python\``/../lib/python3.7/site-packages/sparkmagic/kernels/pysparkkernel --user

You first need to conda activate your environment so that which python points to the right location.

Answer 2

From what I see in your error and install messages, you installed spark magic for python2.7 and are trying to run it in python 3.6. Its confusing and when I started, I too had the same issue many many times. I would suggest that you check which version of python you are using by running !python --version on the notebook. If its the correct version of python, then install sparkmagic using !pip install sparkmagic and check the install messages if its installing to the correct version of python.

You could also set up an environment, makes life much more easier. More here https://www.youtube.com/watch?v=LjQlmee58hg