IPython uses a different $PATH environment than the shell

Question

I had some problems with calling pip within the IPython REPL, and after a while I noticed that IPython doesn't use the same $PATH environment as my shell.

$ echo $PATH
/Users/jimmy/dev/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/texbin

$ ipython
In [1]: !echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/texbin:/Users/jimmy/dev/anaconda/bin

It does seem to scramble the path order, and I'm not quite sure what's wrong.

I'm using ZSH installed through oh-my-zsh as shell, if it is of help.

This is what sys.path() returns:

['',
 '/Users/jimmy/dev/anaconda/bin',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/sparsesvd-0.2.2-py2.7-macosx-10.5-x86_64.egg',
 '/Users/jimmy/dev/projects/pyresult',
 '/Users/jimmy/dev/work/gavagai/userdata',
 '/Users/jimmy/dev/work/gavagai/gavapi',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python27.zip',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/plat-darwin',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/plat-mac',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/plat-mac/lib-scriptpackages',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/lib-tk',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/lib-old',
 '/Users/jimmy/dev/anaconda/python.app/Contents/lib/python2.7/lib-dynload',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/runipy-0.1.0-py2.7.egg',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/setuptools-3.6-py2.7.egg',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/PIL',
 '/Users/jimmy/dev/anaconda/lib/python2.7/site-packages/IPython/extensions',
 '/Users/jimmy/.ipython']

And this is what os.environ['PATH'] returns:

'/Users/jimmy/dev/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/texbin'

Both seem to place it in the correct order.

Grateful for all help.

Answer 1

To summarize and complement holdenweb's helpful answer, particularly with respect to macOS:

• A subshell started from IPython with ! is a non-interactive non-login instance of the user's default shell - even if IPython was started from a different shell.
• Behind the scenes, the subshell is started with path/to/default/shell -c ...
• To see specifics, run !ps -p $$ && :

• echo $SHELL always tells you the default shell - even when run from a different shell.

• Initialization files sourced in non-interactive non-login shells:

  • zsh:: /etc/zshenv and ~/.zshenv
  • bash: a script pointed to in the $BASH_ENV variable, if defined.

• As pointed out, shells load different/additional initialization files depending on whether:

  • the shell is a login shell or not
  • an interactive shell or not

  Note that a login shell can be interactive or not, and an interactive shell can be a login shell or not.

Thus, in the case at hand, potentially two additional initialization files were loaded in the interactive shell, explaining the difference in behavior between the interactive shell and the subshell created by IPython:

• ~/.zprofile - if the shell is a login shell - which will be the case if zsh is the default shell (on macOS, all interactive instances of the default shell created in a terminal such as Terminal.app are login shells).
• ~/.zshrc

---

Finally, on a related note, note that on macOS the default $PATH for NON-shell processes is:

/usr/bin:/bin:/usr/sbin:/sbin  # Note the absence of /usr/local/bin.

and only shells add /usr/local/bin (by default; extensible) to that, via system-wide initialization files (that call /usr/libexec/path_helper):

• zsh:
  • /etc/zshenv
  • Note: takes effect for ALL zsh instances.
• bash (also when invoked as sh), ksh:
  • /etc/profile
  • Note: takes effect for LOGIN shells only.

Upshot:

• Non-shell apps see only the default $PATH (notably, without /usr/local/bin and other additions made via shell initialization files).
• bash/sh and ksh non-interactive non-login shells not launched from a login shell also see only the default $PATH. This happens, for instance, when GUI command-line launchers such as Alfred create shell instances.
• zsh is not affected, because /etc/zshenv is read by every zsh instance.

Answer 2

First of all, sys.path has little to do with this; it's just the list of locations in which the Python interpreter can look when importing modules, and does not determine where executable programs are found by the shell. Some elements of the PATH are seen in that list, though, because the interpreter uses the path of its executable to build certain entries on sys.path.

os.environ['PATH'] is, unsurprisingly, the same as the $PATH variable from the environment in which IPython is running. !echo $PATH prints out the $PATH variable from the subshell started by IPython to execute the shell escape (!).

One possible cause is that IPython is executing the shell commands using the OS standard shell which has not been tailored to set its path in the same way that your zsh has. You can confirm this by executing the IPython command !echo $SHELL. Since you confirmed this was not the case then the difference can be accounted for by the differences between a login shell and an interactive shell.

I don't know of a configuration item that will tell IPython to use another shell, but it's possible there is one. As a workaround, just make sure the other shell has a correctly configured path, or that interactive shells also see the environment you require.

See @mklement0's extremely authoritative answer for the whole, gory, messy details.