Navigating a big Python codebase faster

Question

As programmers we read more than we write. I've started working at a company that uses a couple of "big" Python packages; packages or package-families that have a high KLOC. Case in point: Zope.

My problem is that I have trouble navigating this codebase fast/easily. My current strategy is

• I start reading a module I need to change/understand
• I hit an import which I need to know more of
• I find out where the source code for that import is by placing a Python debug (pdb) statement after the imports and echoing the module, which tells me it's source file
• I navigate to it, in shell or the Vim file explorer.
• most of the time the module itself imports more modules and before I know it I've got 10KLOC "on my plate"

Alternatively:

• I see a method/class I need to know more of
• I do a search (ack-grep) for the definition of that method/class across the whole codebase (which can be a pain because the codebase is partly in ~/.buildout-eggs)
• I find one or more pieces of code that define that method/class
• I have to deduce which one of them is the one I need to read

This costs a lot of time, which is understandable for a big codebase. But I get the feeling that navigating a large and unknown Python codebase is a common enough problem.

So I'm looking for technical tools or strategic solutions for this problem. ...

I just can't imagine hardcore Python programmers using the strategies outlined above.

Answer 1

Try red pill: https://github.com/klen/python-mode

Answer 2

on Vim, I like NERDTree (a file browser) and taglist.vim (source code browser --> http://www.vim.org/scripts/script.php?script_id=273)

also in Vim, you can use CTRL-] to jump to a definition (:h CTRL-]):

1. download exuberant ctags http://ctags.sourceforge.net/
2. follow the install directions and put it somewhere on your PATH
3. from the 'root' directory of your source code, make a tags file from the shell: "ctags -R"
4. (make sure you have :set noautochdir, and make sure :pwd is the root directory from step 3)
5. go into Vim, cursor over some function or class name, hit CTRL-]

by default, if there's multiple matches for the tag, it shows you everywhere it was imported, and where it was declared

if the tag only has one match, it immediately jumps to it

...then use Ctrl+O and Ctrl+I to move back and forth from where you were

(repeat above steps for the source code of particular libraries you use, i usually keep a separate Vim window open to study stuff)

Answer 3

I use ipython's ?? command

You just need to figure out how to import the things you want to look for, then add ?? to the end of the module or class or function or method name to view their source code. And the command completion helps on figuring out long names as well.