Python Development in multiple repositories

Question

We are trying to find the best way to approach that problem. Say I work in a Python environment, with pip & setuptools. I work in a normal git flow, or so I hope. So:

1. Move to feature branch in some app, make changes.
2. Move to feature branch in a dependent lib - Develop thing.
3. Point the app, using "-e git+ssh" to the feature branch of the dependent lib.
4. Create a Pull Request.

When this is all done, I want to merge stuff to master, but I can't without making yet another final change to have the app (step 3 above) requirements.txt now point to the main branch of the feature.

Is there any good workflow for "micro services" or multiple dependent source codes in python that we are missing?

Answer 1

We ended up using git dependencies and not devpi. I think when git is used, there is no need to add another package repository as long as pip can use this.

The core issue, where the branch code (because of a second level dependency) is different from the one merged to master is not solved yet, instead we work around that by working to remove that second level dependency.

Answer 2

Python application workflow from development to deployment

It looks like you are in search for developing Python application, using git.

Following description is applicable to any kind of Python based application, not only to Pyramid based web ones.

Requirements

Situation:

• developing Python based solution using Pyramid web framework
• there are multiple python packages, participating in final solution, packages might be dependent.
• some packages come from public pypi, others might be private ones
• source code controlled by git

Expectation:

• proposed working style shall allow:
  • pull requests
  • shall work for situations, where packages are dependent
• make sure, deployments are repeatable

Proposed solution

Concepts:

• even the Pyramid application released as versioned package
• for private pypi use devpi-server incl. volatile and release indexes.
• for package creation, use pbr
• use tox for package unit testing
• test, before you release new package version
• test, before you deploy
• keep deployment configuration separate form application package

Pyramid web app as a package

Pyramid allows creation of applications in form of Python package. In fact, whole initial tutorial (containing 21 stages) is using exactly this approach.

Despite the fact, you can run the application in develop mode, you do not have to do so in production. Running from released package is easy.

Pyramid uses nice .ini configuration files. Keep development.ini in the package repository, as it is integral part for development.

On the other hand, make sure, production .ini files are not present as they should not mix with application and belong to deployment stuff.

To make deployment easier, add into your package a command, which prints to stdout typical deployment configuration. Name the script e.g. myapp_gen_ini.

Write unittests and configure tox.ini to run them.

Keep deployment stuff separate from application

Mixing application code with deployment configurations will make problem at the moment, you will have to install to second instance (as you are likely to change at least one line of your configuration).

In deployment repository:

• keep here requirements.txt, which lists the application package and other packages needed for production. Be sure you specify exact package version at least for your application package.
• keep here production.ini file. If you have more deployments, use one branch per deployment.
• put here tox.ini

tox.ini shall have following content:

[tox]
envlist = py27
# use py34 or others, if your prefer

[testenv]
commands = 
deps =
    -rrequirements.txt

Expected use of deployment respository is:

1. clone it to the server
2. run tox, this will create virtualenv .tox/py27
3. activate the virtualenv by $ source .tox/py27/bin/activate
4. if production.ini does not exist in the repo yet, run command $ myapp_gen_ini > production.ini to generate template for production configuration
5. edit the production.ini as needed.
6. test, it works.
7. commit the production.ini changes to the repository
8. do other stuff needed to deploy the app (configure web server, supervisord etc.)

For setup.py use pbr package

To make package creation simpler, and to keep package versioning related to git repository tags, use pbr. You will end up with setup.py being only 3 lines long and all relevant stuff will be specified in setup.cfg in form of ini file.

Before you build the first time, you have to have some files in git repository, otherwise it will complain. As you use git, this shall be no problem.

To assign new package version, set $ git tag -a 0.2.0 and build it. This will create the package with version 0.2.0.

As a bonus, it will create AUTHORS and ChangeLog based on your commit messages. Keep these files in .gitignore and use them to create AUTHORS.rst and ChangeLog.rst manually (based on autogenerated content).

When you push your commits to another git repository, do not forget to push the tags too.

Use devpi-server as private pypi

devpi-server is excellent private pypi, which will bring you following advantages:

• having private pypi at all
• cached public pypi packages
• faster builds of virtual environments (as it will install from cached packages)
• being able to use pip even without having internet connectivity
• pushing between various types of package indexes: one for development (published version can change here), one for deployment (released version will not change here).
• simple unit test run for anyone having access to it, and it will even collect the results and make them visible via web page.

For described workflow it will contribute as repository of python packages, which can be deployed.

Command to use will be:

• $ devpi upload to upload developed package to the server
• $ devpi test <package_name> to download, install, run unit test, publish test results to devpi-server and clean up temporary installation.
• $ devpi push ... to push released package to proper index on devpi-server or even on public pypi.

Note, that all the time it is easy to have pip command configured to consume packages from selected index on devpi server for $ pip install <package>.

devpi-server is also ready for use in continuous integration testing.

How git fits into this workflow

Described workflow is not bound to particular style of using git.

On the other hand, git can play it's role in following situations:

• commit: commit message will be part of autogenerated ChangeLog
• tag: defines versions (recognized by setup.py based on pbr).

As git is distributed, having multiple repositories, branches etc., devpi-server allows similar distribution as each user can have it's own working index to publish to. Anyway, finally there will be one git repository with master branch to use. In devpi-server will be also one agreed production index.

Summary

Described process is not simple, but the complexity is relevant to complexity of the task.

It is based on tools:

• tox
• devpi-server
• pbr (Python package)
• git

Proposed solution allows:

• managing python packages incl. release management
• unit testing and continuous integration testing
• any style of using git
• deployment and development having clearly defined scopes and interactions.

Your question assumes multiple repositories. Proposed solution allows decoupling multiple repositories by means of well managed package versions, published to devpi-server.