CODEWITHSUNDEEP
pythondjangosvnwsgipopen
Walty Yeung
Walty Yeung

Reputation: 3582

django wsgi + Popen + svn checkout

For some reason, I am need to checkout a source code folder from a django view, and I use 'Popen' for this.

Everything works fine, and it works perfect when using django runserver.

However, after I deployed the code to apache2 + wsgi, the Popen does not work properly. It always return before the command is actually finished. It does not throw an error either, it just throw out in-complete output, and I checked the checked out folders, they are incomplete too.

The whole svn check out process took about 5-6 seconds, and the standard output is quite large (around 3000 characters).

I know there is a library of pysvn, but it seems difficult to install it on an out-dated ubuntu server.

And basically it is the only thing that I got stuck now.

The code piece I used to call the check out is following:

def run_cmd(argument_list, output_file = None):
    print "arguments", argument_list
    p = subprocess.Popen(argument_list, stdout=subprocess.PIPE)

    content = ""
    while True:
        line = p.stdout.read(50)

        if not line:
            break

        content += line

    if output_file:
        fout = file(output_file, "w")
        fout.write(content)
        fout.close()

    return content


output = run_cmd(["/usr/bin/svn", "--ignore-externals", "co", svn_url, src_folder] )

Here are some information that may be useful:

  1. number of files to check out: around 3000
  2. time required to check out: around 5-6 second (just file based SVN location)
  3. python version: 2.6.4
  4. django version: 1.1.2
  5. mod wsgi version: 3.3

I have been stuck on this for hours, any hint is greatly appreciated!

thanks.

Upvotes: 0

Views: 343

Answers (1)

Walty Yeung
Walty Yeung

Reputation: 3582

ok, after struggled for another 3 hours today. I have finally solved the problem.

Here is what's going on, the wsgi & popen are actually fine, the real problem is some source code files for check out actually has special characters and that break the svn check out process (with the following error)

svn: Can't convert string from 'UTF-8' to native encoding

The wsgi and console have different value for LC_LANG, and that explains the different behaviours between runserver and wsgi.

finally, I solved the problem by modifying the file of '/etc/apache2/envars' and uncommented the following line:

. /etc/default/locale

Note that you have to restart the server by 'apache2ctl stop' and 'apache2ctl start', instead of 'apache2ctl restart', in order to have the setting to be effective.

I actually used pexpect to find out the problem, but reverted back to Popen later because obvious latency problem.

Here is my final code for the run_cmd, and hopefully it could help someone else in the future:

def run_cmd(argument_list, output_file = None):
    print "arguments", argument_list


    #command = " ".join(argument_list)
    #content = pexpect.run(command)
    #if output_file:
        #fout = file(output_file, "w")
        #fout.write(content)
        #fout.close
    #return content

    p = subprocess.Popen(argument_list, bufsize=50, stderr=subprocess.STDOUT, stdout=subprocess.PIPE) #showed error message as well

    content = ""
    while True:
        line = p.stdout.read(50)

        if not line:
            break

        content += line

    #raise Exception(content)   #for debug

    if output_file:
        fout = file(output_file, "w")
        fout.write(content)
        fout.close()

    return content

Upvotes: 1

Related Questions