CODEWITHSUNDEEP
pythonmultithreading
turbonerdo
turbonerdo

Reputation: 77

How to use threads in python

I want to use it for reading values in 17770 files and to add all of them at the end to one dictionary object. I have a machine with 8 cores.

This is the code

def run_item_preprocess ():
    scores = {};
    for i in range(1,17771):
        filename1 = "x_" + str(i) + ".txt";
        lines1 = open(filename1).readlines();
        Item1 = {};
        for line in lines1:
            tokens = line.split(',');
            Item1[int(tokens[1])] = int(tokens[2]);
        for j in range(1,17771):
            if j == i:
                continue;
            filename2 = "x_" + str(i) + ".txt";
            lines2 = open(filename2).readlines();
            Item2 = {};
            for line in lines2:
                tokens = line.split(',');
                u = int(tokens[1]);
                r = int(tokens[2]);
                if u in Item1:
                    Item2[u] = r;
            if i not in scores:
                scores[i] = {};
            scores[i]= (s(Item1,Item2),j);

Upvotes: 0

Views: 1261

Answers (3)

Tobu
Tobu

Reputation: 25436

Here is the wonderful multiprocessing module. It lets you parallelise code, using processes not threads. This will use all cores.

An important difference is that processes don't share memory; a queue will help with the reduce step.

Upvotes: 4

Dan Breslau
Dan Breslau

Reputation: 11522

How do you think that using threads would help with this?

Although Python supports threading, the standard implementation (CPython) executes only one thread at a time. Hence it's hard to see how this would make the process run faster, even on multiple cores.

(If you're using JPython or IronPython, though, this restriction doesn't apply.)

Upvotes: 0

DGH
DGH

Reputation: 11549

Here's some good parts of the python library reference to start with.

http://docs.python.org/py3k/library/threading.html

http://docs.python.org/py3k/library/_thread.html

As for how to use threads effectively, I recommend you google 'python thread tutorial' or something like that.

Upvotes: 1

Related Questions