CODEWITHSUNDEEP
python
Eric PASCUAL
Eric PASCUAL

Reputation: 219

Python string concatenation loop

I've been surprised by the results provided by %timeit for these two implementations :

def f1():                                                                       
    s = ''                                                                    
    for i in range(len(values)):                                                
        s += str(values[i][0])                                                
        s += '\t'                                                             
        s += str(values[i][1])                                    
        s += '\r\n'                                                           
    return s

and

def f2():                                                                       
    return ''.join((                                                            
                str(ts) + '\t' + str(v) + '\r\n'                                  
                for ts, v in values                                             
            ))

knowing that values is a list of approx 2400 tuples. f1() is the original code I found in a script written by a colleague more used to C/C++ that to Python at the time he wrote it, and f2 is IMHO the more Pythonic style I would have written for the same processing.

I would have expected that f2 being a lot faster that f1, mainly because f1 uses a lot of concatenations and string re-allocations but it occurs that %timeit gives the same magnitude order for both ones (approx. 18ns), and more surprisingly, giving f2 1ns faster sometimes 1ns.

What could be the explanation for such a result ?

[EDITED Jul 14] fixed f1 for the str override by the local variable with the same name. This error was not present in the profiled code however.

Upvotes: 2

Views: 6272

Answers (3)

TemporalWolf
TemporalWolf

Reputation: 7952

I can be fairly confident your test methodology is invalid, as can be demonstrated by repl.it for Py2.7 and repl.it for Py3. It's the same code, as shown below, but the results vary:

f1 is your f1 function
f2 is your f2 function
f3 is your f2 function using c-style string formatting "%s" % str
f4 is your f2 function using .format()

Results:

Python 2.7.10 (default, Jul 14 2015, 19:46:27)
[GCC 4.8.2] on linux

1.67547893524
1.33767485619
0.72606086731
1.32540607452

There are some differences, but in no case does f1 outperform any of the following methods.

Python 3.6.1 (default, Dec 2015, 13:05:11)
[GCC 4.8.2] on linux

3.0050943629757967
2.016791722999187
0.9476796620001551
1.9396837950043846

In both cases c style strings formatting is more than twice as fast.

The functions used:

def f1():
    s = ''
    for i in range(len(values)):
        s += str(values[i][0])
        s += '\t'
        s += str(values[i][1])
        s += '\r\n'
    return s

def f2():    
    return ''.join((                        
        str(ts) + '\t' + str(v) + '\r\n'                
        for ts, v in values))

def f3():           
    return ''.join((
        "%s\t%s\r\n" % (ts, v)  
        for ts, v in values))

def f4():
    return ''.join((
        "{}\t{}\r\n".format(ts, v)
        for ts, v in values))

Interestingly, by making a small change to your f1 function, we can attain a decent speed up by exploiting the bytecode speedup referenced by danny:

def f1opt():
    s = ''
    for i in range(len(values)):
        s += str(values[i][0]) + '\t' + str(values[i][1]) + '\r\n'
    return s

yields

Python 2.7.10 (default, Jul 14 2015, 19:46:27)
[GCC 4.8.2] on linux

f1()         1.68486714363
f1bytecode() 0.999644994736

Upvotes: 1

Eric PASCUAL
Eric PASCUAL

Reputation: 219

Since the observed results were i bit surprising, I did the same profiling differently, using the following script :

import random
import timeit

data = [(random.randint(0, 100000), random.randint(0, 1000)) for _ in range(0, 2500)]

def f1():
    return ''.join(('{}\t{}\r\n'.format(ts, v) for ts, v in data))

def f2():
    s = ''
    for i in range(len(data)):
        s += str(data[i][0])
        s += '\t'
        s += str(data[i][1])
        s += '\r\n'

    return s


if __name__ == '__main__':
    repeat = 10000

    for f in ['f1', 'f2']:
        t = timeit.timeit(
            '%s()' % f, number=repeat, setup="from __main__ import %s" % f
        )
        print(
            "%s : avg time per loop = %f ms" % (f, t * 1000 / repeat)
        )

The output is now :

    f1 : avg time per loop = 0.779966 ms
    f2 : avg time per loop = 1.144340 ms

Which is more in line with the expected results.

I'll investigate more to understand the differences in behaviour between both tests.

Upvotes: 0

danny
danny

Reputation: 5270

The f2 code is still bound by string concatenation due to

str(ts) + '\t' + str(v) + '\r\n'

The fact it is worse than the original version, also based on string concat, is likely due to the implementation detail mentioned in another question.

If you change the inner concatenations to also use join you would get better performance.

def f2(values):                                                                       
    return '\r\n'.join(
        ('\t'.join([str(ts), str(v)])
      for ts, v in values))

Upvotes: 2

Related Questions