CODEWITHSUNDEEP
python
Jelmer
Jelmer

Reputation: 351

Shuffle string data in python

I have a column with 10 million strings. The characters in the strings need to be rearranged in a certain way.

Original string: AAA01188P001

Shuffled string: 188A1A0AP001

Right now I have a for loop running that takes each string and repositions every letter, but this takes hours to completed. Is there a quicker way to achieve this result?

This is the for loop.

for i in range(0, len(OrderProduct)):
    s = list(OrderProduct['OrderProductId'][i])
    a = s[1]
    s[1] = s[7]
    s[7] = a 
    a = s[3]
    s[3] = s[6]
    s[6] = a 
    a = s[2]
    s[2] = s[3]
    s[3] = a 
    a = s[5]
    s[5] = s[0]
    s[0] = a 
    OrderProduct['OrderProductId'][i] = ''.join(s)

Upvotes: 0

Views: 1454

Answers (3)

J. Ternent
J. Ternent

Reputation: 116

Can you just reconstruct the string with slices if that logic is consistent?

s = OrderProduct['OrderProductId'][i]
new_s = s[5]+s[7]+s[1:2]+s[6]+s[4]+s[0]+s[3]+s[1]

or as a format string:

new_s = '{}{}{}{}{}{}{}'.format(s[5],s[7]...)

Edit : +1 for Dave's suggestion of ''.join() the list vs. concatenation.

Upvotes: 2

Alain T.
Alain T.

Reputation: 42129

I made a few performance tests using different methods:

Here are the results I got for 1000000 shuffles:

188A1AA0P001 usefString 0.518183742
188A1AA0P001 useMap     1.415851829
188A1AA0P001 useConcat  0.5654986979999999
188A1AA0P001 useFormat  0.800639699
188A1AA0P001 useJoin    0.5488918539999998

based on this, a format string with hard coded substrings seems to be the fastest.

Here is the code I used to test:

def usefString(s): return f"{s[5:8]}{s[0]}{s[4]}{s[1:4]}{s[8:]}"

posMap = [5,6,7,0,4,1,2,3,8,9,10,11]
def useMap(s): return "".join(map(lambda i:s[i], posMap))

def useConcat(s): return s[5:8]+s[0]+s[4]+s[1:4]+s[8:]

def useFormat(s): return '{}{}{}{}{}'.format(s[5:8],s[0],s[4],s[1:4],s[8:])

def useJoin(s): return "".join([s[5:8],s[0],s[4],s[1:4],s[8:]])

from timeit import timeit
count = 1000000
s = "AAA01188P001"

t = timeit(lambda:usefString(s),number=count)
print(usefString(s),"usefString",t)

t = timeit(lambda:useMap(s),number=count)
print(useMap(s),"useMap",t)

t = timeit(lambda:useConcat(s),number=count)
print(useConcat(s),"useConcat",t)

t = timeit(lambda:useFormat(s),number=count)
print(useFormat(s),"useFormat",t)

t = timeit(lambda:useJoin(s),number=count)
print(useJoin(s),"useJoin",t)

Performance: (added by @jezrael)

N = 1000000
OrderProduct = pd.DataFrame({'OrderProductId':['AAA01188P001'] * N})

In [331]: %timeit [f'{s[5:8]}{s[0]}{s[4]}{s[1:4]}{s[8:]}' for s in OrderProduct['OrderProductId']]
527 ms ± 16.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [332]: %timeit [s[5:8]+s[0]+s[4]+s[1:4]+s[8:] for s in OrderProduct['OrderProductId']]
610 ms ± 18.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [333]: %timeit ['{}{}{}{}{}'.format(s[5:8],s[0],s[4],s[1:4],s[8:]) for s in OrderProduct['OrderProductId']]
954 ms ± 76.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [334]: %timeit ["".join([s[5:8],s[0],s[4],s[1:4],s[8:]]) for s in OrderProduct['OrderProductId']]
594 ms ± 10.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Upvotes: 5

Amir
Amir

Reputation: 1905

If you just want to shuffle the strings (no particular logic), you can do that in a several ways:

Using string_utils:

import string_utils
print string_utils.shuffle("random_string")

Using built-in methods:

import random
str_var = list("shuffle_this_string")
random.shuffle(str_var)
print ''.join(str_var)

Using numpy:

import numpy
str_var = list("shuffle_this_string")
numpy.random.shuffle(str_var)
print ''.join(str_var)

But if you need to do so with a certain logic (e.g. put each element in a specific position), you can do this:

s = 'some_string'
s = ''.join([list(s)[i] for i in [1,6,2,7,9,4,0,8,5,10,3]])
print(s)

Output:

otmrn_sisge

If this is still taking too long, you can use multiprocessing. Like this:

from multiprocessing import Pool
p = Pool(4) # 4 is the number of workers. usually is set to the number of CPU cores

def shuffle_str(s):
    # do shuffling here, and return


list_of_strings = [...]
list_of_results = p.map(shuffle_str, list_of_strings)

Upvotes: 1

Related Questions