CODEWITHSUNDEEP
python
user741592
user741592

Reputation: 925

Using set to avoid duplicate processing

I have a part in a code like below where file name is supplied to the loop iteratively. I want that no two file names with same name should get processed ( to avoid duplicate processing) so I used the approach of "set" as above.

However this does not seem to work as expected. I get an empty processed_set and logic is not executed as expected.

else:
    create_folder(filename)
    processed_set=set()

    if xyz ==1:
        if filename not in processed_set:
          createdata(name)
          processed_set.add(filename)
        else:
          avoid_double_process(name)

Upvotes: 0

Views: 199

Answers (3)

AI Generated Response
AI Generated Response

Reputation: 8845

From what I can infer from the code sample and guess based on function names, what you want to do is to avoid running the code if filename has already been processed. You would do it this way:

processed_set = set() #initialize set outside of loop
for filename in filenames: #loop over some collection of filenames
    if filename not in processed_set: #check for non-membership
        processed_set.add(filename) #add to set since filename wasn't in the set
        create_folder(filename) #repositioned based on implied semantics of the name
        createdata(filename)

Alternatively, if createdata and create_folder are both functions you don't want to run multiple times for the same filename, you could create a filtering decorator. If you actually care about the return value, you would want to use a memoizing decorator 

def run_once(f):
    f.processed = set()
    def wrapper(filename):
        if filename not in f.processed:
        f.processed.add(filename)
        f(filename)
    return wrapper

then include @run_once on the line prior to your function definitions for the functions you only want to run once.

Upvotes: 1

fenmarel
fenmarel

Reputation: 363

A bit more of your code might be helpful. Not sure if this is the case, but your if conditional is in the same block as where you initialize

processed_set = set()

So basically if xyz != 1 then you are left with an empty set and nothing gets added to the set. If you have a loop set up that hits the first else statement multiple times, you are clearing the contents of your set each time.

Again, I am not really sure of how your code flows since it is a bit obscure at the moment. Add more of the surrounding code and it will be a bit easier for people to help you out.

Upvotes: 0

dirk
dirk

Reputation: 101

Why don't you build your set first, and then process the files in the set afterwards? the set will not add the same element if it is already present;

>>> myset  = { element for element in ['abc', 'def', 'ghi', 'def'] }
>>> myset
set(['abc', 'ghi', 'def'])

Upvotes: 0

Related Questions