CODEWITHSUNDEEP
gomutex
SomeUser
SomeUser

Reputation: 390

Golang defer fails at times

I am working on a application with multiple routines. The processor receives an ID (string) and performs some operations. The IDs might be duplicated and I don't want multiple routines to process an ID when another routine is processing it.

I use a sync mutex map for it.

type cache struct{
   sync.Mutex
   ids map[string]struct{}
}

func(c *cache) addIfNotPresent(string)bool{
   c.Lock()
   defer c.Unlock()
   if _, ok := c.ids[id]; ok{
       return false
   }
   c.ids[id] = struct{}{}
   return true
}
func(c *cache) delete(string){
   c.Lock()
   defer c.Unlock()
   delete(c.ids, id)
}

My processor has an instance of this map. Now my process looks something like this

func process(string){
   ok := cache.addIfNotPresent(id)
   if !ok{
      return
   }
   defer cache.delete(id)

   ctx, cancel := context.WithTimeout(context.Background(), timeout)
   defer cancel()
   err := doOne(ctx)
   if err {
      return err
   }
   ...
   return nil
}

Using defer so the id gets removed regardless of what happens in processor.

Sometimes (not always) the value does not gets evicted from the map. From the logs/metrics I am certain it was not the error case but the process function was complete and the key was not evicted from the map.

Is there any behavior of mutex or defer I am missing here ?

Upvotes: 0

Views: 369

Answers (2)

Ivaylo Novakov
Ivaylo Novakov

Reputation: 905

The only way I can see this code causing your metrics function (executed after checking the ok in process) to see the value in the map is the following:

  1. thread1 adds the value to the map and starts processing it. It yields the CPU.
  2. thread2 goes into the process function, calls addIfNotPresent and gets false. It yields the CPU before going into if !ok {.
  3. thread1 gets the CPU, finishes its work, removes the value from the map and exits.
  4. thread3 gets the value, checks the map, sees the value is not there anymore, starts processing it. It yields the CPU.
  5. thread2 gets the CPU, checks the ok and sees it's false. At this point it hits your metrics function. Your metrics functions checks the map and sees the value there. Confusion ensues.

A way to verify this is to check the map after the processing is done. If my hypothesis is right you will not see any values still lingering there. If I'm wrong and the values are really not evicted correctly you will still see them there.

Upvotes: 0

eugenioy
eugenioy

Reputation: 12383

Sometimes (not always) the value does not gets evicted from the map.

How and when are you checking that?

The code for the function you pasted is actually removing the key when finished, but after that execution finishes there might be another goroutine processing (thus adding) the same key again.

Upvotes: 1

Related Questions