Linux Kernel wait_for_completion_timeout not wakeup by complete

Question

I am working on a strange issue with the i2c-omap driver. I am not sure if the problem happens at other time or not, but it happens around 5% of the time I tried to power off the system. During system power off, I write to some registers in the PMIC via I2C. In i2c-omap.c, I can see that the calling thread is waiting on wait_for_completion_timeout with a timeout value set to 1 second. And I can see the IRQ called "complete" (I added printk AFTER "complete"). However, after "complete" gets called, the wait_for_completion_timeout did not return. Instead, it takes up to 5 MINUTES before it returns. And the return value of wait_for_completion_timeout is positive indicating that there is no timeout. And the whole I2C transaction was successful.

In the meantime, I can see printk messages from other drivers. And the serial console still works. It is on Android, and if I use "top" I can see system_server is taking about 95% of the CPU. Killing system_server can make the wait_for_completion_timeout return immediately.

So my question is what could a user space app (system_server) do to make a kernel "wait_for_completion_timeout" not being wake up?

Thanks!

Answer 1

Well, I kind of figured it out. In the CFS scheduling, in enqueue_entity, it does "vruntime += min_vruntime" in some condition, and in dequeue_entity it does the opposite under some condition. However, those are not always executed in pair. So under some unknown condition, when min_vruntime is pretty big, the vruntime can get pretty big, so the task would be put to the right side of the rbtree and not get scheduled for a long time. I am not sure what is the best way to fix this from the root cause, what I did is a hack in enqueue_entity, if I found vruntime>min_vruntime and the function is called for WAKEUP, I always set vruntime=min_vruntime, thus the task would be put to the relatively left side of the rbtree. The kernel version I am using is 2.6.37 Anyone has a suggestion on how this should be fixed in a better way?

Answer 2

wait_for_completion_timeout only guarantees that the thread waiting on a condition would become "runnable" when either (i) completion happens or (ii) timeout expires.
After that it's the job of scheduler to schedule that thread and change it's state from "runnable" to "running". The thread itself(or the completion framework) is not responsible to make a thread runnable, that's the job of scheduler.
As you have pointed out, system_server is consuming 95% of cpu and therefore making it hard for the completion thread to get scheduled. That explains why the thread is not getting scheduled.