How do I make emacs ido not stopping at found a file in current directory?

Question

Example, I have the following files, all of which were opened in emacs recently:

c:\test\a.org, c:\test\b.org

D:\test\a.org, D:\test\b.org

Now I fire up emacs, and c-x c-f into file finding, and I type a.or, wait a bit and ido will find both c:\test\a.org and D:\test\a.org.

However, now that I have c:\test\a.org open, if I c-x c-f again, and now type b.or, only c:\test\b.org will be found by ido, and ido won't bother looking in D:\test.

How do I make ido look in other recently opened folders, even if it found a match in current folder?