Storing commands in the video memory vs. accessing them via PCIe

Question

I'm currently reading part 2 of the "A trip through the Graphics Pipeline" blog series by Fabian "ryg" Giesen.

In this particular part, he talks about one interesting point. The commands that the GPU works on can either be read from the video memory which would reside on the GPU, or from the CPU side via the PCI Express bus.

I implicitly understood a few pros and cons. However, I'm quite hazy on them. I cannot say that I perfectly understand the trade-offs. Anyway, I'm going to try to present my thoughts coherently.

Storing the Data in the Video Memory:

• Definitely faster.
• However, it has a memory overhead (not sure about bandwidth).

Reading from the CPU:

• Could be better in situations where storing the data in the video memory is a waste (maybe because it would never be used again?).
• I'd think that storing data in the video memory would be useful when that data is reused since it would save bandwidth on the PCIe bus.
• But this communication will be slower than directly reading from the video memory.

I have a feeling that there are many more intricacies and trade-offs between the two. Also, I'm not entirely sure that what I've said above is true. I'd like for someone to:

1. Illustrate some important trade-offs and possibly other intricacies just to better my understanding.
2. Verify whether whatever I've said above is true or not! (I'm not quite clear on this yet.)