CODEWITHSUNDEEP
linuxlinux-device-driverxilinxzynqpetalinux
ngc6027
ngc6027

Reputation: 101

How can I force a cache flush for a process from a Linux device driver?

I'm working on a research project that requires me to perform a memory capture from custom hardware. I am working with a Zedboard SoC (dual-core ARM Cortex-A9 with FPGA fabric attached). I have designed a device driver that allows me to perform virtual memory captures and physical memory captures (using an AXI4-Lite peripheral that controls the Xilinx AXI DMA IP).

My goal is to capture all mapped pages, so I check /proc/pid/maps for mapped regions, then obtain PFNs from /proc/pid/pagemaps, pass the physical addresses into my device driver, and then pass them to my custom hardware (which invokes the Xilinx AXI DMA to obtain the contents from physical memory).

NOTE: I am using Xilinx's PetaLinux distribution, which is built on Linux version 4.14.

My device driver implements the following procedure through a series of IOCTL calls:

  1. Stop the target process.
  2. Perform virtual memory capture (using the access_process_vm() function).
  3. Flush the cache (using the flush_user_range() function).
  4. Perform physical memory capture.
  5. Resume the target process.

What I'm noticing, however, is that the virtual memory capture and the physical memory capture differ in the [heap] section (which is the first section that extends past one page). The first page matches, but none of the other pages are even close. The [stack] section does not match at all. I should note that for the first two memory sections, .text and .rodata, the captures match exactly. The conclusion for now is that data that does not change during runtime matches between virtual and physical captures while data that does change during runtime does not match.

So this leaves me wondering: am I using the correct function to ensure coherency between the cache and the RAM? If not, what is the proper function to use to force a cache flush to RAM? It is necessary that the data in RAM is up-to-date to the point when the target process is stopped because I cannot access the caches from the custom hardware.

Edit 1: In regards to this question being marked as a possible duplicate of this question, I am using a function from the accepted answer to initiate a cache flush. However, from my perspective, it doesn't seem to be working as the physical memory does not match the virtual memory as I would expect if a cache flush were occurring.

Upvotes: 4

Views: 2013

Answers (1)

ngc6027
ngc6027

Reputation: 101

For anyone coming across this question in the future, the problem was not what I thought it was. The flush_user_range() function I mentioned is the correct function to use to push pages back to main memory from cache.

However, what I didn't think about at the time was that pages that are virtually contiguous are not necessarily (and are very often not) physically contiguous. In my kernel code, I passed the length of the mapped region to my hardware, which requested that length of data from the AXI DMA. What I should have done was a virtual-to-physical translation to get the physical address of each page in each region, then requested one page length of data from main memory, repeating that process for each page in each mapped region.

I realize this is a pretty specific problem that likely won't help anyone else doing the same thing I was doing, but hopefully the learned lesson can help someone in the future: Linux allocates pages in physical memory (that are usually 4kB in size, though you shouldn't assume that's the case), and a collection of physical pages is contained in one mapped region. If you're working on any code that requires you to check physical memory, be sure to be wary of where your data may cross physical page boundaries and act accordingly.

Upvotes: 4

Related Questions