How to implement mmap for PCI bar memory?

Question

I have a SRAM memory addressable at PCI bar 1. I would like to know how to correctly write mmap function in driver so that my user process can mmap the sram memory and read/write into it.

I wrote the following probe and mmap functions in driver to do the same:

The probe function:

static int probe(struct pci_dev * pcidev, const struct pci_device_id * id)
{
    int r,i;
    unsigned long start,end;
    int sram;

    r = pci_enable_device(pcidev);
    if(r < 0)
    {
        printk(KERN_ALERT DEVICE_NAME ": Enable of my_pci device fails %d\n",r); 
        return -1;
    }
    pci_set_master(pcidev);

    start = pci_resource_start(pcidev, 1);
    end = pci_resource_end(pcidev, 1);
    size = end - start + 1;

    sram_raw_base_address = start;
    sram_base_address = (unsigned long)ioremap_nocache(start,size);
    sram_size = size;
}

The mmap function is:

static int my_mmap(struct file *file, struct vm_area_struct *vma)
{   
    size_t size = vma->vm_end - vma->vm_start;
    unsigned long ioaddress, iosize;

    if(vma->vm_end < vma->vm_start)
    {
        return -EAGAIN;
    }

    /* mapping sram. */
    ioaddress = sram_base_address;
    iosize = sram_size;

    /* simple check to prevent mapping past io area. */
    if( (size > iosize) ||
        ((vma->vm_pgoff << PAGE_SHIFT) > iosize) ||
        ( ((vma->vm_pgoff << PAGE_SHIFT)+size) > iosize) )
    {
        return -EINVAL;
    }

    if (remap_pfn_range(vma,vma->vm_start,vmalloc_to_pfn(ioaddress+(vma->vm_pgoff << PAGE_SHIFT)),
            size,vma->vm_page_prot)) {
    return -EAGAIN;
    }

    vma->vm_ops = &vmops;
    vmops_open(vma);

    return 0;
}

Whenever I try to mmap the SRAM memory in my application, I get the following error in /var/log/syslog:

Jul 26 13:21:12 manik kernel: [86635.262485] my_pci my_mmap - called
Jul 26 13:21:12 manik kernel: [86635.262524] BUG: unable to handle kernel  paging request at 02530000
Jul 26 13:21:12 manik kernel: [86635.262549] IP: [<c11ab31d>] vmalloc_to_pfn+0xd/0x30
Jul 26 13:21:12 manik kernel: [86635.262569] *pdpt = 00000000050f8001 *pde = 0000000000000000
Jul 26 13:21:12 manik kernel: [86635.262584] Oops: 0000 [#1] SMP
Jul 26 13:21:12 manik kernel: [86635.262595] Modules linked in: qcomdl(POE) gdapdl(POE) gammadl(POE) aspdl(POE) ccce(POE) scard(POE) kmikohn(OE) fmon(OE) khcd(OE) kcid(OE) kspi(POE) kpwrfail(OE) mk7i_pci(OE) snd_usb_audio input_leds hid_multitouch joydev snd_usbmidi_lib intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm i2c_ocores snd_soc_rt5640 snd_hda_codec_hdmi irqbypass crc32_pclmul snd_soc_rl6231 snd_hda_intel snd_hda_codec lpc_ich ie31200_edac edac_core snd_hda_core shpchp snd_hwdep snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine 8250_fintek snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd snd_soc_sst_acpi soundcore 8250_dw dw_dmac soc_button_array dw_dmac_core elan_i2c spi_pxa2xx_platform i2c_designware_platform i2c_designware_core mac_hid parport_pc ppdev lp parport autofs4 nouveau mxm_wmi i915 wmi i2c_algo_bit ttm ahci libahci drm_kms_helper syscopyarea e1000e sysfillrect sysimgblt fb_sys_fops ptp drm pps_core sdhci_acpi video i2c_hid sdhci fjes hid_generic usbhid hid [last unloaded: mk7i_pci]
Jul 26 13:21:12 manik kernel: [86635.262938] CPU: 0 PID: 14025 Comm: test_sram Tainted: P        W IOE   4.4.0-31-generic #50
Jul 26 13:21:12 manik kernel: [86635.262950] Hardware name: ADLINK Technology Inc. Express-HL./SHARKBAY, BIOS 1.14 01/01/2013
Jul 26 13:21:12 manik kernel: [86635.262962] task: d0c05a00 ti: ca928000 task.ti: ca928000
Jul 26 13:21:12 manik kernel: [86635.262972] EIP: 0060:[<c11ab31d>] EFLAGS: 00010246 CPU: 0
Jul 26 13:21:12 manik kernel: [86635.262983] EIP is at vmalloc_to_pfn+0xd/0x30
Jul 26 13:21:12 manik kernel: [86635.262991] EAX: 02530000 EBX: c5617138 ECX: ee9d9000 EDX: fff1b000
Jul 26 13:21:12 manik kernel: [86635.263001] ESI: 02000000 EDI: f3200000 EBP: ca929e70 ESP: ca929e70
Jul 26 13:21:12 manik kernel: [86635.263010]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Jul 26 13:21:12 manik kernel: [86635.263019] CR0: 80050033 CR2: 02530000 CR3: 2dc02500 CR4: 001406f0
Jul 26 13:21:12 manik kernel: [86635.263028] Stack:
Jul 26 13:21:12 manik kernel: [86635.263034]  ca929e90 f0c566af f0c58260 024080c0 c5617138 c56171b4 b5321000 c56171b0
Jul 26 13:21:12 manik kernel: [86635.263063]  ca929ee4 c11a64f8 b7321000 000000fb 00000000 c6eac0c0 00000000 00000000
Jul 26 13:21:12 manik kernel: [86635.263091]  00000000 00000008 00000000 00000000 c56171b4 00002000 c6eac0c0 c5617138
Jul 26 13:21:12 manik kernel: [86635.263120] Call Trace:
Jul 26 13:21:12 manik kernel: [86635.263132]  [<f0c566af>] my_mmap+0x6f/0xf0 [mk7i_pci]
Jul 26 13:21:12 manik kernel: [86635.263145]  [<c11a64f8>] mmap_region+0x388/0x590
Jul 26 13:21:12 manik kernel: [86635.263155]  [<c11a6956>] do_mmap+0x256/0x360
Jul 26 13:21:12 manik kernel: [86635.263167]  [<c118d19a>] vm_mmap_pgoff+0x9a/0xc0
Jul 26 13:21:12 manik kernel: [86635.263177]  [<c11a4c54>] SyS_mmap_pgoff+0x194/0x230
Jul 26 13:21:12 manik kernel: [86635.263190]  [<c100393d>] do_fast_syscall_32+0x8d/0x150
Jul 26 13:21:12 manik kernel: [86635.263202]  [<c17b229c>] sysenter_past_esp+0x3d/0x61
Jul 26 13:21:12 manik kernel: [86635.263210] Code: 8b 80 c0 20 d2 c1 83 e0 fc 8d 1c d0 eb bb 31 db 89 d8 5b 5e 5f 5d c3 8d b4 26 00 00 00 00 55 89 e5 3e 8d 74 26 00 e8 d3 fe ff ff <8b> 10 5d c1 ea 19 c1 e2 04 8b 92 c0 20 d2 c1 83 e2 fc 29 d0 c1
Jul 26 13:21:12 manik kernel: [86635.263388] EIP: [<c11ab31d>] vmalloc_to_pfn+0xd/0x30 SS:ESP 0068:ca929e70
Jul 26 13:21:12 manik kernel: [86635.263406] CR2: 0000000002530000
Jul 26 13:21:12 manik kernel: [86635.269070] ---[ end trace f85e2d73d203ae00 ]---

clearly, the mmap function of the driver is failing in vmalloc_to_pfn call. I do not understand why.

I am using ioremap_nocache in probe function to create non-cacheable virtual addresses as I want the writes/read happening at PCI bar memory regions to be read/write at the memory hardware and not cpu cache. I hope I am doing it right.

Now, for mmap call I am using vmalloc_to_pfn to get the physical page frame number.

Is there something that I am doing wrong here ?

Answer 1

Do you need to write any code for this at all? Usermode can open the BAR of your device in sysfs (/sys/bus/pci/...) and mmap that. It works without any effort on your side.