6

I have a Terasic-SoCKIT(fpga & arm cortex a9) and I have Linux running on the HPS. I'm trying to access the memory mapped I/O, wrote a simple character driver with functions "request_mem_region" and "ioremap".

The memory mapped IO is an AXI bus, using which I can transmit data to the FPGA. I see that each write is taking almost 6us and for my application I need it to be less than 1us. Also, the driver stops writing to the mapped IO after a few writes(don't see the data being changed in the fpga; is the buffer in the driver getting full??).

The question is am I missing something or because the writes are happening from the virtual address to the physical address it cannot be anymore faster? If the writing from virtual address is slowing down, is there a way to speed it up? I know, that ARM has a DMAC but I haven't explored it yet.

Thank you, Karthik

I'm sorry, I missed to tell that I was measuring the time in the user space code. Later, I checked the time it took to write in the driver and it was in nanoseconds. So, I figured most of the time taken was for the write from the userspace to kernel.

So, I did some further reading and understood that ioremap() maps the physical address to kernel virtual address and remap_pfn_range() maps the physical address/IO memory to the user virtual space (this is what I need; write to IO memory from user space). And I used the simple mmap example - http://web.cecs.pdx.edu/~jrb/ui/linux/examples.dir/simple/simple.c as the kernel driver. The following code is my userspace code:

    using namespace std;
    #include <iostream>
    #include <sys/mman.h>
    #include <sys/types.h>
    #include <sys/stat.h>
    #include <sys/mman.h>
    #include <fcntl.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <unistd.h>
    #include <stdint.h>
    #include <ctime>


    #define PAGE_SIZE 4096
    #define HPS2FPGA_BRIDGE_BASE 0xc0000000
    #define BLINK_OFFSET 0x0

    volatile unsigned char *blink_mem;
    void *bridge_map;

    int main()
    {
        int fd, ret = EXIT_FAILURE;
        unsigned int i;
        unsigned char value;
        int dummy;
        off_t blink_base = HPS2FPGA_BRIDGE_BASE;
        clock_t start, stop;
        double duration;

        /* open the memory device file */
        fd = open("/dev/HPS2FPGA", O_RDWR|O_SYNC);
        if (fd < 0) {
            perror("open");
            exit(EXIT_FAILURE);
        }

        /* map the LWHPS2FPGA bridge into process memory */
        bridge_map = mmap(NULL, PAGE_SIZE, PROT_WRITE|PROT_READ|PROT_EXEC, MAP_SHARED,
                    fd, blink_base);
        if (bridge_map == MAP_FAILED) {
            perror("mmap");
            goto cleanup;
        }


        /* get the delay_ctrl peripheral's base address */
        blink_mem = (unsigned char *) (bridge_map + BLINK_OFFSET);

        start = clock();
        /* write the value */
        for(i = 0; i < 1000000; i++)
        {
            *blink_mem = i;
            dummy = *blink_mem;
        }
        stop = clock();
        duration = ( stop - start ) / (double) CLOCKS_PER_SEC;
        printf("%f", duration);

        if (munmap(bridge_map, PAGE_SIZE) < 0) {
            perror("munmap");
            goto cleanup;
        }

        ret = 0;

    cleanup:
        close(fd);
        return ret;
    }

I'm writing to the virtual address space mmap returns and I can verify the write by reading the value at that address but I don't see the value getting updated in the FPGA.

How does the physical address get written when I write to the user virtual space? Is there a way to debug and see if the physical address space is actually being written?

6
  • Have you made your map uncached when you used ioremap()? Commented Oct 24, 2015 at 1:16
  • Also, since you wrote the driver, shouldn't you know if the buffer is getting full? ... Note that using memory mapping, a user program can write at bus clock rate i.e. nanoseconds, not microseconds. If I had to guess, I would say your FPGA is not doing the bus signalling correctly and each access is terminating after a bus timeout, or something like that. Commented Oct 24, 2015 at 1:23
  • @MurrayJensen: I have edited my question, could you please take a look at it. Thanks!
    – Karthik
    Commented Oct 26, 2015 at 23:41
  • If you are using memory mapped access, there is no writing from userspace to kernel necessary. Commented Oct 27, 2015 at 12:32
  • 1
    1. try using pgprot_noncached(vma->vm_page_prot) as the last arg to remap_pfn_range() - this will ensure that the accesses will be uncached; 2. what is the minor number of your "/dev/HPS2FPGA" device? and are you sure the major number is correct (assuming you are using the "simple" driver you referred to); 3. are you sure you have programmed the memory controller correctly so that the physical address 0xc0000000 will access your device? you will need to know the correct chip select etc Commented Oct 27, 2015 at 13:02

1 Answer 1

3

OK, it looks like the subject of this question is a furphy ... memory mapped I/O (done correctly) will be as fast as the processor can do it for the hardware being accessed and there will be no overhead for doing this from user mode as opposed to kernel mode (i.e. there is no "write from the userspace to kernel").

However, you still have to think about what is happening when you do a read from or a write to an address (this is where the question moved on to). In most architectures, there are two mappings - the virtual to the physical, and the physical to the device. The first is set up in the virtual memory hardware and the second is set up in the memory controller.

In addition to the mappings, all accesses usually go via cache hardware, so you have to decide whether you want the accesses to be cached or not. If the underlying device being accessed is RAM of some sort, then you usually want accesses to be cached. For other types of devices, generally you don't.

There may be a lot of other things to think about (e.g. whether the VM mapping is resident in the VM hardware, the width and timing of accesses, priority, permissions, etc) but cache is the first.

In @Karthik's case, because he had not turned off cache in his mapping, depending on the type of cache, either an entire cache line was being written when he wrote to the address (write-through), or the write was being delayed (write-back) (if you want some nitty gritty about cache, try this).

To answer the specific (follow up) questions, once the virtual address mapping is done and the cache has done its job, the access goes to the memory controller - this hardware decides which bus and/or device is being accessed and does the "right thing" for that hardware, usually involving asserting a chip select and/or a write enable signal, maybe copying a part or all of the physical address onto address lines, maybe some setup timing, etc.

... and the best way to debug this stuff is to have an analyser of some sort connected to your device or bus, or if this is too difficult/expensive, there might be some support for debugging in the memory controller.

One other minor but important point ... take note of the declaration of blink_mem in the code above - the volatile type qualifier is very important. It tells the compiler not to muck with accesses to the address. In addition to this, you should be aware of any special pipeline instructions to do with memory accesses (check out the eieio instruction in powerpc - someone has a sense of humour :-)

Finally, just to re-iterate what was said in the comments, which turned out to be the real answer to the question, when calling remap_pfn_range() you turn off cache by modifying the page protections specified in the last argument (prot) using the pgprot_noncached() macro. Also read this and this and particularly this. Cheers!

2
  • 1
    Thank you for the explanation! Being a FPGA digital design engineer this was my first try writing device drivers and it has been a learning curve. Now, I know a lot more about operating systems than how to use the UI to move the mouse over and copying from one folder to another :P. I got misled after reading the textbook, which said "use ioremap(); will map the physical address to virtual address". I didn't know that there was separate virtual memory for the Kernel and User space. This was cleared only after reading about remap_pfn_range().
    – Karthik
    Commented Nov 3, 2015 at 17:03
  • I guess technically they aren't separate as it would be the same hardware doing the translation, but most architectures would have support for context switching to reduce the cost of changing the VM mappings. note also that there is an ioremap_nocache() companion to ioremap() and also that the physical address mappings in the memory controller are usually (but not always) set up at boot time outside the kernel e.g. by boot code (aka BIOS) ... and this set up is usually communicated to the kernel in some way (e.g. device tree) or else the kernel has to discover the set up in some way. Commented Nov 3, 2015 at 20:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .