1

Basically I'm using Linux 2.6.34 on PowerPC (Freescale e500mc). I have a process (a kind of Virtual Machine that was developed in-house) that uses about 2.25 G of mlocked VM. When I kill it, I notice that it takes upwards of 2 minutes to terminate.

I investigated a little. First, I closed all open file descriptors but that didn't seem to make a difference. Then I added some printk in the kernel and through it I found that all delay comes from the kernel unlocking my VMAs. The delay is uniform across pages, which I verified by repeatedly checking the locked page count in /proc/meminfo. I've checked with programs that allocate that much memory and they all die as soon as I signal them.

What do you think I should check now? Thanks for your replies.

Edit: I had to find a way to share more information about the problem so I wrote this below program:

#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <string.h>
#include <errno.h>
#include <signal.h>
#include <sys/time.h>

#define MAP_PERM_1              (PROT_WRITE | PROT_READ | PROT_EXEC)
#define MAP_PERM_2              (PROT_WRITE | PROT_READ)

#define MAP_FLAGS               (MAP_ANONYMOUS | MAP_FIXED | MAP_PRIVATE)

#define PG_LEN                  4096
#define align_pg_32(addr)       (addr & 0xFFFFF000)
#define num_pg_in_range(start, end)     ((end - start + 1) >> 12)

inline void __force_pgtbl_alloc(unsigned int start)
{
        volatile int *s = (int *) start;
        *s = *s;
}

int __map_a_page_at(unsigned int start, int whichperm)
{
        int perm = whichperm ? MAP_PERM_1 : MAP_PERM_2;

        if(MAP_FAILED == mmap((void *)start, PG_LEN, perm, MAP_FLAGS, 0, 0)){
                fprintf(stderr,
                        "mmap failed at 0x%x: %s.\n",
                        start, strerror(errno));
                return 0;
        }

        return 1;
}

int __mlock_page(unsigned int addr)
{
        if (mlock((void *)addr, (size_t)PG_LEN) < 0){
                fprintf(stderr,
                        "mlock failed on page: 0x%x: %s.\n",
                        addr, strerror(errno));
                return 0;
        }

        return 1;
}

void sigint_handler(int p)
{
        struct timeval start = {0 ,0}, end = {0, 0}, diff = {0, 0};
        gettimeofday(&start, NULL);
        munlockall();
        gettimeofday(&end, NULL);
        timersub(&end, &start, &diff);

        printf("Munlock'd entire VM in %u secs %u usecs.\n",
                diff.tv_sec, diff.tv_usec);

        exit(0);
}

int make_vma_map(unsigned int start, unsigned int end)
{
        int num_pg = num_pg_in_range(start, end);

        if (end < start){
                fprintf(stderr,
                        "Bad range: start: 0x%x end: 0x%x.\n",
                        start, end);
                return 0;
        }

        for (; num_pg; num_pg --, start += PG_LEN){
                if (__map_a_page_at(start, num_pg % 2) && __mlock_page(start))
                        __force_pgtbl_alloc(start);
                else
                        return 0;
        }

        return 1;
}

void display_banner()
{
        printf("-----------------------------------------\n");
        printf("Virtual memory allocator. Ctrl+C to exit.\n");
        printf("-----------------------------------------\n");
}

int main()
{
        unsigned int vma_start, vma_end, input = 0;
        int start_end = 0; // 0: start; 1: end;

        display_banner();

        // Bind SIGINT handler.
        signal(SIGINT, sigint_handler);

        while (1){
                if (!start_end)
                        printf("start:\t");
                else
                        printf("end:\t");

                scanf("%i", &input);

                if (start_end){
                        vma_end   = align_pg_32(input);
                        make_vma_map(vma_start, vma_end);
                }
                else{
                        vma_start = align_pg_32(input);
                }
                start_end = !start_end;
        }

        return 0;
}

As you would see, the program accepts ranges of virtual addresses, each range being defined by start and end. Each range is then further subdivided into page-sized VMAs by giving different permissions to adjacent pages. Interrupting (using SIGINT) the program triggers a call to munlockall() and the time for said procedure to complete is duly noted.

Now, when I run it on freescale e500mc with Linux version at 2.6.34 over the range 0x30000000-0x35000000, I get a total munlockall() time of almost 45 seconds. However, if I do the same thing with smaller start-end ranges in random orders (that is, not necessarily increasing addresses) such that the total number of pages (and locked VMAs) is roughly the same, observe total munlockall() time to be no more than 4 seconds.

I tried the same thing on x86_64 with Linux 2.6.34 and my program compiled against the -m32 parameter and it seems the variations, though not so pronounced as with ppc, are still 8 seconds for the first case and under a second for the second case.

I tried the program on Linux 2.6.10 on the one end and on 3.19, on the other and it seems these monumental differences don't exist there. What's more, munlockall() always completes at under a second.

So, it seems that the problem, whatever it is, exists only around the 2.6.34 version of the Linux kernel.

  • It's an in-house application and you're asking on unix.SE what your application does when it exits? – lcd047 Jun 30 '15 at 12:08
  • Nothing basically. I'm talking about all kinds of exits, even those upon receiving unhandled signals (KILL). What happens after that should be outside the scope of the in-house application? – Anirban Ghoshal Jun 30 '15 at 12:10
  • No: what happens at exit depends essentially on the state of the application before exit. – lcd047 Jun 30 '15 at 12:12
  • Fair enough. What I'm looking for is some advice on this. My application allocates around 2.25 G of memory and then proceeds to mlock() down upon it. What I suspected initially was maybe pending data transfer, etc, and so I closed all open fd's (by handling the SIGTERM). It didn't make an iota of difference. So, after that we added printk's in the kernel, and from that we found out that all delay comes from exit_mm. The delay is more or less uniform across the VMA's as the locked count in /proc/meminfo decreases uniformly. My application does fragment VM a lot, but 2 minutes is just too high. – Anirban Ghoshal Jun 30 '15 at 12:19
  • Also, my machine is a diskless system - completely based out of tmpfs, etc. – Anirban Ghoshal Jun 30 '15 at 12:20
2

If you've over-committed memory, a lot of tmpfs may be on disk. You may need to page stuff in to process the shutdown. mlock() is likely to force a lot of the other memory to disk. As you indicate you are diskless, you are likely reading over the network.

Run sar gathering all stats while the server is shutting down. (sar may not be installed by default.) Use the option capture data to a file, and an interval of 5 to 10 seconds. You can then look examine what resources were the bottleneck at leisure.

Edit: Based on your comment, you may have created a locking issues in the page table. How many mlock calls do you do? Are they all required? Try dumping the list of locked memory segments before you shutdown your process.

  • Thanks for your reply! I don't have a swap space configured. Just about all of my process's memory map consists of anonymous memory mappings (except a small portion which is NFS pages). Maybe I'll look at the sar capture for a clue to where the bottleneck is. However, one thing for sure, the page table spinlock/mm sem are not available while the process is exiting - if I do a cat /proc/<pid>/maps the command hangs and is able to continue only when the former exits. – Anirban Ghoshal Jun 30 '15 at 12:43
  • @AnirbanGhoshal See edit. – BillThor Jun 30 '15 at 12:51
  • You're right about that. Basically, all 2.25 G is mlocked, and since there are a large number of VMAs, it takes a large number of mlock() calls as well. I tried to simulate the issue by proactively unlocking my memory using munlockall() and yes, the delay is replicated. After munlock() completes, they process exits immediately. I wouldn't be so much bothered if it did not exit at all, but it does, so there's no deadlocks and such like :/ Btw, I suppose there's no such thing as a semtimedop in the kernel? It's either up or down? – Anirban Ghoshal Jun 30 '15 at 12:56
  • @AnirbanGhoshal I wasn't expecting deadlocks, but the table will have locking code to ensure different processes cannot make conflicting changes. No such luck with a semitimedrop, a process is an entry in a table to which resources may be allocated. Until all resources are released it exists. Even kill -9 will wait for resources to be de-allocated, but it is usually a very fast process. – BillThor Jun 30 '15 at 13:13
  • I see. Thanks. I'll keep you posted on the debug progress. – Anirban Ghoshal Jun 30 '15 at 13:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.