Why does my process take too long to die?

Question

Basically I'm using Linux 2.6.34 on PowerPC (Freescale e500mc). I have a process (a kind of Virtual Machine that was developed in-house) that uses about 2.25 G of mlocked VM. When I kill it, I notice that it takes upwards of 2 minutes to terminate.

I investigated a little. First, I closed all open file descriptors but that didn't seem to make a difference. Then I added some printk in the kernel and through it I found that all delay comes from the kernel unlocking my VMAs. The delay is uniform across pages, which I verified by repeatedly checking the locked page count in /proc/meminfo. I've checked with programs that allocate that much memory and they all die as soon as I signal them.

What do you think I should check now? Thanks for your replies.

Edit: I had to find a way to share more information about the problem so I wrote this below program:

#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <string.h>
#include <errno.h>
#include <signal.h>
#include <sys/time.h>

#define MAP_PERM_1              (PROT_WRITE | PROT_READ | PROT_EXEC)
#define MAP_PERM_2              (PROT_WRITE | PROT_READ)

#define MAP_FLAGS               (MAP_ANONYMOUS | MAP_FIXED | MAP_PRIVATE)

#define PG_LEN                  4096
#define align_pg_32(addr)       (addr & 0xFFFFF000)
#define num_pg_in_range(start, end)     ((end - start + 1) >> 12)

inline void __force_pgtbl_alloc(unsigned int start)
{
        volatile int *s = (int *) start;
        *s = *s;
}

int __map_a_page_at(unsigned int start, int whichperm)
{
        int perm = whichperm ? MAP_PERM_1 : MAP_PERM_2;

        if(MAP_FAILED == mmap((void *)start, PG_LEN, perm, MAP_FLAGS, 0, 0)){
                fprintf(stderr,
                        "mmap failed at 0x%x: %s.\n",
                        start, strerror(errno));
                return 0;
        }

        return 1;
}

int __mlock_page(unsigned int addr)
{
        if (mlock((void *)addr, (size_t)PG_LEN) < 0){
                fprintf(stderr,
                        "mlock failed on page: 0x%x: %s.\n",
                        addr, strerror(errno));
                return 0;
        }

        return 1;
}

void sigint_handler(int p)
{
        struct timeval start = {0 ,0}, end = {0, 0}, diff = {0, 0};
        gettimeofday(&start, NULL);
        munlockall();
        gettimeofday(&end, NULL);
        timersub(&end, &start, &diff);

        printf("Munlock'd entire VM in %u secs %u usecs.\n",
                diff.tv_sec, diff.tv_usec);

        exit(0);
}

int make_vma_map(unsigned int start, unsigned int end)
{
        int num_pg = num_pg_in_range(start, end);

        if (end < start){
                fprintf(stderr,
                        "Bad range: start: 0x%x end: 0x%x.\n",
                        start, end);
                return 0;
        }

        for (; num_pg; num_pg --, start += PG_LEN){
                if (__map_a_page_at(start, num_pg % 2) && __mlock_page(start))
                        __force_pgtbl_alloc(start);
                else
                        return 0;
        }

        return 1;
}

void display_banner()
{
        printf("-----------------------------------------\n");
        printf("Virtual memory allocator. Ctrl+C to exit.\n");
        printf("-----------------------------------------\n");
}

int main()
{
        unsigned int vma_start, vma_end, input = 0;
        int start_end = 0; // 0: start; 1: end;

        display_banner();

        // Bind SIGINT handler.
        signal(SIGINT, sigint_handler);

        while (1){
                if (!start_end)
                        printf("start:\t");
                else
                        printf("end:\t");

                scanf("%i", &input);

                if (start_end){
                        vma_end   = align_pg_32(input);
                        make_vma_map(vma_start, vma_end);
                }
                else{
                        vma_start = align_pg_32(input);
                }
                start_end = !start_end;
        }

        return 0;
}

As you would see, the program accepts ranges of virtual addresses, each range being defined by start and end. Each range is then further subdivided into page-sized VMAs by giving different permissions to adjacent pages. Interrupting (using SIGINT) the program triggers a call to munlockall() and the time for said procedure to complete is duly noted.

Now, when I run it on freescale e500mc with Linux version at 2.6.34 over the range 0x30000000-0x35000000, I get a total munlockall() time of almost 45 seconds. However, if I do the same thing with smaller start-end ranges in random orders (that is, not necessarily increasing addresses) such that the total number of pages (and locked VMAs) is roughly the same, observe total munlockall() time to be no more than 4 seconds.

I tried the same thing on x86_64 with Linux 2.6.34 and my program compiled against the -m32 parameter and it seems the variations, though not so pronounced as with ppc, are still 8 seconds for the first case and under a second for the second case.

I tried the program on Linux 2.6.10 on the one end and on 3.19, on the other and it seems these monumental differences don't exist there. What's more, munlockall() always completes at under a second.

So, it seems that the problem, whatever it is, exists only around the 2.6.34 version of the Linux kernel.

Answer 1

If you've over-committed memory, a lot of tmpfs may be on disk. You may need to page stuff in to process the shutdown. mlock() is likely to force a lot of the other memory to disk. As you indicate you are diskless, you are likely reading over the network.

Run sar gathering all stats while the server is shutting down. (sar may not be installed by default.) Use the option capture data to a file, and an interval of 5 to 10 seconds. You can then look examine what resources were the bottleneck at leisure.

Edit: Based on your comment, you may have created a locking issues in the page table. How many mlock calls do you do? Are they all required? Try dumping the list of locked memory segments before you shutdown your process.