I have some before/after perf traces provided to me and I'm trying to understand why they have radically different (about 2x) numbers of samples in each trace. The traces were collected via the same script and so they represent the same amount of wall time being profiled. The traces were recorded with
perf record -ag.
By the time the traces reach me they've been run through perf script, so I have the text output. My first question was how to interpret the fields of each 'sample header':
swapper 0  27324.824397: cycles:
In particular I'm not sure what the  represents.
Secondly, with 12 reported as nrcpus online/avail, I can't figure out why I dont' see 12 stacks captured for each sample. In this particular example I see six swapper stacks , followed by six swapper stacks , for a bit until the more interesting processes spin up. I'm wondering if that's a factor of halted cpus as opposed to cpus that were caught actually running the idle process (swapper)? By my thinking, two profiles with -ag on the same machine for the same wall time should have roughly the same number of samples since all cores should be recorded on each sample, regardless of how busy the system is.
Any help figuring out what I'm missing would be much appreciated!
Here's the full header information if that helps:
# captured on: Tue Feb 23 06:17:04 2016 # hostname : du76-tip-EC-1 # os release : 3.13.0-66-generic # perf version : 3.13.11-ckt27 # arch : x86_64 # nrcpus online : 12 # nrcpus avail : 12 # cpudesc : Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz # cpuid : GenuineIntel,6,62,4 # total memory : 65813064 kB # cmdline : /usr/lib/linux-tools-3.13.0-66/perf record -g --output /var/log/cpu_profile/record --all-cpus # event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_gu est = 1, precise_ip = 0, attr_mmap2 = 0, attr_mmap = 1, attr_mmap_data = 0 # HEADER_CPU_TOPOLOGY info available, use -I to display # HEADER_NUMA_TOPOLOGY info available, use -I to display # pmu mappings: cpu = 4, software = 1, uncore_irp = 24, uncore_pcu = 13, tracepoint = 2, uncore_imc_0 = 16, uncore_imc_1 = 17, unc ore_imc_2 = 18, uncore_imc_3 = 19, uncore_imc_4 = 20, uncore_imc_5 = 21, uncore_imc_6 = 22, uncore_imc_7 = 23, uncore_qpi_0 = 25, uncore_qpi_1 = 26, uncore_qpi_2 = 27, uncore_cbox_0 = 7, uncore_cbox_1 = 8, uncore_cbox_2 = 9, uncore_cbox_3 = 10, uncore_cbox_4 = 11, uncore_cbox_5 = 12, uncore_r2pcie = 28, uncore_r3qpi_0 = 29, uncore_r3qpi_1 = 30, breakpoint = 5, uncore_ha_0 = 14, uncore_ha _1 = 15, uncore_ubox = 6 # ========