I have some before/after perf traces provided to me and I'm trying to understand why they have radically different (about 2x) numbers of samples in each trace. The traces were collected via the same script and so they represent the same amount of wall time being profiled. The traces were recorded with perf record -ag.

By the time the traces reach me they've been run through perf script, so I have the text output. My first question was how to interpret the fields of each 'sample header':

swapper     0 [000] 27324.824397: cycles: 

In particular I'm not sure what the [000] represents.

Secondly, with 12 reported as nrcpus online/avail, I can't figure out why I dont' see 12 stacks captured for each sample. In this particular example I see six swapper stacks [000], followed by six swapper stacks [001], for a bit until the more interesting processes spin up. I'm wondering if that's a factor of halted cpus as opposed to cpus that were caught actually running the idle process (swapper)? By my thinking, two profiles with -ag on the same machine for the same wall time should have roughly the same number of samples since all cores should be recorded on each sample, regardless of how busy the system is.

Any help figuring out what I'm missing would be much appreciated!

Thanks, Terry

Here's the full header information if that helps:

# captured on: Tue Feb 23 06:17:04 2016
# hostname : du76-tip-EC-1
# os release : 3.13.0-66-generic
# perf version : 3.13.11-ckt27
# arch : x86_64
# nrcpus online : 12
# nrcpus avail : 12
# cpudesc : Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz
# cpuid : GenuineIntel,6,62,4
# total memory : 65813064 kB
# cmdline : /usr/lib/linux-tools-3.13.0-66/perf record -g --output /var/log/cpu_profile/record --all-cpus 
# event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, excl_host = 0, excl_gu
est = 1, precise_ip = 0, attr_mmap2 = 0, attr_mmap  = 1, attr_mmap_data = 0
# HEADER_CPU_TOPOLOGY info available, use -I to display
# HEADER_NUMA_TOPOLOGY info available, use -I to display
# pmu mappings: cpu = 4, software = 1, uncore_irp = 24, uncore_pcu = 13, tracepoint = 2, uncore_imc_0 = 16, uncore_imc_1 = 17, unc
ore_imc_2 = 18, uncore_imc_3 = 19, uncore_imc_4 = 20, uncore_imc_5 = 21, uncore_imc_6 = 22, uncore_imc_7 = 23, uncore_qpi_0 = 25, 
uncore_qpi_1 = 26, uncore_qpi_2 = 27, uncore_cbox_0 = 7, uncore_cbox_1 = 8, uncore_cbox_2 = 9, uncore_cbox_3 = 10, uncore_cbox_4 =
 11, uncore_cbox_5 = 12, uncore_r2pcie = 28, uncore_r3qpi_0 = 29, uncore_r3qpi_1 = 30, breakpoint = 5, uncore_ha_0 = 14, uncore_ha
_1 = 15, uncore_ubox = 6
# ========

1 Answer 1


I can't figure out why I dont' see 12 stacks captured for each sample

It is because the processor

Intel(R) Xeon(R) CPU E5-1650 v2 @ 3.50GHz

has only 6 physical cores able to run 12 threads (Intel Hyper-Threading Technology).

Please, post also details how exactly do you report the results from the records. I used to work only with perf-stat, which gave pretty straight-forward results. Also important is to know what program you run, if it is multi-threading program and so forth. The performance might be also affected by Intel Turbo.

  • Thank you @Jakuje. traces are gathered with perf -ag and then vanilla 'perf script' converts to the output I receive. And yes the code of interest is multi-threaded.
    – Terry E.
    Commented Mar 6, 2016 at 20:10
  • sorry accidentally posted too soon there. Anyway, your response may hold the information I need. Can I interpret it to mean that a Hyper-Threaded core will only report two threads (sampled stacks) if there are actually two threads concurrently scheduled? And otherwise I'll either see a single thread of some process or the swapper? Makes sense if that's the case. Thanks!
    – Terry E.
    Commented Mar 6, 2016 at 20:16
  • The swapper 0 (pid) is the "doing nothing", when you use -a switch (idle cores/threads). Intel HT can not schedule both threads simultaneously (as far as I know, at least not on low level). It can only swap context of two threads "faster". But yes, it reports to the os, 12 logical cores and therefore you got 12 stacks. The program is probably instructed to run on 6 cores only (reasonable). And it depends on what you mean by "before/after".
    – Jakuje
    Commented Mar 6, 2016 at 20:40
  • Let me try and back up some. My question is specifically, how many stacks would I expect to get from every sample recorded by perf -ag on a single socket, 6 physical core, 12 HT thread system under test. If the answer is 'it depends' as opposed to some constant like 6 or 12, please elaborate. I have two profiles with a difference of roughly 2x in the number of samples which I am trying to explain.
    – Terry E.
    Commented Mar 6, 2016 at 22:01
  • I would simply expect 12 stacks (with the -a switch and less without -a depending on the actually used cores) and they should have roughly same amount of samples. But I am no an expert on perf (yet). As I said, there might be a lot of explanations for this. With having only 2 reports and having no idea how were the programs run is still like using magic 8 ball.
    – Jakuje
    Commented Mar 6, 2016 at 22:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .