Questions tagged [perf]
perf is Linux profiling tool for both Linux kernel and underlying hardware (hardware counters).
89
questions
0
votes
0
answers
14
views
Cache or save results of perf report
For large files, perf report takes a while. e.g. around 5 minutes for 500G perf.data. Is there a way to dump (certain states of) perf report to a file, or tell perf to cache something that enables ...
- 126
0
votes
1
answer
21
views
see vdso function with perf report
On my perf report I see a bunch of lines with vdso modules and no function names. I am guessing these are for gettimeofday type calls. But how can I see actual function calls?
My perf record command:
...
- 1
0
votes
0
answers
31
views
perf error: sys_perf_event_open() syscall returned with 4 (Interrupted system call) for event
When I run the following perf command I get the error:
sudo perf record -e python:take_gil -- python3-dbg -c "import time; time.sleep(1)"
Error:
The sys_perf_event_open() syscall returned ...
- 177
0
votes
2
answers
49
views
INST_RETIRED.ANY no longer works as a performance counter with perf on linux 6.7
With the older kernels INST_RETIRED.ANY (as well as many of the other counters documented in https://perfmon-events.intel.com/ahybrid.htm ) worked as counters for perf.
I am now using perf with a 6.7 ...
- 1
2
votes
0
answers
53
views
A simple memcpy loop with stride of 8-16-32 Bytes, no L1 cache misses, what stalls backend cycles?
I am trying to understand the CPU cache performance in a single producer single consumer queue algorithm, but cannot pinpoint the cause of performance degradation in some cases. The following ...
- 2,183
0
votes
1
answer
129
views
Which tools for micro-benchmarking?
I'm not sure which tool to use for micro-benchmarking a C program.
I would like to measure both:
Memory usage, RSS ( Resident Set Size )
CPU cycles
I did use perf record -g and perf script piped ...
1
vote
0
answers
21
views
What is futex-default-S?
I can't find more info about futex-default-S
Is this a system call? module? or else?
https://github.com/torvalds/linux/blob/06dc10eae55b5ceabfef287a7e5f16ceea204aa0/tools/perf/Documentation/perf-lock....
- 925
1
vote
0
answers
107
views
perf instruction count
so i've been playing with perf and assembly i have the following program:
.intel_syntax noprefix
.global _start
_start:
mov cl, 2
mov ebx, 0b101
shr ebx, cl
and bl, 1
je do_stuff
...
- 11
0
votes
1
answer
465
views
Unable to find any information about TLB on my computer or obtain information about hardware counters about TLB
The Ubuntu version I am using is Ubuntu 18.04.6 LTS, and the kernel version is 5.4.0-148 generic. My processor is 12th Gen Intel (R) Core (TM) i7-12700.
I would like to know the number of TLB ...
- 1
0
votes
0
answers
95
views
Security of perf under root access
when I use perf it asked for more permission to be used:
You may not have permission to collect stats.
I was lazy so I did sudo perf (some short program I created myself) but I later realize you can ...
1
vote
0
answers
493
views
PAPI doesn't have available events on Ubuntu even though perf seems to be working properly
I'm trying to install PAPI on my machine:
$ uname -a
Linux XXXXXX 5.19.0-35-generic #36~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 17 15:17:25 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
and it does not ...
- 11
3
votes
0
answers
301
views
Lower system call perfermance on aarch64 vs. x86-64
Does anyone know why system calls take longer to execute on 64-bit ARM (aarch64) compared to 64-bit Intel CPUs (x86-64)? Please see below my benchmarking setup.
64-bit ARM CPU: Ampere Altra Q80-30, ...
- 31
0
votes
0
answers
1k
views
Unable to use perf to find L3 Cache miss rate
I am trying to get the L3 cache miss rate when I try to use perf. The following is the command I run:
perf stat -e instructions,LLC-load-misses,LLC-loads ./program <arg1> <arg2> <arg3&...
1
vote
0
answers
291
views
How to measure memory bandwidth of an app?
My coworkers are trying to figure out a performance problem. I suspect they're reading lots of unnecessary memory and hitting a memory bandwidth problem but I don't want to suggest that unless I know ...
- 11
0
votes
0
answers
113
views
perf in docker doesn't show functions
I am trying to use perf on an application running in docker. In order to get a trace, I am running perf on the host and attaching to the application running in the container. This works fine.
However, ...
- 343
1
vote
0
answers
192
views
perf lock <...> for non-root user?
root can run perf lock record, whereas non-root can't:
$ /code/linux/tools/perf/perf lock record true
tracepoint lock:lock_acquire is not enabled. Are CONFIG_LOCKDEP and CONFIG_LOCK_STAT enabled?
...
- 361
1
vote
1
answer
1k
views
how to use metrics with perf stat
perf list shows a bunch of what it calls metrics. The list starts off with
List of pre-defined events (to be used in -e):
Metrics:
BAClear_Cost
[Average Branch Address Clear Cost (fraction ...
- 15
0
votes
1
answer
161
views
performance comparison with perf-stat
please check the below code.
the code be implemented with two solutions. one is array, the other one is linking list.
i expected the array version should be faster than the linking list one,
because ...
0
votes
1
answer
163
views
the column values shown by perf sched record don't make sense
The problem I'm seeing occurs with virtually any command I run with
perf sched record
I'll use ls as a very simple example. So if I do
perf sched record ls
followed by
perf sched latency
the ...
- 15
0
votes
1
answer
413
views
what is the difference between itlb_misses.miss_causes_a_walk, itlb_misses.walk_completed, and itlb_misses.walk_pending
I ran
perf stat -e itlb_misses.miss_causes_a_walk,itlb_misses.walk_completed,itlb_misses.walk_pending grep Hecuba /storage/nvme6/complete_shakespeare_4096_times
and got
Performance counter stats ...
- 15
0
votes
1
answer
689
views
perf top not working as expected
I'm completely confused about the output of perf top
I'm running a C program called sac2, which burns an entire CPU. Specifically it iterates through a 100 billion byte array adding all the bytes as ...
- 15
1
vote
0
answers
754
views
How do I debug `perf`'s "Permission error mapping pages." when running `perf` on multiple processes in parallel?
perf sometimes gives me
Permission error mapping pages.
Consider increasing /proc/sys/kernel/perf_event_mlock_kb,
or try again with a smaller value of -m/--mmap_pages.
(current value: 4294967295,0)
...
- 121
2
votes
1
answer
2k
views
linux-tools-common when on a newer kernel version
I'm currently running kernel 5.16.15-051615-generic - I've landed on it while diagnosing and trying to fix an audio issue. Everything works great now so I'm reluctant to go back to 5.4.0
I'd like to ...
- 123
2
votes
1
answer
1k
views
How do I use perf trace record?
I can use perf trace as a low-overhead replacement for strace, e.g. to trace all Apache instances:
perf trace -p $(pidof apache2 | tr ' ' ',')
To run the trace only for up to 10 seconds:
perf trace -...
- 14.4k
3
votes
2
answers
2k
views
how to set capabilities (setcap) on perf
I'd like to use the perf utility. I was following instructions to set up a privileged group of users who are permitted to execute performance monitoring and observability without limits (as instructed ...
- 133
0
votes
0
answers
328
views
perf mem record <command> does not work as non root user
As a non root user, perf mem record does not work even though I have set /proc/sys/kernel/perf_event_paranoid to -1 and set CAP_SYS_ADMIN on /usr/bin/perf
getcap /usr/bin/perf
/usr/bin/perf = ...
- 11
2
votes
0
answers
139
views
Error running python script
I am following this tutorial: How to analyze your system with perf and Python.
While running the following perf script --gen-script python command I am getting an error.
Python scripting not supported....
1
vote
0
answers
158
views
linux perf: limiting to just some small periods of time
I'm writing a video game, and have used perf(1) and flame graphs to get some decent "whole program performance" information.
However, I also want to investigate specific worst-case periods, ...
- 1,489
11
votes
3
answers
8k
views
How to run linux perf without root
I want to benchmark an application of mine. Up to now I used gnu time, but perf yields much better stats.
As a matter of principle I would like to go the route of a decicated perf user instead of ...
- 221
5
votes
1
answer
8k
views
How does the linux perf tool get the miss rate of the l2 cache?
I want to use perf tool to get the miss rate of l2 cache. I know how to get L1, and LLC.
perf stat -e L1-dcache-loads,L1-dcache-load-misses,L1-dcache-stores command
perf stat -e LLC-loads,LLC-load-...
user450119
1
vote
0
answers
411
views
How to take stack samples using `perf` based on wall-clock time
I am trying to use Linux's perf_events framework to investigate an issue with an application on one of our servers. Based on my reading about the perf tool, collecting stacks is relatively ...
- 111
3
votes
2
answers
4k
views
Perf permission for users other than root
I don't know why user is not able to collect perf stat for a service (nginx process which has been started by sudo). As you can see the kernel and perf versions match.
$ uname -r
4.19.125
$ which perf
...
- 1,251
0
votes
1
answer
257
views
Locating printf object/module
I would like to know which kernel module is called when printf is executed in my C code. How can I find that? Let's say, when I use ldd I can find which libraries are attached to my binary file. I am ...
- 1,251
1
vote
0
answers
895
views
Can't install Perf on Kali Linux
I tried different ways to install perf but they are all failed:
root@kali:~/Desktop# perf
/usr/bin/perf: line 13: exec: perf_5.3: not found
E: linux-perf-5.3 is not installed.
root@kali:~/Desktop# ...
- 111
1
vote
1
answer
333
views
Where does `perf script` timestamps come from?
The third column from perf script seems to be close but not quite the uptime, where is that timestamp coming from? Is there a way to access that timestamp other than accessing sampled events?
$ perf ...
- 3,764
0
votes
0
answers
659
views
High-frequency performance counter sampling using perf record/report
I want to retrieve performance counter counts at a high frequency (i.e. 100-200Hz) using the perf tool (similar in functionality to https://github.com/RRZE-HPC/likwid/wiki/likwid-perfctr#the-timeline-...
- 121
1
vote
1
answer
101
views
How to get sleep time of a program
I am trying to collect some statistics for a script that I run on a daily bases. I would like to get the amount of time that is actually spent sleeping. Is there a way to just get the sum of all sleep ...
- 11
1
vote
0
answers
18
views
Which of two methodologies is the correct one getting perf stats on a multi-threaded process?
I'm investigating the performance of a multi-threaded database sever. There is a particular workload that runs in roughly 61 seconds on a particular machine. The pid of the database process was 79894 ...
1
vote
0
answers
540
views
Perf tool, used for profiling, crashes on heavy load
When I am using "perf record -F 99 -g -p -- sleep 300" during average load, I am getting the perf.data file without any problem. But when the load is high, it is crashing. The only way to get an ok ...
- 19
2
votes
0
answers
263
views
How to trace script execution events
My goal:
To receive events from the kernel when a file/script is opened for execution with the absolute path of the file (or maybe the working directory, to resolve the absolute path of the file).
I ...
- 21
0
votes
1
answer
1k
views
What does `perf stat` option `--cpu` is actually showing?
I am trying to understand the --cpu option of perf-stat. In the man pages it says:
Count only on the list of CPUs provided. Multiple CPUs can be provided
as a comma-separated list with no space: ...
- 101
13
votes
1
answer
10k
views
Security implications of changing “perf_event_paranoid”
I'd like to use the perf utility to gather measurements for my program. It runs on a shared cluster machine with Debian 9 where by default the /proc/sys/kernel/perf_event_paranoid is set to 3, ...
- 2,732
7
votes
1
answer
9k
views
How long time does a context switch take in Linux (ubuntu 18.04)
I'm interessted in getting the number of context switches a two processes in a KVM vm takes on a singel CPU over some time.
Earlier i have used perf, is this best practice?
And how much time is used ...
- 113
1
vote
0
answers
131
views
Determine average and maximum execution time of a task by using ftrace/perf
I've gathered some traces containing scheduling events ('sched:*') by using ftrace and perf record. By analyzing the results many of my questions could already be answered.
However, I was wondering ...
- 121
4
votes
0
answers
1k
views
Does `perf trace -p <PID>` trace all threads?
strace -f -p <PID> traces system calls in all threads of a multi-threaded process. (This also follows forks, there is an option -b execve that tries to address that).
Does perf trace -p <...
- 51.4k
2
votes
1
answer
502
views
Is there a command to show tracepoints "live", which includes the PID?
I have tried looking at the documentation for perf script, perf trace, and trace-cmd, including the list of commands in "SEE ALSO".
I can trace e.g. sched:sched_process_exec "live" using perf trace -...
- 51.4k
3
votes
0
answers
790
views
Perf Tool monitoring by PID
I'm using linux perf tool to monitor the performance of some processes. The documentation tells that you can flag it to collect profiles on per-thread, per-process and per-cpu basis.
I get that the ...
1
vote
1
answer
3k
views
Why 'perf' needs to match the exact running Linux kernel version?
If you try to run a random perf binary that does not match your currently running Linux kernel, it says:
$ perf
WARNING: perf not found for kernel 4.13.0-45
Of course, if I get the perf for this ...
- 113
0
votes
0
answers
2k
views
unresolved symbols sudo perf record/report
I am trying to use perf to profile a program that I have written on a work server. I have sudo privilege but should limit what I do with it.
$ rm perf.data
$ perf record ./catch-hedging ...
- 101
3
votes
1
answer
2k
views
Why does `perf stat` show 0 context switches?
I ran a shell pipeline under perf stat, using taskset 0x1 to pin the whole pipeline to a single CPU. I know taskset 0x1 had an effect, because it more than doubled the throughput of the pipeline. ...
- 51.4k