I have to find warnings in logs and I'm using grep with awk(code below). But, on VM(Ubuntu 18.04) first times of executing all script took about 120 sec(24.5 gb of logs), then I tried to run it on Win 10 with Ubuntu console and it took 20 sec(If somebody could explain why, please!). Then, I did some changes in code and speed of executing took about 500 sec(all what I did it was change grep -v ".tmp" folder/. on grep -vF "~$" folder/.) After that, I tried to change back, but speed was the same ~ 500 sec. Also, I tried to execute the same script on Win 10 with Cygwin and result was ~400 sec. Could somebody explain why it happened? Why speed of executing increased on Win, and why fluctuates depends on case(20 sec,400 sec, 500 sec)? (Yes, I understand that my code is terrible, but I assume that in future number of filters will increase, and imho it will be hard to expand script). If you have any advice to optimize script, I'm willing to any suggestions or any opinions(It's my first time of using bash script)


start=`date +%s`

rm /Users/me/Desktop/Errors/*

echo "Hello"

grep -nr "Error" . | gawk -F" " '$4 ~ /Error/' | grep -v "@Trash"| grep -v "Cien" | grep -Fv "~$" | grep "\\\\server" | grep -v "Folder1" | grep "Object is being used" > /Users/me/Desktop/Errors/OU_Other.txt

gr=`grep -c "Error" /Users/me/Desktop/Errors/OU_Other.txt`
echo "OU_Other done with $gr errors"

grep -nr "Error" . | gawk -F" " '$4 ~ /Error/' | grep -v "@Trash"| grep -v "Cien" | grep -Fv "~$" | grep "\\\\server" | grep  "Folder1" | 
grep "Object being is used" >  /Users/me/Desktop/Errors/OU_Cien.txt

gr=`grep -c "Error" /Users/me/Desktop/Errors/OU_Cien.txt`
echo "OU_Cien done with $gr errors"

grep -nr "Error" . | gawk -F" " '$4 ~ /Error/' | grep -v "@Trash"| grep -Fv "~$" | grep "\\\\server" | grep -v "Access is denied" | grep -v "Object is being used" | grep -v "Cannot enumerate" > /Users/me/Desktop/Errors/Other.txt

gr=`grep -c "Error" /Users/me/Desktop/Errors/Other.txt`
echo "Other done with $gr errors"

end=`date +%s`
echo "Runtime: $runtime sec"

ls -l /Users/me/Desktop/Errors/

closed as primarily opinion-based by Rui F Ribeiro, G-Man, slm Jul 15 '18 at 18:44

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 5
    You might be better off asking a question about how to extract what you are trying to extract. Include some sample input and explain what you are trying to get out of it. Show the grep pipeline you have built, and someone will likely come up with a more efficient one. As it is now it's very difficult to even know what these pipelines are trying to accomplish without seeing some sample input. – Jesse_b Jul 15 '18 at 11:37
  • 1
    @Jesse_b, thank you, I'll take into account your advice – Comatose Jul 15 '18 at 12:23

To briefly answer your concerns about speed,

  • Bash on Windows wires directly into the Windows kernel to handle all system calls and the different executable format. Think of it as a reverse Wine. Since it relies on fewer abstraction layers and on a built-in system specifically designed with the performance of Linux apps in mind, it's faster.

  • Cygwin, on the other hand, allows to compile POSIX(y) software to run directly on Windows natively as an .exe. That's a different goal, and a more taxing one, since sometimes they have to go the distance to provide 1:1 compatibility for various foreign system calls that don't have a direct mapping in the Windows kernel.

  • Virtual machines emulate the whole system, and are generally slower than the former two options, especially in cases of intense I/O, though they've become increasingly better at it over time. Consult your virtual machine documentation on how to tune it. (example)

If speed is any of your concern, you should strongly consider installing Linux directly on your machine. You will notice it performs much faster than any of the options you've tried.

As already noted in the comments though, the performance of your script will likely significantly improve across the board once your script is rewritten, preferably without so many unnecessary consecutive grep and awk invocations.

  • Thank you, I'll try to do this. But how about those, first time on Win 10, my script executed in 20 sec, and after some changes time increased to 500 sec, even if I tried to change back, time keeps in 500 sec? Have you any guesses about that? – Comatose Jul 15 '18 at 12:27
  • @Comatose It is impossible to tell with the scant amount of information you've provided. Could be some changes in your script that you forgot to roll back, could be different input you've used, could be additional OS activity or background tasks taking up the resources, could be something related to caching. When clueless about your system's performance, strive to document everything that you change, use a version control system for your code, and employ OS-specific performance analysis tools such as perf. – undercat Jul 15 '18 at 12:56
  • @Comatose You're welcome, but you should consider providing additional information that was requested in the comments and re-opening the question if you want suggestions on how to actually change your code to make it cleaner and run faster on any system, virtual or not. – undercat Jul 15 '18 at 13:16
  • first, I'd like to rewrite it by my own, and then I'll ask :) – Comatose Jul 15 '18 at 13:29
  • If speed is a concern, ditch shell scripting and write a Perl script. – dr01 Jul 16 '18 at 7:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.