Tag Info

Hot answers tagged

32

You'd think there'd be a utility for that, but I couldn't find it. However, this Perl one-liner should do the trick: perl -pe 's/\e\[?.*?[\@-~]//g' Example: $ command-that-produces-colored-output | perl -pe 's/\e\[?.*?[\@-~]//g' > outfile Or, if you want a script you can save as stripcolorcodes: #! /usr/bin/perl use strict; use warnings; while ...


16

The POSIX 2008 standard has a section describing "Shell and Utilities". Generally, if you stick to that your scripts should be fairly future-proof, except possibly for deprecations, but those hardly happen overnight so you should have plenty of time to update your scripts. In some cases where output format for a single utility varies widely across ...


15

cat ./text | awk ' { if ( length > x ) { x = length; y = $0 } }END{ print y }' UPD: summarizing all the advices in the comments awk 'length > max_length { max_length = length; longest_line = $0 } END { print longest_line }' ./text


14

Remove color codes (special characters) with sed sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" Or Strip ANSI escape sequences in Python Install colorama python package (pip install colorama). Put into stripcolorcodes: #!/usr/bin/env python import colorama, fileinput, sys; colorama.init(strip=True); for line in fileinput.input(): ...


13

Switching the color is done through escape sequences embedded in the text. Invariably, programs issue ANSI escape sequences, because that's what virtually all terminals support nowadays. The escape sequence to switch the foreground color to red is \e[31m, where \e designates an escape character (octal 033, hexadecimal 1b, also known as ESC, ^[ and various ...


12

If you can install the Term::ANSIColor module, this perl script works: #!/usr/bin/env perl use Term::ANSIColor qw(colorstrip); print colorstrip $_ while <>;


12

I'll try to answer from my experience. Commands don't really adhere to a formal specification, but they do adhere to a requirement to consume and generate line-oriented text. Yes, of course. Before the GNU utilities became a de facto standard, a lot of vendors would have quirky output, especially with respect to ps and ls. This caused a lot of pain. ...


9

First, very brief answers to your questions: Formal standardization of input/output conventions: no Breakage in the past due to changing output: yes Absolutely impossible to break future filters: no How can I protect myself against changes: be conservative When you say "API", you're using a term that (for good or ill) implies too much formality around ...


9

Case 1: awk '!NF {if (++n <= 2) print; next}; {n=0;print}' Case 2: awk '!NF {s = s $0 "\n"; n++; next} {if (n>1) printf "%s", s; n=0; s=""; print} END {if (n>1) printf "%s", s}'


8

Instead of matching based on the URL, try matching based on contents of the certificate. iptables -t nat -I INPUT --sport 443 -m string \ --string www.facebook.com --algo bm -j REJECT You can also match on the fingerprint but if the destination changes or updates their certificate, it will invalidate your rule.


8

This should do (under Linux): ps --ppid 2 -p 2 --deselect kthreadd (PID 2) has PPID 0 (on Linux 2.6+) but ps does not allow to filter for PPID 0; thus this work-around.


8

awk -F '","' 'BEGIN {OFS=","} { if (toupper($5) == "STRING 1") print }' file1.csv > file2.csv Output "12310","42324564756","a simple string with a , comma","string with or, without commas","string 1","USD","12","70%","08/01/2013","" "23525","74535243123","string , with commas, and - hypens and: semicolans","string with or, without commas","string ...


7

The firewall cannot control which HTTPS URLs the client is trying to access, because the URL is encrypted. The firewall can only control which sites the client is connecting to, using IP addresses, but this doesn't help if the HTTP and HTTPS versions of the site are at the same URL (and even if they aren't, you'd have to maintain a huge list of IP ...


7

You can use uniq to collapse multiple instance of blank lines into one blank line, but it will also collapse lines which contain text if they are the same and below each other.


6

With GNU ls (the version on non-embedded Linux and Cygwin, sometimes also found elsewhere), you can exclude some files when listing a directory. ls -I 'temp_log.*' -lrt With zsh, you can let the shell do the filtering. Pass -d to ls so as to avoid listing the contents of matched directories. setopt extended_glob # put this in your .zshrc ls ...


6

You can parse the second part of that filter thusly not ( (src and dest) net localnet ) It's shorthand for not src net localnet and not dest net localnet


6

One of the particularity of those processes is that they are not backed by an executable file, so you could do (in zsh): ps /proc/[0-9]*/exe(^-@:h:t) Or with any POSIX shell: ps -p "$(find -L /proc/[0-9]*/exe ! -type l | cut -d/ -f3 | paste -sd, -)" That is check for processes whose /proc/<pid>/exe is a link to a file. But that means you need to ...


6

You can have grep look for control characters, some of which are responsible for making the pretty colors on the terminal. dolongtask | grep '[[:cntrl:]]' For example, this echoes a red "test" into grep, which finds it due to it being surrounded by control characters: $ echo -e '\033[00;31mtest\033[00m' | grep --color=none '[[:cntrl:]]' test <-- ...


5

sed -rn "/.{$(<file expand -t1 |wc -L)}/{p;q}" file This first reads the file inside the command substitution and outputs the length of the longest line, (previously, expand converts tabs to spaces, to overcome the semantics of wc -L -- each tab in the line will add 8 instead of 1 to line length). This length is then used in a sed expression meaning ...


5

Use opencv. There are plenty of examples of processing video and face detection.


5

With -d " ", the field separator is one (and only one) space character. Contrary to the shell word splitting, cut doesn't treat space any different than any other character. So cut -d " " -f2 returns "" in root   19, just like it would return "" for cut -d: -f2 in root:::19. You'd need to either squeeze the blanks to transform any sequence of space into one ...


5

One way to recognize kernel processes is that they don't use any user memory, so the vsz field is 0. This also catches zombies (thanks to Stephane Chazelas for this observation), which can be eliminated based on their status. ps axl | awk '$7 != 0 && $10 !~ "Z"' To list just the PIDs: ps -e -o pid= -o state= -o vsize= | awk '$2 != "Z" && ...


5

use classic regex: grep -i 'spider\|bot' or extended regex (or even perl regex -P): grep -Ei 'spider|bot' or multiple literal patterns (faster than a regular expression): grep -Fi -e 'spider' -e 'bot'


5

Perhaps awk? awk 'BEGIN{a = "foo"}; a !~ /foo|bar/ && $0 !~ /foo|bar/{print a}; {a = $0};END{if(a !~ /foo|bar/){print a}}' InputFile


4

Only covering 1) of your question. Naturally APIs can always change at the will of their creators, and thusly break dependent software, in any language. That said, the great idea of the Unix tools' I/O "APIs" is that there is practically none (maybe 0x0a as line end). A good script filters data with the Unix tools instead of creating it. That means that ...


4

Try this : grep -v "$(grep -E -B1 "foo|bar" InputFile)" InputFile


3

if you need all of them find . -maxdepth 1 -type f | sort | awk '$0 > "./reference"' if you need the first find . -maxdepth 1 -type f | sort | awk '$0 > "./reference"{print;exit}'


3

OK, if the length of the columns is not known, I'd switch to a more powerful language than bash: #!/usr/bin/perl use warnings; use strict; my $string = shift; open my $FH, '<', '1.txt' or die $!; my $first_line = <$FH>; my ($before, $name) = $first_line =~ /(.* )(NAME *)/; my $column = length $before; $string .= ' ' x (length($name) - length ...


3

I know of one option. If you have internal DNS servers for use, then put some static references in your TLD zone data that resolve the domains (that you do not wish to establish the outside connections) to just 127.0.0.1. This way, all the hosts using the central DNS within your network will resolve ( facebook.com/twitter.com per se) domains into loopback ...


3

Not sure if this would work well for your 400MB file, but here are some CLI one liners that would do the trick. If you're looking for entries for a specific date, grep -c can probably do what you need. Otherwise, you could probably use sed: sed -n '/date1/,/date2/p' filename For example with an input file "test": Day 0: foo Day 1: hello Day 2: world ...



Only top voted, non community-wiki answers of a minimum length are eligible