Tag Info

Hot answers tagged

27

You'd think there'd be a utility for that, but I couldn't find it. However, this Perl one-liner should do the trick: perl -pe 's/\e\[?.*?[\@-~]//g' Example: $ command-that-produces-colored-output | perl -pe 's/\e\[?.*?[\@-~]//g' > outfile Or, if you want a script you can save as stripcolorcodes: #! /usr/bin/perl use strict; use warnings; while ...


16

The POSIX 2008 standard has a section describing "Shell and Utilities". Generally, if you stick to that your scripts should be fairly future-proof, except possibly for deprecations, but those hardly happen overnight so you should have plenty of time to update your scripts. In some cases where output format for a single utility varies widely across ...


15

Remove color codes (special characters) with sed sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" Or Strip ANSI escape sequences in Python Install colorama python package (pip install colorama). Put into stripcolorcodes: #!/usr/bin/env python import colorama, fileinput, sys; colorama.init(strip=True); for line in fileinput.input(): ...


12

cat ./text | awk ' { if ( length > x ) { x = length; y = $0 } }END{ print y }' UPD: summarizing all the advices in the comments awk 'length > max_length { max_length = length; longest_line = $0 } END { print longest_line }' ./text


12

I'll try to answer from my experience. Commands don't really adhere to a formal specification, but they do adhere to a requirement to consume and generate line-oriented text. Yes, of course. Before the GNU utilities became a de facto standard, a lot of vendors would have quirky output, especially with respect to ps and ls. This caused a lot of pain. ...


9

First, very brief answers to your questions: Formal standardization of input/output conventions: no Breakage in the past due to changing output: yes Absolutely impossible to break future filters: no How can I protect myself against changes: be conservative When you say "API", you're using a term that (for good or ill) implies too much formality around ...


7

Instead of matching based on the URL, try matching based on contents of the certificate. iptables -t nat -I INPUT --sport 443 -m string \ --string www.facebook.com --algo bm -j REJECT You can also match on the fingerprint but if the destination changes or updates their certificate, it will invalidate your rule.


7

awk -F '","' 'BEGIN {OFS=","} { if (toupper($5) == "STRING 1") print }' file1.csv > file2.csv Output "12310","42324564756","a simple string with a , comma","string with or, without commas","string 1","USD","12","70%","08/01/2013","" "23525","74535243123","string , with commas, and - hypens and: semicolans","string with or, without commas","string ...


5

sed -rn "/.{$(<file expand -t1 |wc -L)}/{p;q}" file This first reads the file inside the command substitution and outputs the length of the longest line, (previously, expand converts tabs to spaces, to overcome the semantics of wc -L -- each tab in the line will add 8 instead of 1 to line length). This length is then used in a sed expression meaning ...


5

The firewall cannot control which HTTPS URLs the client is trying to access, because the URL is encrypted. The firewall can only control which sites the client is connecting to, using IP addresses, but this doesn't help if the HTTP and HTTPS versions of the site are at the same URL (and even if they aren't, you'd have to maintain a huge list of IP ...


5

With -d " ", the field separator is one (and only one) space character. Contrary to the shell word splitting, cut doesn't treat space any different than any other character. So cut -d " " -f2 returns "" in root   19, just like it would return "" for cut -d: -f2 in root:::19. You'd need to either squeeze the blanks to transform any sequence of space into one ...


5

One of the particularity of those processes is that they are not backed by an executable file, so you could do (in zsh): ps /proc/[0-9]*/exe(^-@:h:t) Or with any POSIX shell: ps -p "$(find -L /proc/[0-9]*/exe ! -type l | cut -d/ -f3 | paste -sd, -)" That is check for processes whose /proc/<pid>/exe is a link to a file. But that means you need to ...


5

One way to recognize kernel processes is that they don't use any user memory, so the vsz field is 0. This also catches zombies (thanks to Stephane Chazelas for this observation), which can be eliminated based on their status. ps axl | awk '$7 != 0 && $10 !~ "Z"' To list just the PIDs: ps -e -o pid= -o state= -o vsize= | awk '$2 != "Z" && ...


5

use classic regex: grep -i 'spider\|bot' or extended regex (or even perl regex -P): grep -Ei 'spider|bot' or multiple literal patterns (faster than a regular expression): grep -Fi -e 'spider' -e 'bot'


4

Only covering 1) of your question. Naturally APIs can always change at the will of their creators, and thusly break dependent software, in any language. That said, the great idea of the Unix tools' I/O "APIs" is that there is practically none (maybe 0x0a as line end). A good script filters data with the Unix tools instead of creating it. That means that ...


4

With GNU ls (the version on non-embedded Linux and Cygwin, sometimes also found elsewhere), you can exclude some files when listing a directory. ls -I 'temp_log.*' -lrt With zsh, you can let the shell do the filtering. Pass -d to ls so as to avoid listing the contents of matched directories. setopt extended_glob # put this in your .zshrc ls ...


3

With sed it is more concise: $ find -maxdepth 1 -type f | sort | sed '/^\.\/reference/,$d' Which means (after sorting) delete the reference line (or greater) and all lines following to the last line. The sed 'd' command is used here with an address range, where '/^./reference/' is the start and '$' is the end of the range. (And '$' as an address means ...


3

eg. script-name "$HOME" "reference" ... find doesn't always output the leading ./, as in the case of find bin, or find /tmp. So if you only want the file base-names, this works. update: Added tolower() to allow for a case insensitive comparison, which produces the alphabetical collation mentioned in the question... #!/bin/bash dir="$1"; ref="$2" ...


3

I know of one option. If you have internal DNS servers for use, then put some static references in your TLD zone data that resolve the domains (that you do not wish to establish the outside connections) to just 127.0.0.1. This way, all the hosts using the central DNS within your network will resolve ( facebook.com/twitter.com per se) domains into loopback ...


3

OK, if the length of the columns is not known, I'd switch to a more powerful language than bash: #!/usr/bin/perl use warnings; use strict; my $string = shift; open my $FH, '<', '1.txt' or die $!; my $first_line = <$FH>; my ($before, $name) = $first_line =~ /(.* )(NAME *)/; my $column = length $before; $string .= ' ' x (length($name) - length ...


3

Not sure if this would work well for your 400MB file, but here are some CLI one liners that would do the trick. If you're looking for entries for a specific date, grep -c can probably do what you need. Otherwise, you could probably use sed: sed -n '/date1/,/date2/p' filename For example with an input file "test": Day 0: foo Day 1: hello Day 2: world ...


2

If the field widths are constant - i.e. the file format you've shown with the field widths you have are at their maximum - you can use GNU awk (gawk(1)) and set the FIELDWIDTHS variable to use fixed width parsing: gawk -v searchstr="Ideas worth zero" -- ' BEGIN { FIELDWIDTHS="6 15 27 5" } # assuming the final field width is 5 # Pre-process data ...


2

The following example was going to be, and should have been, a comment to dmitry.malikov's answer, but because of the Useless Use of Visible Comment Space there, I've chosen to present it here, where it will at least be seen... This is a simple variation of the dmitry's single-pass awk method. It prints all "equal longest" lines. (Note. delete array ...


2

In pure bash: #!/bin/bash _max_length=0 while IFS= read -r _line; do _length="${#_line}" if (( _length > _max_length )); then _max_length=${_length} _max_line=( "${_line}" ) elif (( _length == _max_length )); then _max_line+=( "${_line}" ) fi done printf 'Max line length: %d\n' "${_max_length}" printf 'Lines ...


2

With iptables firewall this works (Openwrt also uses iptables): iptables -t nat -A PREROUTING -s 192.168.1.0/24 -p udp --dport 53 -j DNAT --to 192.168.1.1 iptables -t nat -A PREROUTING -s 192.168.1.0/24 -p tcp --dport 53 -j DNAT --to 192.168.1.1 On your router use Opendns servers. 192.168.1.1 is the Openwrt router ip. 192.168.1.0/24 is the LAN network ...


2

Obligatory zsh answer, using the . glob qualifier to select only regular files and e to further select among matches: echo *(.e\''[[ $REPLY > reference ]]'\')


2

The top level of procmail recipes are reserved for assignment of procmail variables. Add the following to the top of your procmail recipe. MAILDIR="$HOME/Maildir/" When defining where the mail should be delivered, you have defined Xyz as a file, not a directory. It should instead read: :0: * ^TO_myemailaddress@domain.com Xyz/ This is one of my favorite ...


2

You can address Case #1 like this with GNU sed: sed -r ':a; /^\s*$/ {N;ba}; s/( *\n *){2,}/\n\n/' That is, collect empty lines in pattern space, and if there are more than three or more lines, reduce it to two lines. To join single-spaced lines, as in Case #2, you can do it like this: sed -r '/^ *\S/!b; N; /\n *$/!b; N; /\S *$/!b; s/\n *\n/\n/' Or in ...


1

I agree with the comment that dialog is probably where you need to start. To show you how you could use it, here is an example script #!/bin/bash #make some temporary files command_output=$(mktemp) menu_config=$(mktemp) menu_output=$(mktemp) #make sure the temporary files are removed even in case of interruption trap "rm $command_output; rm ...


1

If it were the third element, that would be "SynPS/2", not "SynPS/2 Synaptics TouchPad" The best you can achieve here is report what's before id= as it's the only thing that can identify where the device name ends. xinput list | sed -n 's/^[ ⎜↳]*\(.*[^[:blank:]]\)[[:blank:]]*id=.*slave pointer.*/\1/p' Or with GNU grep built with PCRE support (-P ...



Only top voted, non community-wiki answers of a minimum length are eligible