Tag Info

Hot answers tagged

40

Yes, we see a number of things like: while read line; do echo $line | cut -c3 done Or worse: for line in `cat file`; do foo=`echo $line | awk '{print $2}'` echo whatever $foo done (don't laugh, I've seen many of those). Generally from shell scripting beginners. Those are naive literal translations of what you would do in imperative languages ...


16

sed G is a well known one-liner for that. Performance-wise, the most effective with the standard Unix tool chest would probably be: paste -d '\n' - /dev/null If you don't want to add an empty line after the last line: sed '$!G' To add the empty lines before the input lines: paste -d '\n' /dev/null - Or: sed 'i\ /'


14

you simply do seq 1 n | xargs -n 5 echo n being the number you want to reach If your OS has bash but not seq, here is an alternative (thx to @cuonglm and @jimmyj for their remarks) echo {1..n} | xargs -n5 (you may have to be careful when reaching very high number with that one, depending on the OS and bash version, and if bash actually tried to expand ...


12

You can just use uniq with its -f option: uniq -f 4 input.txt From man uniq: -f, --skip-fields=N avoid comparing the first N fields Actually this will display the first line: [Fri Oct 31 20:27:05 2014] The Brown Cow Jumped Over The Moon If that is a problem you can do: tac input.txt | uniq -f 4 or if you don't have tac but your tail ...


11

Pure shell implementation: while read -r filename content ; do printf '%s\n' "$content" >> "${filename}.seq" done < /source/file


11

As far as conceptual and legibility goes, shells typically are interested in files. Their "addressable unit" is the file, and the "address" is the file name. Shells have all kinds of methods of testing for file existence, file type, file name formatting (beginning with globbing). Shells have very few primitives for dealing with file contents. Shell ...


10

You can use a vi script: $ vi test.txt -c '%s/aaa/NNN/ | wq' $ cat test.txt NNN NNN bbb ccc ddd You're simply automating what would normally be entered when using vi in command mode (accessed using Esc: usually): % - carry out the following command on every line: s/aaa/NNN/ - subtitute aaa with NNN | - command delimiter w - write changes to file q - ...


10

paste -sd '::\n' file ​​​​​​​ Or: awk '{ORS=NR%3?":":"\n";print}' < file (note the difference if the number of records in the input is not a multiple of 3 though).


10

A version in perl, using negative lookaheads: $ perl -0pe 's/\n(?!([0-9]{8}|$))//g' test.txt 20141101 server contain dump 20141101 server contain nothing {uekdmsam ikdas jwdjamc ksadkek} ssfjddkc * kdlsdlsddsfd jfkdfk 20141101 server contain dump -0 allows the regex to be matched across the entire file, and \n(?!([0-9]{8}|$)) is a negative lookahead, ...


9

Another option - ed line editor: ed -s test.txt <<< $',s/aaa/NNN/g\nw'


9

$ sed ':again;$!N;$!b again; :b; s/{[^{}]*}//g; t b' file3 This is that wants anyway. Explanation: :again;$!N;$!b again This reads in the whole file. :again is a label. N reads in the next line and $!N reads in the next line on the condition that we are not already at the last line. $!b again branches back to the again label on the condition that ...


9

shuf is the command you are looking for. From man shuf, -n, --head-count=COUNT output at most COUNT lines So, for example to get 4 random lines from the file, you could use the command as, shuf -n 4 file You could even use the below approach. head -$((${RANDOM} % `wc -l < file` + 1)) file | tail -1 Where, the final pipe to ...


9

If you're checking the content of the PATH environment variable, as opposed to looking for something in a file, then grep is the wrong tool. It's easier (and faster and arguably more readable) to do it in the shell. In bash, ksh and zsh: if [[ :$PATH: = *:/opt/gnome:* ]]; then : # already there else PATH=$PATH:/opt/gnome fi Portably: case :$PATH: in ...


8

Here's a hybrid perl/fold approach: $ echo "The cat hopped in a box." | fold -w 1 | perl -lne 'push @k, "$_ "; push @l,sprintf "%-2s",$.; END{print "@k\n@l"}' T h e c a t h o p p e d i n a b o x . 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Explanation fold -w 1: this will fold the ...


8

If you can guarantee that the identical lines will be consecutive, you can use uniq your_file and cuonglm's answer will work even if they're not.


8

Pure coreutils: $ seq 15 | paste - - - - - 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Pure perl: $ perl -e '@a=1..15; while($i<=$#a){print "@a[$i..$i+4]\n";$i+=5}' 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 GNU grep (shamelessly stolen from @1_CR): $ echo {1..15} | grep -oP '(\d+ ){4}\d+' 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 If you don't ...


8

You can do this with coreutils' sort with any of the following: sort -R file sort --random-sort file sort --sort=random file from man sort: -R, --random-sort sort by random hash of keys --sort=WORD sort according to WORD: general-numeric -g, human-numeric -h, month -M, numeric -n, random -R, version -V


8

You can use extended regular expressions by just using grep -E You have to match the beginning and the end of the path you are trying to find if you want to avoid false positives. Matches the instance at the beginning: $ TEST=/opt/gnome:/sbin:/usr/sbin:/var/opt/gnome $ echo $TEST | grep -E "(:|^)/opt/gnome(:|$)" /opt/gnome:/sbin:/usr/sbin:/var/opt/gnome ...


8

Using awk: awk '{printf "%s\n", $2>$1".seq"}' file From the nominated file, print the second field in each record ($2) to a file named after the first field ($1) with .seq appended to the name. As Thor points out in the comments, for a large dataset, you may exhaust the file descriptors, so it would be wise to close each file after writing: awk '{printf ...


8

My snap response would have been awk but if you're processing lots of lines —and I'm talking about With that in mind (and awk already being taken as an answer) I wrote a few implementations in different languages and benchmarked them on the same 10,000-line dataset on a PCI-E SSD. me* (C) 0m1.734s me (C++) 0m1.991s me ...


8

To select only lines modulo N with awk try awk '!(NR%2)' file or awk 'NR%3==0' file Here NR denotes number of rows processed so far. In your specific case (remove all lines with Y): $ echo 'YYYYYY XXXXXX XXXXXX YYYYYY XXXXXX XXXXXX' | awk '!(NR%3==1)' XXXXXX XXXXXX XXXXXX XXXXXX


8

If you have GNU sed, you could use the n~m (n skip m) address notation sed '1~3d' file which deletes every third line, starting at the first.


8

Try xargs -n2 < file | while read x y; do ((X+=x*y)); echo $X; done xargs -n2: groups numbers in pairs read x y: store first and second number in variables x and y ((...)) is just a arithmetic evaluation in bash You will see on the screen how sum is growing.


8

So you want to remove runs of 0s, but only when they are preceded by a colon. sed -e 's/:00*/:/g'


7

It's very simple to use the shell and basic tools: $ echo "The cat hopped in a box." | (read a ; echo $a | sed 's/./& /g' ; seq -ws" " 01 ${#a}) T h e c a t h o p p e d i n a b o x . 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Explanation The string is read into the $a variable. The ...


7

Using sponge: #!/bin/bash pattern='aaa' replacement='NNN' while read -r line do printf '%s\n' "${line//$pattern/$replacement}" done < "${1}" Call with: ./script.sh test.txt | sponge test.txt


7

You can use awk: awk '/^\*/ {sub(/\*/, ++i)} 1' <<END A list * one * two Blah * three END A list 1 one 2 two Blah 3 three


7

If you are not wed to grep, you can use awk and separate the records on : awk 'BEGIN {RS=":"} /^\/opt\/gnome$/'


7

I can answer with grep command: Input file: 9228 Hello 8473 World War 1 1914-1918 Hello 8391 World War 2 1939-1945 Command: grep -Eo '\<[0-9]{4}\>' file |tr '\n' ' ' Return any number with length=4. -E switches to extended regex -o print only the matching part Output: 9228 8473 1914 1918 8391 1939 1945 Update answer: Input file: 9228 ...


6

Here is something with find + wc + date. find . -maxdepth 1 -exec sh -c '[ -f "$0" ] && \ printf "%6s\t\t%s\t%s\n" "$(wc -l<"$0")" "$(date -r "$0")" "$0"' {} \; Instead of date -r one can also use for example stat -c%y. The output looks like this: 394 Thu Oct 16 22:38:14 UTC 2014 ./.zshrc 7 Thu Oct 30 11:19:01 UTC 2014 ...



Only top voted, non community-wiki answers of a minimum length are eligible