New answers tagged

0

Unix have these wonderful little tools called cut and paste. The cut tool will extract a set of columns from its input while the paste tool inserts columns. We will be using these. I'm not going to care about your pipeline too much (right now, but see the end of this answer), I'm just concerned with the problem of switching the columns around. Let's say I ...


0

Even though I gave my previous answer, I have looked at the sub-problem of extracting a particular sub-sequence out of a fasta file. The solution is in two parts: An sh shell script that does the command line parsing, and calls... An awk script that does the parsing of the fasta file. I decided to post this here because it shows How to do command line ...


0

This question requires a larger programming effort than may be offered by this forum (I do this kind of programming for a living). The DDBJ/ENA/GenBank file format (the first file in the question) is complex and allows CDSs (the coding parts of a genomic sequence) to be not just plain or joined, but complemented and combinations thereof. Furthermore, the ...


2

If you want to use one of those tools you mentioned, you can do it with sed: echo "/Folder/F1/F2/F3/F4/.../Fn/abc.txt" | sed "s/\/[^/]*$//" sed -i "s/\/[^/]*$//" path.txt If you are in bash, you can use Bash parameter expansion for this: path=$(cat path.txt) echo ${path%/*} And as mentioned by ikkachu you can use dirname.


2

basename and dirname print the last and all-but-last parts of a filename given as argument, so: $ dirname "/Folder/F1/F2/F3/F4/.../Fn/abc.txt" /Folder/F1/F2/F3/F4/.../Fn (or from a file: dirname $(< pathname.txt) )


0

myvariable=$(yourmysterycommand | sed -e 's/id1=//; s/id5=.*//; s/"//g; s/id.=/./g; s/[[:blank:]]\+//g' | paste -sd' ') That's a one-liner, but reformatted with extra newlines and indentation for readability. $ echo "$myvariable" 172....


5

You can do this with an awk command most easily: your-command | awk -F\" -v OFS=. -v ORS=' ' '{print $2, $4, $6, $8}' To set it as a variable, use command substitution: myVariable="$(your-command | awk -F\" -v OFS=. -v ORS=' ' '{print $2, $4, $6, $8}')" -F sets the input field separator (which is by default any whitespace) to a custom value; in this ...


2

You can also take advantage of the fact that the shell strips newlines from command substitutions. So, instead of find $PWD -type f -name "file.txt" > paths.txt, you can do (note that you don't need the $PWD, it is the default value for find): echo $(find $PWD -type f -name "file.txt") > paths.txt or printf '%s ' $(find $PWD -type f -name "file.txt"...


7

One more way, assuming GNU find(1), just for fun: find $PWD -type f -name "file.txt" -printf '%p '


4

You can use, paste too, find . -type f -name "path.txt" -exec paste -d' ' -s {} \; > path.txt


4

One simple way would be to pipe the find output through xargs (whose default action when no explicit command is given is to echo its arguments) find $PWD -type f -name "file.txt" | xargs > paths.txt Unlike simply replacing all the newlines with spaces, this preserves the final newline.


5

You can replace the LF character with a space using the 'tr' command tr '\012' ' ' < path.txt This can be part of the original command: find $PWD -type f -name "file.txt" | tr '\012' ' ' > paths.txt


0

This is the completely unquoted version: grep ^\\*\\* test.out. To pass a literal backslash from the shell to grep, it needs to be escaped. This works as long as you have no files in the directory starting with ^\ and containing another backslash.


0

Check the --output-delimiter option of cut, i.e. cut -f 1 -d ' ' --output-delimiter='\n' in your case


2

Here's one way to do it with sed: sed '/Os version rhel5\.6/{ a\ apache 4.2 $!{ n /^apache 4\.2$/d } }' infile This appends apache 4.2 unconditionally to all lines matching Os version rhel5.6 then (if not on the last line) it pulls in the next line via n (printing the pattern space) and if the new pattern space content matches apache 4.2 it deletes it. ...


2

This perl expression will do the trick, perl -i -ne 'next if /apache 4.2/;s+Os version rhel5.6+Os version rhel5.6\napache 4.2+; print' ssss Explanation next if /apache 4.2/ skips any lines matching apache 4.2. s+Os version rhel5.6+Os version rhel5.6\napache 4.2+; print search Os version rhel5.6 and replaces line with same with appending apache 4.2 at ...


1

perl -i -p -e 'if ($. == 1) {s/^\s+#!/#!/}' * This will remove leading white space before #! ONLY on the first line ($. == 1) of every file. all other lines are passed through unmodified. perl will update the files whether anything has changed or not (i.e. they'll have a new inode and timestamps will be updated). See man perlrun and search for the ...


2

You were on the right track to use awk. You should write a script that reads your logs, and outputs with the fields separated with tabs¹. Then use the column command to re-align the columns: extract.awk²: BEGIN {OFS="\t"; print "Timestamp\tEmailTo:\tEmailFrom:\tIPAddress:\tErrorCodes:"} {print $1, $6, $7, $NF, $(NF-5)} Then run it with this command: awk ...


3

It's not the shell None of the answers so far has touched on the real problem. It would be helpful to explain why it does not work as you expect. grep -i "^**" test.out Because you have quoted the pattern to grep, * is not expanded by the shell. It is passed to grep as-is. This is explained in the manual page[1] for bash[2]: Enclosing characters in ...


1

You can do it like this: $ MYVAR='<li class="rainbow-list-item southern disrupted " data-line-id="lul-southern" data-severity-codes="4,6,9" data-mode="national-rail">' $ echo $MYVAR | tr ' ' '\n' | awk -F= '/data-severity-codes/ {print $2}' "4,6,9" You can also use sed to remove " and replace , by a blank if you need it: $ echo $MYVAR | tr ' ' '\n' ...


1

You can use the matching operator =~: #! /bin/bash var='<li class="rainbow-list-item southern disrupted " data-line-id="lul-southern" data-severity-codes="4,6,9" data-mode="national-rail">' if [[ $var =~ data-severity-codes=\"([^\"]+)\" ]] ; then echo ${BASH_REMATCH[1]} fi Output: 4,6,9 Or use an XML-aware tool: xmllint --xpath li/@data-...


7

As you wanted to check the line which starts with ** and ends with ), you can combine two grep operation like this, grep '^*\*' test.out | grep ')$' Or with single grep command like this, grep -E '^\*\*.*\)$' test.out Explanation ^\*\* : match line which starts with ** .* : match everything after ** \)$ : match line which also has ) at the end of ...


2

Other options. You can use sed or awk also $ sed -n '/^*\*/p' test.out $ awk '/^*\*/' test.out To know lines that end with ) use also grep or sed or awk $ grep ')$' test.out $ sed -n '/)$/p' test.out $ awk '/)$/' test.out


12

Use the \ character to escape the * to make it a normal character. grep '^\*\*' test.out Also note the single quote ' and not double quote " to prevent the shell expanding things


4

Assuming there's a file called dates containing the list of dates, one per line (and nothing else), something like this might work to count the ones older than 14 days: $ date=$(date --date="14 days ago" +%Y%m%d) $ awk '($0 < "'$date'") {count += 1} END {print count}' < dates 20 (Given they are in yyyymmdd format, the comparison is easy.)


1

I used a combination of grep and sed to accomplish your goal of after \n( and before )\n grep -o '\\n(.*)\\n' test.txt|sed -e 's/\\n//g' Sample output (T.a = 1) (public.cde.newcol = \'013\')


1

You can accomplish this with sed like so: sed -e 's,\\n(.*,,' unset or with awk you have to do a lot of escaping awk -F'\\\\n\\(' '{print $1}' unset to get the right escaping for both the \ before the n and to protect against the special interpretation of the (


3

With gnu sed/shuf: sed '1b;s/^*/\x0*/' infile | shuf -zn 5 | tr -d '\000' This turns input into nul separated records i.e. on each line that starts with a * (except for the first one) it adds a nul char before the * then uses shuf with --zero-terminated switch to extract five random records and tr to delete those nul chars.


2

That's quite simple: sed -i "s/\[ \]/[$(cat output.txt | xargs)]/" input.txt


2

cat listfile | tr '\n' , | sed 's/,\*/\n*/g;s/,$//' | shuf | head -n 5 | tr , '\n'


1

As others have pointed out, grep isn't the best tool for this. If you insist on using it, and if your grep supports the -o (only print the matched portion of the line) and -P (use Perl Compatible Regular Expressions), you can do this: $ grep -oP '^[^:]+|.*:\K[^:]+(?=:[^:]+)' /etc/password terdon /home/terdon bob /home/bob Note that this will print all ...


0

If you want to pass the output to less you don't need a sed command at all, just set how less should highlight searched pattern (e.g. in .bashrc): export LESS_TERMCAP_so=$'\e[93m' # begin standout mode export LESS_TERMCAP_se=$'\e[m' # end standout mode and then run less with -p option: less -p '.*DATA.*' file


3

Perhaps you meant to use less on the output of sed, rather than the reverse: sed -e 's/\(.*DATA.*\)/\o033[93m\1\o033[39m/' file | less -R Further reading: less - opposite of more


17

You can use cut to split files with columns on a specific delimiter: cut -d: -f6 /etc/passwd Or -f1,6 for columns (fields) 1 and 6.


9

Grep is really not the tool for parsing out data this way, grep is more for pattern matching and you're trying to do text-processing. You would want to use awk. awk -F":" '$7 == "/bin/false" {print "User: "$1 "Home Dir: "$6}' /etc/passwd awk The command -F":" Sets the data delimiter to : $7 == "/bin/false" Checks if the 7th data column is /bin/false {...


2

You can accomplish this with awk awk '{sub(/".*",/,"");print}' filename


1

you can give multiple instruction in one shot of sed, for example: sed 's/\t/ /g;/^ *$/d;s/^#/NODIESE/' testfile this single line replace tab with space delete line that start with empty stuff (or empty line) replace Dash at start with the word NODIESE so your test file is processed only once and you launch sed only one time.


1

POSIX Awk; this works with an arbitrary amount of files, and the files don’t even have to have the same amount of lines. The script keeps going until all files are out of lines: BEGIN { do { br = ch = 0 while (++ch < ARGC) if (getline < ARGV[ch]) { printf ch < ARGC - 1 ? $0 FS : $0 RS br = 1 } } while (br) }


0

You could try abiword software. e.g: abiword --to=doc example.pdf


0

with GNU sed sed -r 's/^([^.]+)\.[0-9]+ /\1 /' filename ^([^.]+) capture starting string upto first dot character \.[0-9]+ match dot character followed by more than 1 digit characters and if number of characters is consistent as given in the example, sed -r 's/^(.{8}).{7} /\1 /' filename


2

I would use awk awk --posix '{ gsub(/\.[[:digit:]]{6}/, "", $1); print }' filename Will target the first field (space delimited) and search for a . followed by 6 numbers and empty it out.


1

With awk probably the simplest way is to do a regular expression substitution on the first whitespace-separated field, replacing everything from the period to the end of the field: awk '{sub(/\..*/,"",$1)}1' somefile


0

I tend to use Perl. sed or awk is fine in this case, but sometimes the flexibility is Perl is useful. perl -pi -e 's/^(mynetworks.*)/$1 0.0.0.0\/0/' /path/to/file or if this is in a pipe chain cat file | perl -pe 's/^(mynetworks.*)/$1 0.0.0.0\/0/'


1

To add the specified text to a line in the file - if that line is the only one that starts with mynetworks, you can do this: sed --in-place '/^mynetworks/s_.*_& 0.0.0.0/0_' /path/to/file


3

Using sed sed -i 's+^mynetworks.*+& 0.0.0.0/0+' log.txt Using awk awk '/^mynetworks/ {$0=$0" 0.0.0.0/0"} 1' log.txt or awk '{if ($1 ~ /^mynetworks/) print $0, "0.0.0.0/0"; else print $0}' log.txt Using bash while read -r line ; do [[ $line == mynetworks* ]] && line+=" 0.0.0.0/0" echo "$line" done < log.txt


0

The join utility merges lines of two files based on a common column. It requires the files to be sorted on that column. join -t $'\t' -1 10 -2 1 -o 2.1,2.2,1.7 <(sort -t $'\t' -k10 file1) <(sort -t $'\t' file2) $'\t' is a tab character, -t $'\t' says to use that as the field separator. join -1 10 -2 1 means to join lines when the field 10 of the ...


2

A grep solution: $ echo "2.5 test. test -50.8" | tr ' ' '\n' | grep -E '^[+-]?[0-9]*\.?([0-9]+)$' 2.5 -50.8 The tr just converts the line into multiple lines by replacing the spaces with newlines. The grep command looks for strings that starts with an optional + or -, possibly followed by some digits and an optional decimal point. Then we require some ...


4

grep works well for this: $ echo "2.5 test. test -50.8" | grep -Eo '[+-]?[0-9]+([.][0-9]+)?' 2.5 -50.8 How it works -E Use extended regex. -o Return only the matches, not the context [+-]?[0-9]+([.][0-9]+)?+ Match numbers which are identified as: [+-]? An optional leading sign [0-9]+ One or more numbers ([.][0-9]+)? An optional period followed ...


0

With perl you can have this way, #!/usr/bin/perl use strict; use warnings; my $file=$ARGV[0]; open my $fId, '<', $file or die "Error opeining file <$file>."; my @qty = (); my @price = (); my @discount = (); while(<$fId>){ chomp; my @fields = split (/\t/, $_); push @qty , $fields[1] if $fields[0] =~ m/Quantity/; ...


0

Everything can be done within single awk command by building big array with all data, but if the file is very big you may have problems with available memory. Thus I would do this in several steps: header=$(awk '{print $1}' file | uniq | tr '\n' ',') printf "${header%?}\n" > output paste -d, <(awk '$1=="Quantity"{print $2}' file) \ <(awk '...



Top 50 recent answers are included