New answers tagged

0 votes

Determine maximum column length for every column in a simplified csv-file (one line per row)

Using Raku (formerly known as Perl_6) ~$ raku -ne 'BEGIN my %h; my @F = $_.split(","); map { %h{$_} = chars(@F[$_]) max %h{$_} }, 0..^@F.elems; ...
jubilatious1's user avatar
  • 3,255
0 votes

Count the maximum character length for all the data fields in a simplified csv file and output to txt

Using Miller (mlr) to calculate the max length of each field's values. The input is read as CSV, and the output is produced as an "xtab" file (one key+value pair on each): $ mlr --c2x ...
Kusalananda's user avatar
  • 335k
0 votes

Count the maximum character length for all the data fields in a simplified csv file and output to txt

Using Raku (formerly known as Perl_6) ~$ raku -ne 'BEGIN my @a; unless ++$ == 1 { @a.push: $_.split(",").map: *.chars; }; ...
jubilatious1's user avatar
  • 3,255
4 votes
Accepted

How can I use $0 in Makefile?

You were on the right path; to escape the dollar symbol in a Makefile, you need to use $$, not \$: copy_files: sed 's/^/\./' ./input_file | \ rev | \ LC_ALL=C sort -u | \ ...
Stephen Kitt's user avatar
1 vote

How to create grep output in the order of the input file?

Using Raku (formerly known as Perl_6) #taking scrambledA.txt inline: ~$ raku -ne 'BEGIN my @scrambled = <efrrsu aehmmr aeeglnr alnors acflno aaahmy aceimru 1469en aelprsy cehrry>; ...
jubilatious1's user avatar
  • 3,255
3 votes

How to create grep output in the order of the input file?

A common technique when you need to preserve the order of a list is to add numbers to it before processing. Arrange to preserve the numbers and use them to sort after processing, then you can cut the ...
Gilles 'SO- stop being evil''s user avatar
0 votes

Delete range of lines above pattern with sed (or awk)

Using Raku (formerly known as Perl_6) ~$ raku -ne 'BEGIN my @lines; push @lines, $_; splice @lines, 0, 3 if /banana/; put shift @lines if @lines > 2; ...
jubilatious1's user avatar
  • 3,255
2 votes

How to create grep output in the order of the input file?

You can control the order of search by putting your command in a loop: $ while IFS= read -r term; do grep -F "${term}" wordlistC.txt || echo "not found: ${term}"; done < ...
user9101329's user avatar
  • 1,117
1 vote

Display occurrences and count lines between them and after the last one

Using Raku (formerly known as Perl_6) ~$ raku -ne 'BEGIN my $c = 0; if /^table/ { $c && put $c; .put; $c = 0; next}; $c++;' file Raku is a programming language in the Perl-family that ...
jubilatious1's user avatar
  • 3,255
13 votes

How to create grep output in the order of the input file?

Strictly speaking, you are getting the words in the order they occur in your input file. It's just that your "input file" to grep is the wordlist file, not your list of scrambled words. ...
Kusalananda's user avatar
  • 335k
9 votes

How to create grep output in the order of the input file?

Provided that your wordlistC.txt can be read entirely into memory you can descramble your scrambledA.txt using awk: awk -F'-' ' # Save lookup key/value FNR==NR { w[$1]=$2 } # Lookup or ...
Chris Davies's user avatar
0 votes

How to fetch fasta sequences corresponding to header lines in another file

Using Raku (formerly known as Perl_6) ~$ raku -e 'my @query = "/path/to/fastaIDs.txt".IO.lines; .grep(/@query/).print for lines.join("\n").split(/^^ <?before \> ...
jubilatious1's user avatar
  • 3,255
2 votes

Display occurrences and count lines between them and after the last one

My answer assumes a scenario in which there can be empty tables ("empty table" lines / matching lines) and a generalized scenario in which there might be extra (non-table / non-matching) ...
Aeronautix's user avatar
4 votes

Display occurrences and count lines between them and after the last one

I put together a Perl solution:. perl -Mfeature=say -e ' while (<>) { if (/^table/) { $c && say $c; print; $c = 0; next; ...
kos's user avatar
  • 2,977
5 votes
Accepted

Display occurrences and count lines between them and after the last one

Clearly, you’re 90% of the way there: awk '/table/ {if (n) print NR-1-n; n=NR; print} END {if (n) print NR-1-n}' file You don’t need $0 ~; that’s implied.
G-Man Says 'Reinstate Monica''s user avatar
0 votes

How to replace lines which has no data with the previous line's data

Using Raku (formerly known as Perl_6) ~$ raku -e 'my @init = [Z=>] lines[0].words, (DateTime.new("2024-01-01T00:00:00Z"), "0.00", "Initial"); put @init....
jubilatious1's user avatar
  • 3,255
0 votes

How to replace lines which has no data with the previous line's data

Using Miller (mlr) to keep track of the timestamps and values from each record in the tab-delimited input, and insert an incremented timestamp and a new value in any record that has a NoValue flag: $ ...
Kusalananda's user avatar
  • 335k
1 vote

Remove subdomains or existing domains

$ sed -e 's/\./\\./g' -e 's,.*,/\\.&$/d,' file | sed -f /dev/stdin file google.com bgoogle.com yahoo.co.uk stats.wikipedia.org media.wikipedia.org bible.com This first uses sed to create a sed ...
Kusalananda's user avatar
  • 335k
0 votes

Remove subdomains or existing domains

This might be what you're trying to do, using any awk: $ cat tst.sh #!/usr/bin/env bash awk ' { doms[$0] } END { for ( dom in doms ) { n = gsub(/[.]/,"&",dom)...
Ed Morton's user avatar
  • 31.7k
0 votes

Remove subdomains or existing domains

Using Raku (formerly known as Perl_6) ...translating @terdon's excellent Perl solutions: ~$ raku -ne 'BEGIN my %k; %k{$0}++ for s/ .*? (<-[.]>+ \. <-[.]>+ $)/$0/; END .key.put for %k;' ...
jubilatious1's user avatar
  • 3,255
1 vote

Remove subdomains or existing domains

It sounds like your problem can be reformulated as "only keep the longest non-. followed by a . and the no other . until the end of the line; make these unique". If so, you can do something ...
terdon's user avatar
  • 242k
0 votes

Remove lines from csv file

Thank you all for your help. I've found a solution that works for me: grep -v "path\\\\1.-.folder.01\\\\1.-.folder2\\\\"
Hugo's user avatar
  • 11
0 votes

Remove lines from csv file

In awk you can use command like: awk -F\\ 'NF<4{ print}' input_file This will use \ as delimiter and count the number of field. And print if fields are less than 4 P.S. As you edit the post and ...
Romeo Ninov's user avatar
  • 17.5k
0 votes

Remove lines from csv file

Using Raku (formerly known as Perl_6) ~$ raku -ne '.put unless /^ \\\\ path \\ [[ <digit>+ " - "]? folder \s? <digit>* ]**3..* \% "\\" /;' file Raku is a programming ...
jubilatious1's user avatar
  • 3,255
3 votes

Remove lines from csv file

$ { head -n 1; grep -E '^.{32};'; } < file "FolderName";"ADGroupOrUser";"Permissions" \\path\1 - folder 01\1 - folder2; user.user; Modify, Synchronize \\path\1 - ...
Ed Morton's user avatar
  • 31.7k
-3 votes

Remove lines from csv file

if this work is to be done on a one-off, I would import the csv into MS Excel or LibreOffice (import file separated by ';' and use a trivial filter to delete those row that count the '' char in the ...
Andrew001's user avatar
0 votes

Grep variable number before / after string from textline

You can use grep and tr: $ grep -o 'is.* days' log.txt | tr -dc '[0-9]\n' 123
user9101329's user avatar
  • 1,117
0 votes

Extracting json fields in a log line using awk or some other cli tool

Using Raku (formerly known as Perl_6) ...with Raku's JSON::Tiny module: ~$ raku -MJSON::Tiny -ne 'my %json = from-json($_.comb( / "\{" ~ "\}" .* / )); ...
jubilatious1's user avatar
  • 3,255
0 votes

Comparing Indices of two strings

paste -d : <(fold -w 1 <<<"$con") <(fold -w 1 <<<"$prot") | sed -n 's/^*://p' This first produces a set of two columns delimited by : where the 1st column ...
Kusalananda's user avatar
  • 335k
1 vote

How do I delete the same line numbers across all files in a directory?

You are on the right track. Just need to edit your command a little. You probably wanted to use xargs (instead of exec). Here is one version that should work: $ find . -type f | xargs -I{} sed -i '151,...
user9101329's user avatar
  • 1,117
1 vote

How to remove embedded newlines from CSV fields

Using TXR: $ txr fix.txr data "host1","host1","linuxserver","" "host2","host2","linux server","" "host1","...
Kaz's user avatar
  • 8,476
3 votes

How to remove embedded newlines from CSV fields

Using any awk: $ awk 'BEGIN{RS=ORS="\""} !(NR%2){gsub(/\n/," ")} 1' file.csv "host1","host1","linux server","" "host2","...
Ed Morton's user avatar
  • 31.7k
4 votes

How to remove embedded newlines from CSV fields

Using Raku (formerly known as Perl_6) Inspired by @terdon's excellent Perl answer: ~$ raku -ne '/ <!after \" > $/ ?? print "$_ " !! put $_;' file Here are answers written in ...
jubilatious1's user avatar
  • 3,255
2 votes

How to remove embedded newlines from CSV fields

We assume that despite many unwanted line breaks all double quotes match and all field separators are present. In that case you could use this command: $ sed '/^"/! s/^/ /' infile | tr -d '\n' | ...
user9101329's user avatar
  • 1,117
6 votes
Accepted

How to remove embedded newlines from CSV fields

The last thing you want to do is try to do this in bash. See Why is using a shell loop to process text considered bad practice?. Now, if what you want can be expressed as "remove any newline ...
terdon's user avatar
  • 242k
7 votes

How to remove embedded newlines from CSV fields

Using Miller (mlr), a CSV-aware multi-purpose processing utility for various structured document formats, to clean up the whitespace of all fields: $ cat file "host1","host1","...
Kusalananda's user avatar
  • 335k
0 votes

How to remove embedded newlines from CSV fields

Some modern implementations of awk actually bring a fully-fledged csv parser (I think it's GoAWK?), yours probably doesn't. GNU awk starting from 5.3 have the --csv flag. You would need to have a very ...
Marcus Müller's user avatar
0 votes

Histogram with FASTA file

It is more sensible to use a proper graphical tool for histogram. But if you simply want to replace scaffolds with some length representaion you could try this awk command: $ awk '{if (NR%2){printf $0&...
user9101329's user avatar
  • 1,117
0 votes

How to fetch fasta sequences corresponding to header lines in another file

The following awk command will process both files and output only those records from the file2 whose headers are present in file1: awk 'BEGIN{FS="\n";RS=">";ORS=""} (...
AdminBee's user avatar
  • 22.8k
0 votes

How to fetch fasta sequences corresponding to header lines in another file

This awk command might do what you want: $ cat file1 | xargs -I{} awk -v tag={} '$0==tag{p=1; print; next} /sp/{p=0}p' file2 >sp|B7UM99|TIR_ECO27 MPIGNLGNNVNGNHLIPPAPPLPSQTDGAA ...
user9101329's user avatar
  • 1,117
0 votes

replace header in a file with list of lines in another file

It is good to see that old questions are still answered years later! This might be a lot easier with awk: $ awk '/sp/{getline nuhead <"file2";$0=nuhead}1' file1 sp|B7UM99|TIR_ECO27OS=...
dhm's user avatar
  • 552
0 votes

Listing a certain part of filenames and their contents

As the system is using busybox, I assume it won't have things like perl. busybox has a very capable awk implementation though, so you should be able to do: find . -name '*_*_*_*.config' -type f -exec ...
Stéphane Chazelas's user avatar
1 vote

replace header in a file with list of lines in another file

With the GNU implementation of sed: $ sed -e '/^sp|/{R file2' -e 'd}' file1 sp|B7UM99|TIR_ECO27OS=Escherichia coli MPIGNLGNNVNGNHLIPPAPP..... sp|P0ACF8|HNS_ECOLI=Human MSEALKILNNIRTLRAQ........ sp|...
Stéphane Chazelas's user avatar
1 vote

replace header in a file with list of lines in another file

You can do that with sed and paste commands as follows: $ sed 's/$/\n/' file2 | paste -d ' ' file1 - | sed 's/^sp.* sp/sp/' sp|B7UM99|TIR_ECO27OS=Escherichia coli MPIGNLGNNVNGNHLIPPAPP..... sp|P0ACF8|...
user9101329's user avatar
  • 1,117
0 votes

sed "invalid command code W"

This might be easier with awk. Put your prepend text lines in a file (let's call it addit.txt) and use this loop: $ for file in *.vtt ; do awk -i inplace 'NR==1{system ("cat addit.txt")}1' &...
user9101329's user avatar
  • 1,117
1 vote

Script to add a block in a few hundred files

Here is a late answer using sed: $ sed '/location/ r addy' nginx server { location / { if ($http_user_agent ~* "somewebsite" ) { return 444; } } } where: $ cat addy if ($...
user9101329's user avatar
  • 1,117
0 votes

Can grep output only specified groupings that match?

Compare Perl and Raku solutions: Using Perl (answers from @vault and @Gilles 'SO- stop being evil'): ~$ perl -lne 'print $1 if /^foobar (\w+)/;' file #OR: ~$ perl -lne 's/^foobar\s+// or next; s/\...
jubilatious1's user avatar
  • 3,255
0 votes

Parsing FIX messages for further analytics

Using awk you can do: $ awk -F= '{for (i=1;i<=NF;i++) if($i ~ "NOK" || $i ~ "USD"){print $i,$(i+2)}}' input_file | awk '{gsub(/[0-9]/,"",$1)}1' NOK/SEK 1.0735135 USD/...
user9101329's user avatar
  • 1,117
0 votes

Pick off 'first' dates using cal function in bash

In ksh93: $ printf '%(%F)T\n' 'first monday this month' 2024-03-04 $ printf '%(%F)T\n' 'second tuesday this month' 2024-03-12 $ printf '%(%F)T\n' '2nd tuesday in 2016-03' 2016-03-08 (note that 4th, ...
Stéphane Chazelas's user avatar
0 votes

Pick off 'first' dates using cal function in bash

Both gcal (GNU) and ncal (BSD) families support options for transposing rows with columns: $ ncal March 2024 Su 3 10 17 24 31 Mo 4 11 18 25 Tu 5 12 19 26 We 6 13 20 ...
bishop's user avatar
  • 3,209

Top 50 recent answers are included