New answers tagged

3 votes

Show changes between consecutive lines of a file

awk '{ bak=$0; for(i=1; i<=NF; i++)$i=($i==tmp[i]?"-":$i) split(bak, tmp) }1' infile cpu1 cpu2 cpu3 cpu4 5 3 3 6 - - - - - 0 - - 3 - 6 - 5 3 3 0 To keep the records indentation (...
user avatar
  • 38.7k
0 votes

Show changes between consecutive lines of a file

Using your example with a bash script. I called it myscript.sh #!/bin/bash FILENAME=$1 LINEAS=$(cat "$FILENAME" | wc -l) N=1 while read LINE do if [[ $N -gt 1 ]]; then if [[ $N -eq 2 ]]; ...
user avatar
0 votes

Show changes between consecutive lines of a file

watch -d tail my.log (default parameters, equivalent to : watch -d -n 2 tail -n 10 my.log) watch -d -n 5 tail -n 15 my.log At 5 seconds interval, watch last 15 lines from my.log man watch -d, --...
user avatar
  • 519
3 votes
Accepted

How to get the required output with awk and add the output into another file?

Using awk, if the delimiter is a space or more, then you can try; $ awk -F' +|"' 'NR==FNR {a[$11]=$5-$4+1;next} FNR==1{$(NF+1)="num"}{print $0,a[$2]}' test.gtf test2.tsv | column -t ...
user avatar
  • 1,868
2 votes

How to get the required output with awk and add the output into another file?

The spaces between fields in your input are not tabs so don't tell awk they're tabs with -F"\t", just remove that statement and change ID=substr($9, length($9)-16, 15) to ID=substr($0, ...
user avatar
  • 23.6k
1 vote

Show changes between consecutive lines of a file

It's true: this is not a code writing service. But if a task is easy or fun, some of us act as one. (-; Assuming you can't write programs, you still can use sed, no real one-liner with such a line ...
user avatar
  • 12.3k
5 votes

output filename and 'found string' on same line in output file

The filename of the file currently being processed by awk is available in the special variable FILENAME. This means that you could adapt your second awk command like so: awk '/Wall/ { printf "%s:...
user avatar
  • 277k
1 vote

Replacing occurences of multiple "keys" by corresponding replacement "values" in a large text file

Assuming you only need to handle sunny day mappings as in your provided example (i.e. no regexp or backreference metachars, no case changes, no substrings, no cyclic mappings, etc.), then using any ...
user avatar
  • 23.6k
1 vote

Replacing occurences of multiple "keys" by corresponding replacement "values" in a large text file

Assuming that the replacement values in your key-value-mapfile cannot themselves contain keys that would mandate replacing (including their own associated key!), and the mapfile is tab-separated the ...
user avatar
  • 18.9k
0 votes

Splitting columns in TSV into CSV

Using perl, as a one-liner: perl -nle 'my @f = split /\t/; print "$f[0],$_" for split /,/,$f[1]' -n: process input line by line and execute the code given for each line -l: remove end-of-...
user avatar
  • 278
2 votes

Replacing occurences of multiple "keys" by corresponding replacement "values" in a large text file

Note that -i and \b are both non-standard extensions that some sed implementations have copied from perl. Here, why not using perl in the first place: perl -i -pe ' BEGIN { %map = ( "...
user avatar
0 votes

Make 2nd column unique and add the respective value of them in 1st columns

Assuming the data is sorted on the second column, using GNU datamash: datamash -W groupby 2 sum 1 <file This reads the input as whitespace-delimited fields, groups the data by the second field and ...
user avatar
  • 277k
0 votes

Splitting columns in TSV into CSV

This seems fairly simple if you're willing to use python: #!/usr/bin/env python from __future__ import print_function import sys with open(sys.argv[1]) as f: for line in f: key, vals = ...
user avatar
8 votes

Splitting columns in TSV into CSV

Using any awk: $ awk -F'[\t,]' -v OFS=',' '{for (i=2; i<=NF; i++) print $1, $i}' file id1,col1 id1,col2 id1,col3 id2,col4 id2,col5 The above assumes your first column can't contain ,s and your 2nd ...
user avatar
  • 23.6k
0 votes

put text along with unix command then feed to a file

Here's how I would do this in tcsh: foreach a (`cat my_folder.txt`) set count=`find ./netlists/spice/${a}.sp -newermt '8/16/2022 0:00:00' | wc -l` echo "$a $count" end I'm not sure what ...
user avatar
0 votes
Accepted

put text along with unix command then feed to a file

I'm not entirely sure what you're doing, but this might be what you want: foreach a (`cat my_folder.txt`) echo "$a " $(find ./netlists/spice/${a}.sp -newermt '8/16/2022 0:00:00' | wc -l) &...
user avatar
6 votes

compare specific columns in two files file1 & file 2 and update each matching line in file 1

Using awk $ awk -F, 'NR==FNR {a[$2,$3];next} FNR>1 {$0=$0 (($2,$3) in a?" match found" : " match not found")}1' file2.csv file1.csv col1,col2,col3,col4 1,11,111,1111 match found ...
user avatar
  • 1,868
0 votes
Accepted

compare specific columns in two files file1 & file 2 and update each matching line in file 1

If (1) there will be no commas in the files other than the field-separating ones and (2) both fields must match, you can paste both files and handle matches with sed: paste file1.csv file2.csv | sed '...
user avatar
  • 12.3k
2 votes

Splitting columns in TSV into CSV

I don't know the TSV format, so assuming the input has the literal TAB character as the separator, not a '\' followed by a 't', then sed is an option: sed -r ':loop;s:([^\t]+)(\t[^,]+),:\1\2\n\1>:;...
user avatar
7 votes

Splitting columns in TSV into CSV

$ mlr --itsv --inidx --ocsv -N nest --evar , -f 2 file id1,col1 id1,col2 id1,col3 id2,col4 id2,col5 This uses Miller (mlr) to read records consisting of tab-delimited fields (--itsv) which we will ...
user avatar
  • 277k
4 votes

Splitting columns in TSV into CSV

You can use awk with split function for second token: awk 'BEGIN {OFS=","} {n=split($2,a,","); for(i=1;i<=n;i++) print $1,a[i]} ' input_file
user avatar
  • 14.2k
0 votes

Remove lines that do not start with a pattern from a given set of patterns

Just use egrep with the -v option to find all the lines to exclude. egrep -v '^(report|-th|-to)' Test cat <<EOF|egrep -v '(report|-th|-to)' report some goofy stuffs -th more goofy stuffs -to ...
user avatar
1 vote

replace values in the columns of a file based on matching values of first column

The problem is two-fold. First, you are setting the input field separator to , whereas you stated that the input is space-separated. Then, none of your field-based operations will work correctly. If ...
user avatar
  • 18.9k
4 votes
Accepted

used awk to search by unicode in txt file

If using zsh¹, you can do: SEARCH=$'\u64c' awk 'index($0, ENVIRON["SEARCH"])' To print the lines that contain that character. For characters above U+FFFF, use $'\U1F427' (for 🐧 for ...
user avatar
1 vote

search in txt file and every result in new file

Again, awk's your friend here: awk ' !values_processed { result[$0] = "result"FNR".txt"; next } !NF {FNR=0; next} FNR==1 {header=$0; next} { for (value in result) ...
user avatar
0 votes

search in txt file and every result in new file

I think what you want is (untested): awk ' NR==FNR { out[$1] = "result" NR ".txt" next } { for ( i=2; i<=NF; i++ ) { for ( str in ...
user avatar
  • 23.6k
3 votes
Accepted

search in txt file then print in new txt file

You could do something like: <input-file.txt awk ' !NF {FNR=0; next} FNR==1 {header=$0; next} /dn/ {print $0, header}' > result.txt Where we reset the per-File Record Number every ...
user avatar
2 votes

search in txt file then print in new txt file

Basic grep: grep 'dn' input-file.txt > result.txt
user avatar
  • 1,613
1 vote

What is best for multiple word replacement in a large text file, sed or awk?

The answer is a clear: it depends. It depends on how you define »best«. Faster? Easier to write? To read? To maintain? More portable? More elegant? It depends on the exact task. sed may generally be ...
user avatar
  • 12.3k
3 votes

Reading a file with two columns and use them to change filenames

You could use something close to your suggested syntax with GNU parallel in place of awk, ex. $ parallel --colsep ' - ' echo mv -n -- Genomic_data/Mito/CDS/Fungi/{1}.gb Genomic_data/Mito/CDS/Fungi/{2}....
user avatar
  • 73.1k
1 vote

How to test if a function is defined in awk?

You could do: awk 'BEGIN {has_asort = (asort (x)) == "0"} function my_asort(a) { if (has_asort) return asort (a) ... } ... Note that the space between asort and ...
user avatar
0 votes

How to test if a function is defined in awk?

The following works for gawk 5.0.1 under Ubuntu 20.04; YMMV: #!/usr/bin/awk -f @load "rwarray" # just to show how extensions look function DoSomething(a, b){ return a * b } BEGIN { ...
user avatar
  • 121
3 votes

Reading a file with two columns and use them to change filenames

Here's my version, where bash reads your prepared file and builds the mv command (including the path prefix). The script assumes your input file with the old/new filenames is named "infile",...
user avatar
  • 1,091
2 votes
Accepted

Reading a file with two columns and use them to change filenames

You can run mv from awk itself: awk -F" - " '{ system("mv "$1" "$2) }' filename.txt Also you can do it without awk: while read N1 SEP N2; do mv ${N1} ${N2}; done < ...
user avatar
  • 146
2 votes

How can I replace multiple substrings from multiple lines in a file matching a pattern from a different file?

We will use the ip_hostnames file to build a series of sed commands in the BRE regex syntax. These commands are basic s/ip/hostname/. Care is taken to escape the contents that go into the left hand ...
user avatar
  • 5,566
2 votes

How can I replace multiple substrings from multiple lines in a file matching a pattern from a different file?

Using any POSIX awk: $ cat tst.awk NR==FNR { map[$2] = $1 next } match($0,/([[:space:]]+([0-9]{1,3}\.){3}[0-9]{1,3})+$/) { path = substr($0,1,RSTART-1) $0 = substr($0,RSTART,RLENGTH) ...
user avatar
  • 23.6k
-1 votes

How can I replace multiple substrings from multiple lines in a file matching a pattern from a different file?

for i in $(cat ip_hostname.txt| awk '{print $2}') do cap=$(awk -v i="$i" '$NF==i{print $1}' ip_hostname.txt) sed -i "s/$i/$cap/g" path_ips.txt done output /path1/foo/bar host1 ...
user avatar
2 votes

How can I replace multiple substrings from multiple lines in a file matching a pattern from a different file?

This can be done entirely in sed, but the awk answer is generally more readable. #file toggle 1{x;s:^$:<IPs>:;x} /^EOF$/{x;s:<IPs>:<paths>:;x;d} #store hostname file x;/<IPs>/{...
user avatar
6 votes

How can I replace multiple substrings from multiple lines in a file matching a pattern from a different file?

The problem with your script is that you run a separate sed command for each IP address that is matched, which makes the script really slow when the file is big. You also have a nested loop, so you ...
user avatar
  • 3,007
0 votes

awk for loop for each line in a file

cat xtmp | awk '{s++; do_aything_you_want_here }' cat xtmp 111 222 333 444 cat xtmp | awk '{s++; print($1,"QQQ") }' 111 QQQ 222 QQQ 333 QQQ 444 QQQ cat xtmp | awk '{s++; print($1+3,&...
user avatar
  • 19
0 votes

Replace values on the basis of matching ID variable

I am unable to comment (no privilege) - but would like to ask an additional question. I want to do the exact same thing as the first person asked. In addition i want the remaining lines in file 2 that ...
user avatar
  • 3
0 votes

Can I determine type of an awk variable?

If the gawk < 4.2(RHEL7 default version: gawk 4.0.2) As fedorqui answer and the gawk 4.2 release note 14. The new typeof() function can be used to indicate if a variable or array element is an ...
user avatar
2 votes

Awk replace string where specific condition is true using variables

I would suggest something like this: $ awk -v FS=';' -v OFS=';' -v var1="$v1" -v var2="$v2" '$2 == var2 { $1 = var1 } 1' filename 0;20220808163547; 1224;20220808163547; 2048;...
user avatar
  • 120k
5 votes

How could awk be made timezone aware?

The following is all about GNU awk for various extensions and is aware of the timezone in your environment but not directly in your data. It's not entirely the output you wanted (idk if all of your TZ ...
user avatar
  • 23.6k
1 vote

Capitalise few characters in a word

I wrote the program below in TXR Lisp for solving this problem. This relies on a /usr/share/dict/words dictionary file being available; plus it adds some words of its own that are common abbreviations ...
user avatar
  • 6,675
1 vote

Separating CSV (one column) into many columns on delimiter (comma)

You should add always some input and output sample file. I seem to have understood that you have an input of this type, a csv in which a column contains a CSV inside (in example here, the a field) a ...
user avatar
  • 2,123
2 votes

How could awk be made timezone aware?

The only way that I have been able to get that is by using GNU date, which involves a lot of re-formating, calling an external executable for each new line and getting it back in to awk. GNU date can ...
user avatar
  • 65.1k
3 votes

Linux Column with value print

The test.txt file content is a,b,c, d,e,f,g g,h,i j,k,l,m awk with -F to specify comma as separator, and $4 to specify 4th column. awk -F',' '$4!="" {print $3","$4}' test.txt to ...
user avatar
  • 223
0 votes

How to convert a kubernetes style time elapsed into seconds so I can do some comparisons

For a command that understands those 4d3h15m3s, see dateutils's dadd: $ TZ=UTC0 dateutils.dadd -f%s 1970-01-01T00:00:00 4d3h15m3s 357303 Here adding that duration to the start of Unix epoch time, and ...
user avatar
1 vote

How to convert a kubernetes style time elapsed into seconds so I can do some comparisons

In case anyone was looking for a python fucntion like myself, here it is: def kubctle_dur_to_secs(kubectl_time:str) -> int: total_time = 0 for t_char, sec_in_t in zip( ('d', 'h', 'm', 's'), ...
user avatar

Top 50 recent answers are included