awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
3answers
32 views

Delete whole line if after “:” less than 4 chars

I have a file with many lines, such as : blue:fish red:tomato black:moon I would like the whole line to be deleted if after ":", there is less than 4 chars. Thanks for your help!
0
votes
1answer
35 views

Sorting file by second column

How can I manipulate a two columns tab-separated text file by sorting by the first element of the second column (only if the first column element is the same)? Exemple: Input File 1 A 1-2 A ...
1
vote
2answers
83 views

merge csv files, escape comma inside quotes

I have 3 csv files I want to join by first column (id column) Each file has the same 3 columns Row example : id | timestamp | Name 3792318, 2014-07-15 00:00:00, "A, B" When I join ...
1
vote
4answers
111 views

How can I output the difference between 2 files?

My file consists of the the following; roughly: username:username:username:username:username The above line continues to about 600 characters. I use the awk command in order to use it as an ...
2
votes
1answer
64 views

Doesn't awk print all lines if it is passed empty arguments

I thought that awk without arguments is equivalent to awk {print} file. But when I do: awk '' file Nothing is printed. Why?
4
votes
2answers
71 views

Get the maximum sum for a file on 2 keys

I have a file as below. key1 key2 key3 a1 2 l1 a1 2.5 l2 a2 2 l2 a3 2.5 l3 a3 2.1 l4 a3 2.2 l5 I am trying to find the maximum possible sum of key2 that satisfies the ...
1
vote
5answers
398 views

delete lines in a csv file older than 7 days

I have a csv that I need to remove all lines that are older than 7 days. This is the format of the csv Person ID VIP CS SS LT FTLT PS Modified Datestamp T001028 1 1 1 0 0 0 ...
3
votes
2answers
118 views

Counting comma separated characters in a row

How can I count how many elements are separated with comma in a given column of a file (tab_delimited), using awk or sed? Exemple: AC 1,23,5 DC 2,8,89 Would return: AC 3 DC 3
2
votes
1answer
115 views

merge csv files by first column

I have 3 csv files like this. csv 1: 1,aaaa,bbb,2014-04-01 2,qwe,rty,2014-04-03 3,zxc,cvb,2014-04-05 csv 2: 2,j,k,2014-04-01 3,a,s,2014-04-04 5,g,h,2014-04-08 csv 3: 2,a,s,d,f,g,2014-04-01 ...
1
vote
2answers
89 views

Replace columns using a shell

How could I replace column 1 in File1 with column 2 in File2, when a match is found? File1 id1 36987 id2 75685 id3 12345 id4 12896 File2 id1 ID1 id2 ID2 id3 ID3 id5 ID5 I'd like to write this on ...
2
votes
1answer
89 views

Compare the values of a column with all the values in another column

I have 2 input files. Each line of File1 should be compared to each and every line of File2. Logic is: If Column1 of File1 does not match Column1 (all the values under it) of File2, then print the ...
1
vote
1answer
28 views

awk regex matches wrong?

Playing around with awk I noticed this behavior: [root@ror6ax3 ~]# grep open * | awk '$2 ~ /opens*/ {print $0}' install.log:Installing openldap-2.4.23-32.el6_4.1.x86_64 install.log:Installing ...
1
vote
2answers
69 views

Merge csv files with conditions

I have 2 CSV files for each date (csv_2014_4_15 and csv_2014_4_16) , with basic structure and couple of unique columns as below. id,name,created_at,updated_at,other columns 12, joe, 2013-1-1 18:30, ...
5
votes
5answers
520 views

Why does awk not sum to 0

I have this file and I want to sum all the number in first column. Easy: awk '{s+=$1;print $1,s}' file -0.4157 -0.4157 0.0143 -0.4014 0.5973 0.1959 -0.5679 -0.372 -0.2041 -0.5761 0.7130 0.1369 ...
2
votes
3answers
46 views

Merging multiple files by row

Is it possible to copy the rows of File1 and File2 in a new File4 following the instruction given by File3 by using a simple bash script (sed? awk?). File1: /*two or more columns, tab delimited*/ ...
7
votes
2answers
440 views

Conditional block vs conditional statement (if)

Say I have a file: PRO 1 GLN 5.55112e-17 ILE -6.245e-17 THR 5.55112e-17 I want every line that has a number unequal to 1 in the second column to change it to 0 and keep the rest. If I use if (i.e ...
1
vote
3answers
60 views

Sed + how to set parameters in sed command in order to display lines from text file

I find the following way to print line from file In this example we want to print the first line from the hosts file sed -n '1,1p' /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 ...
6
votes
4answers
153 views

How to insert lines with numbers from X to Y into another file after line Z?

I created two files: echo -e "1\n2\n3\n4\n5" > 123.txt echo -e "a\nb\nc\nd\ne" > abc.txt I want to get the file 123.txt with the following contents: 1 2 3 b c d 4 5 So in other words ...
5
votes
2answers
122 views

Is there a way to make this one-liner faster?

Context I have a directory of thousands of zip files that are dated in the form YYYYMMDD_hhmmss.zip and each about 300K. Within each zip file is about 400 xml files each about 3K. The problem I ...
5
votes
2answers
154 views

Sum in block with AWK (restart the sum when the pattern changes)

I have a file like this: A 100 A 200 A 300 #sum=600 B 400 B 500 #sum=900 A 600 A 700 A 800 #sum=2100 I would like the output to be: A 600 B 900 A 2100 C sum_of_C D sum_of_D I can do that with ...
2
votes
2answers
2k views

Diff, show modified line from right file only

I have file a and b and I would like to output lines of b that changed since it was cloned from a. Just the modified lines, no surrounding context, no diff offset marks. How can I do that using shell ...
0
votes
1answer
56 views

change date format and store in variable using awk

I have date in format YYYY-MM-DD in variable 'a'. I want to convert it in format YYYYMMDD and store in variable 'b'. The variable is in an awk variable. How can I do the same ?
1
vote
3answers
470 views

awk a specific column with unexpected line break

I can't seem to find a solution for this and it's driving me crazy. I know I can use awk to print a column(s). I'm having trouble printing a specific column though because of the way my file system is ...
6
votes
5answers
235 views

Strategy to extract movies's name from this ununiform dataset?

I am working on a movie database problem to improve regular expressions, this is the problem I'm running into. My dataset looks like this: Movie Name (variable space and tabs) year Movie1(can ...
2
votes
2answers
58 views

How can I compare two files based on the value of the first column?

I want to compare two files based on the value of their first column: file1 aaa 23 3 bbb 56 6 ccc 45 2 file2 bbb m a aaa a m ccc m m The output file should be: aaa a m bbb m a ccc m m ...
3
votes
1answer
65 views

I can't print blank lines when reading values from a text file using awk in a bash script

awk is behaving very strangely in my script. I am trying to read values from a text file (see file.txt below). awk is ignoring blank spaces in my script, but when I test the awk statement on its own ...
1
vote
1answer
45 views

Retrieving fasta sequences using bed file information from locally installed file

I have a .bed file containing around 30000 rows for which I have the sequences retrieved using fetch-sequences module of the rsat tool (http://rsat.ulb.ac.be/rsat/help.fetch-sequences.html#usage). ...
0
votes
1answer
66 views

Match first fields of two tab separated files and print matching values

I have two files. Match the 1st column of first file to 1st column of second file and print entire line of the second file. Second file is a <tab> separated file. I have tried many awk one ...
3
votes
2answers
184 views

Find and replace different columns using awk

I try to make a change in a file: Find a certain value and substitute it with another value in a different column using awk. Input (UiO-66Zr-EH.mof): unit ntype qqatom 1 'Zr1' 0.0d0 vibration ...
7
votes
4answers
179 views

Sort fields inline

I'm trying to sort within a line of input over an unknown number of fields: Input: ab bc bc ab cd ef bc bc cd ef cd bc ab ef ab bc cd gh Output: ab bc ab bc bc cd ef bc cd ef ab cb cd ab bc cd ...
3
votes
3answers
261 views

calculate sum of squares using shell script in perl/awk

I have 2 files as below. file1 0.34 0.27 0.32 file2 0.15 0.21 0.15 Now, I would like to calculate the sum of squares between each column. For example, [(0.34 - 0.15)^2 + (0.27 - 0.21)^2 + ...
3
votes
1answer
165 views

Awk print problem

I have a code that takes values from a log file and populates them into an excel spreadsheet: function ltrim(s) { sub(/^[ \t]+/, "", s); return s } function rtrim(s) { sub(/[ \t]+$/, "", s); return s ...
1
vote
3answers
38 views

How to truncate only given column length?

INPUT: a@notebook:~$ cat in.csv 'XYZ843141'^'ASDFSAFXYVFSHGDSDg sdGDS dsGDSgfa assfd faSDFAS saDFSAFD adFSA343fa sdfSADF'^'BAAAR'^'YYY'^'..... and so on, further columns ...
1
vote
1answer
27 views

awk getline not working as expected on first loop iteration

Consider this simple awk calculator program and its output $ awk '{while (getline) {n=0;for(i=NF;i>0;i--)n+=$i; print "sum: " n;}}' 1 1 2 2 sum: 4 3 3 sum: 6 Note how the first input line 1 1 ...
1
vote
2answers
207 views

Extract Values from simple html file via grep/awk

I have another grep/awk/sed issue within the attached HTML code below. Given is a simple HTML file with table inside. The HTML is generated by a smart meter (household energy meter). Inside the html ...
2
votes
3answers
63 views

replace a character from a string that is in between first and second search using sed or awk

I need to replace _ (underscore) with ? (question mark) in a file containing email addresses. The file looks like below: EFT_020034-E015133 20140624 /ACC/EMAIL+test_1@mysite.com SHR ...
2
votes
2answers
61 views

new pattern required from an input file using sed or awk

I have data as below: MGW: VMG110 836-16 836-18 836-19 336-20 836-23 MGW: VMG120 3802-1 3802-2 3802-3 3456-1 3456-4 Required output is: VMG110:836-16&-18&&-20&-23 ...
5
votes
2answers
76 views

Grepping for a block of text with parts that can be optional

I have multiple entries that describes an event in a very large log file, say A.log. I would like to do two things with the event entries in the log file: Count the number of occurrences of each ...
1
vote
1answer
30 views

How to echo from specific line of a file to another specific line [duplicate]

Assume I have a file with about 10000 lines. How can I print 100 lines, starting from line 1200 to line 1300?
2
votes
3answers
53 views

Get the last minute dated lines from list

If I have an input file with a list of users and their dates they were last signed in, how can I select with a sort | awk function the last minute lines? The dates are written in a +%H:%M:%S format. ...
4
votes
1answer
138 views

Is getline in awk command safe to use?

I got weird comments on posting an awk answer on SO which uses the getline function. Here is the link to that answer. After posting my answer, a user came up with the below comment,(I don't criticize ...
3
votes
2answers
54 views

awk: Blank line with date

I use the following code to write the online users' process numbers and the date they were checked: w | tail -n +3 | awk ' { "date +%H:%M:%S" | getline tim} {if ($1 != "") ...
0
votes
1answer
65 views

Search a string between two timestamps starting from bottom of file

I was trying to find a string Cannot proceed: the cube has no data in huge test.txt file only between timestamps yesterday 22:30 pm to today 00:30 am. Script: tac test.txt | awk -v today=$(date ...
3
votes
2answers
169 views

Search a string with last two characters as numeric

Suppose I have a file like this: adasfddfd09 dsassd90897 323sdsdsdsd sdddsdf56 dfdf45fdfdf I want to search only those strings which have last two characters numeric with sed, grep and awk.
4
votes
3answers
165 views

Extracting text using sed does not work as expected [duplicate]

I am just starting out using sed and I intend to use it to extract IP addresses from ping output. Here's what I am trying to achieve exactly: input text: ytmti (192.188.2.3) jjggy desired ...
1
vote
4answers
64 views

Printing a new line using awk

I have a load of files in which I need to take out specific rows and then put the data I've taken out into a spreadsheet. An example would be my file shows: Name: w Age: x Height: y Weight: z I ...
4
votes
4answers
283 views

Redirecting Output of print to multiples files in awk

I want to print same lines to 2 different files. What I am doing right now is: print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file1.txt"; print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file2.txt"; ...
2
votes
4answers
217 views

How to initialize an array using awk and bash?

I am trying to store values of the first line of a text file into an array. Here is what I have so far: arr_values=() awk ' NR==1 { for (i=0; i<=NF; i++) ...
0
votes
1answer
46 views

using awk for subsetting fastq file based on length of sequence

I have a fastq file. I will explain what it is. It is something like this @SRR1024120.7 DBRHHJN1:259:D0PM7ACXX:1:1101:1386:1189 length=100 GATACAGGATGCCTGGGTCTAGGCTGTGTGACCTTGGGCCAGTTCCTCTC ...
4
votes
3answers
185 views

UNIX paste columns and insert zero's for all missing values

I would like to merge specific columns from two txt files containing varying number of rows, but same number of columns (as shown below): file1: xyz desc1 12 uvw desc2 55 pqr desc3 ...