awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

3
votes
2answers
150 views

Counting comma separated characters in a row

How can I count how many elements are separated with comma in a given column of a file (tab_delimited), using awk or sed? Exemple: AC 1,23,5 DC 2,8,89 Would return: AC 3 DC 3
2
votes
1answer
159 views

merge csv files by first column

I have 3 csv files like this. csv 1: 1,aaaa,bbb,2014-04-01 2,qwe,rty,2014-04-03 3,zxc,cvb,2014-04-05 csv 2: 2,j,k,2014-04-01 3,a,s,2014-04-04 5,g,h,2014-04-08 csv 3: 2,a,s,d,f,g,2014-04-01 ...
2
votes
1answer
99 views

Compare the values of a column with all the values in another column

I have 2 input files. Each line of File1 should be compared to each and every line of File2. Logic is: If Column1 of File1 does not match Column1 (all the values under it) of File2, then print the ...
1
vote
1answer
29 views

awk regex matches wrong?

Playing around with awk I noticed this behavior: [root@ror6ax3 ~]# grep open * | awk '$2 ~ /opens*/ {print $0}' install.log:Installing openldap-2.4.23-32.el6_4.1.x86_64 install.log:Installing ...
1
vote
2answers
88 views

Merge csv files with conditions

I have 2 CSV files for each date (csv_2014_4_15 and csv_2014_4_16) , with basic structure and couple of unique columns as below. id,name,created_at,updated_at,other columns 12, joe, 2013-1-1 18:30, ...
5
votes
5answers
537 views

Why does awk not sum to 0

I have this file and I want to sum all the number in first column. Easy: awk '{s+=$1;print $1,s}' file -0.4157 -0.4157 0.0143 -0.4014 0.5973 0.1959 -0.5679 -0.372 -0.2041 -0.5761 0.7130 0.1369 ...
2
votes
3answers
50 views

Merging multiple files by row

Is it possible to copy the rows of File1 and File2 in a new File4 following the instruction given by File3 by using a simple bash script (sed? awk?). File1: /*two or more columns, tab delimited*/ ...
7
votes
2answers
459 views

Conditional block vs conditional statement (if)

Say I have a file: PRO 1 GLN 5.55112e-17 ILE -6.245e-17 THR 5.55112e-17 I want every line that has a number unequal to 1 in the second column to change it to 0 and keep the rest. If I use if (i.e ...
1
vote
3answers
65 views

Sed + how to set parameters in sed command in order to display lines from text file

I find the following way to print line from file In this example we want to print the first line from the hosts file sed -n '1,1p' /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 ...
6
votes
4answers
158 views

How to insert lines with numbers from X to Y into another file after line Z?

I created two files: echo -e "1\n2\n3\n4\n5" > 123.txt echo -e "a\nb\nc\nd\ne" > abc.txt I want to get the file 123.txt with the following contents: 1 2 3 b c d 4 5 So in other words ...
5
votes
2answers
123 views

Is there a way to make this one-liner faster?

Context I have a directory of thousands of zip files that are dated in the form YYYYMMDD_hhmmss.zip and each about 300K. Within each zip file is about 400 xml files each about 3K. The problem I ...
5
votes
2answers
165 views

Sum in block with AWK (restart the sum when the pattern changes)

I have a file like this: A 100 A 200 A 300 #sum=600 B 400 B 500 #sum=900 A 600 A 700 A 800 #sum=2100 I would like the output to be: A 600 B 900 A 2100 C sum_of_C D sum_of_D I can do that with ...
2
votes
2answers
2k views

Diff, show modified line from right file only

I have file a and b and I would like to output lines of b that changed since it was cloned from a. Just the modified lines, no surrounding context, no diff offset marks. How can I do that using shell ...
0
votes
1answer
81 views

change date format and store in variable using awk

I have date in format YYYY-MM-DD in variable 'a'. I want to convert it in format YYYYMMDD and store in variable 'b'. The variable is in an awk variable. How can I do the same ?
1
vote
3answers
495 views

awk a specific column with unexpected line break

I can't seem to find a solution for this and it's driving me crazy. I know I can use awk to print a column(s). I'm having trouble printing a specific column though because of the way my file system is ...
6
votes
5answers
237 views

Strategy to extract movies's name from this ununiform dataset?

I am working on a movie database problem to improve regular expressions, this is the problem I'm running into. My dataset looks like this: Movie Name (variable space and tabs) year Movie1(can ...
2
votes
2answers
61 views

How can I compare two files based on the value of the first column?

I want to compare two files based on the value of their first column: file1 aaa 23 3 bbb 56 6 ccc 45 2 file2 bbb m a aaa a m ccc m m The output file should be: aaa a m bbb m a ccc m m ...
3
votes
1answer
68 views

I can't print blank lines when reading values from a text file using awk in a bash script

awk is behaving very strangely in my script. I am trying to read values from a text file (see file.txt below). awk is ignoring blank spaces in my script, but when I test the awk statement on its own ...
1
vote
1answer
48 views

Retrieving fasta sequences using bed file information from locally installed file

I have a .bed file containing around 30000 rows for which I have the sequences retrieved using fetch-sequences module of the rsat tool (http://rsat.ulb.ac.be/rsat/help.fetch-sequences.html#usage). ...
0
votes
1answer
100 views

Match first fields of two tab separated files and print matching values

I have two files. Match the 1st column of first file to 1st column of second file and print entire line of the second file. Second file is a <tab> separated file. I have tried many awk one ...
3
votes
2answers
386 views

Find and replace different columns using awk

I try to make a change in a file: Find a certain value and substitute it with another value in a different column using awk. Input (UiO-66Zr-EH.mof): unit ntype qqatom 1 'Zr1' 0.0d0 vibration ...
7
votes
4answers
184 views

Sort fields inline

I'm trying to sort within a line of input over an unknown number of fields: Input: ab bc bc ab cd ef bc bc cd ef cd bc ab ef ab bc cd gh Output: ab bc ab bc bc cd ef bc cd ef ab cb cd ab bc cd ...
3
votes
3answers
311 views

calculate sum of squares using shell script in perl/awk

I have 2 files as below. file1 0.34 0.27 0.32 file2 0.15 0.21 0.15 Now, I would like to calculate the sum of squares between each column. For example, [(0.34 - 0.15)^2 + (0.27 - 0.21)^2 + ...
3
votes
1answer
173 views

Awk print problem

I have a code that takes values from a log file and populates them into an excel spreadsheet: function ltrim(s) { sub(/^[ \t]+/, "", s); return s } function rtrim(s) { sub(/[ \t]+$/, "", s); return s ...
1
vote
3answers
47 views

How to truncate only given column length?

INPUT: a@notebook:~$ cat in.csv 'XYZ843141'^'ASDFSAFXYVFSHGDSDg sdGDS dsGDSgfa assfd faSDFAS saDFSAFD adFSA343fa sdfSADF'^'BAAAR'^'YYY'^'..... and so on, further columns ...
1
vote
1answer
31 views

awk getline not working as expected on first loop iteration

Consider this simple awk calculator program and its output $ awk '{while (getline) {n=0;for(i=NF;i>0;i--)n+=$i; print "sum: " n;}}' 1 1 2 2 sum: 4 3 3 sum: 6 Note how the first input line 1 1 ...
1
vote
2answers
277 views

Extract Values from simple html file via grep/awk

I have another grep/awk/sed issue within the attached HTML code below. Given is a simple HTML file with table inside. The HTML is generated by a smart meter (household energy meter). Inside the html ...
2
votes
3answers
87 views

replace a character from a string that is in between first and second search using sed or awk

I need to replace _ (underscore) with ? (question mark) in a file containing email addresses. The file looks like below: EFT_020034-E015133 20140624 /ACC/EMAIL+test_1@mysite.com SHR ...
2
votes
2answers
63 views

new pattern required from an input file using sed or awk

I have data as below: MGW: VMG110 836-16 836-18 836-19 336-20 836-23 MGW: VMG120 3802-1 3802-2 3802-3 3456-1 3456-4 Required output is: VMG110:836-16&-18&&-20&-23 ...
5
votes
2answers
83 views

Grepping for a block of text with parts that can be optional

I have multiple entries that describes an event in a very large log file, say A.log. I would like to do two things with the event entries in the log file: Count the number of occurrences of each ...
1
vote
1answer
36 views

How to echo from specific line of a file to another specific line [duplicate]

Assume I have a file with about 10000 lines. How can I print 100 lines, starting from line 1200 to line 1300?
2
votes
3answers
61 views

Get the last minute dated lines from list

If I have an input file with a list of users and their dates they were last signed in, how can I select with a sort | awk function the last minute lines? The dates are written in a +%H:%M:%S format. ...
4
votes
1answer
218 views

Is getline in awk command safe to use?

I got weird comments on posting an awk answer on SO which uses the getline function. Here is the link to that answer. After posting my answer, a user came up with the below comment,(I don't criticize ...
3
votes
2answers
56 views

awk: Blank line with date

I use the following code to write the online users' process numbers and the date they were checked: w | tail -n +3 | awk ' { "date +%H:%M:%S" | getline tim} {if ($1 != "") ...
0
votes
1answer
72 views

Search a string between two timestamps starting from bottom of file

I was trying to find a string Cannot proceed: the cube has no data in huge test.txt file only between timestamps yesterday 22:30 pm to today 00:30 am. Script: tac test.txt | awk -v today=$(date ...
3
votes
2answers
226 views

Search a string with last two characters as numeric

Suppose I have a file like this: adasfddfd09 dsassd90897 323sdsdsdsd sdddsdf56 dfdf45fdfdf I want to search only those strings which have last two characters numeric with sed, grep and awk.
4
votes
3answers
168 views

Extracting text using sed does not work as expected [duplicate]

I am just starting out using sed and I intend to use it to extract IP addresses from ping output. Here's what I am trying to achieve exactly: input text: ytmti (192.188.2.3) jjggy desired ...
1
vote
4answers
83 views

Printing a new line using awk

I have a load of files in which I need to take out specific rows and then put the data I've taken out into a spreadsheet. An example would be my file shows: Name: w Age: x Height: y Weight: z I ...
4
votes
4answers
416 views

Redirecting Output of print to multiples files in awk

I want to print same lines to 2 different files. What I am doing right now is: print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file1.txt"; print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file2.txt"; ...
2
votes
4answers
344 views

How to initialize an array using awk and bash?

I am trying to store values of the first line of a text file into an array. Here is what I have so far: arr_values=() awk ' NR==1 { for (i=0; i<=NF; i++) ...
0
votes
1answer
57 views

using awk for subsetting fastq file based on length of sequence

I have a fastq file. I will explain what it is. It is something like this @SRR1024120.7 DBRHHJN1:259:D0PM7ACXX:1:1101:1386:1189 length=100 GATACAGGATGCCTGGGTCTAGGCTGTGTGACCTTGGGCCAGTTCCTCTC ...
4
votes
3answers
203 views

UNIX paste columns and insert zero's for all missing values

I would like to merge specific columns from two txt files containing varying number of rows, but same number of columns (as shown below): file1: xyz desc1 12 uvw desc2 55 pqr desc3 ...
4
votes
3answers
183 views

Replace multiple strings in a single pass

I'm looking for a way to replace placeholder strings in a template file with concrete values, with common Unix tools (bash, sed, awk, maybe perl). It is important that the replacement is done in a ...
9
votes
5answers
9k views

Remove comma between the quotes only in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...
1
vote
4answers
4k views

sed/awk/perl - delete spaces between two strings/words + solaris 10

please advice how to delete the spaces between "=" to the value/string ( after the "=" character) , should be implemented by awk or sed or perl command and command must to fit for solaris OS , ( ...
0
votes
1answer
47 views

I am having some technical issue with awk code [closed]

This is related to one of my previous question: Comparing text files using Bash and AWK file1.txt Name Col1 Col2 Col3 ----------------------- row1 1 4 7 row2 2 5 8 ...
1
vote
3answers
61 views

multiplication of every line

I would like to calculate a value on every line of a file such as this: 1 1441792,11534336 2 1048576,8388608 3 1441792,1153433 4 1966080,15728640 5 393216,3145728 6 2621440,20971520 7 ...
2
votes
3answers
1k views

Comparing text files using Bash and AWK

I have two text files, and I want to compare their corresponding values according to their rows and columns. By comparing, I mean to check if the values are equal and echo if the values are the same ...
7
votes
4answers
184 views

Groupwise uniq command?

I am searching for a command to get from a file in this format: hello 32 hello 67 hi 2 ho 1212 ho 1390 ho 3000 To this format (deduplicate by taking the last row of a "group"): hello ...
0
votes
2answers
59 views

awk to Read between two Particular Strings and discard the rest

I'm using awk to just read text between two strings "Check" and "Result". I've used many variations of it found on the internet and still can't get the favored result. I've tried: awk ...