awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

2
votes
5answers
282 views

Can the string “abcedf” be matched to string “bafcde” in a single line command?

I am planning to implement an indexing structure in my program. For example, if I have 100 rows in the table, I will number these rows from 1 to 100 in another column by appending an _ to the end of ...
1
vote
4answers
227 views

Changing column of multiple csv files

I have a csv file called scenario1.csv in which the second column has column names like "0-4 years high risk", "65+ years first responder" etc. There are 20 such values. The 21st row, 2nd column has ...
7
votes
7answers
1k views

Command line tool to “cat” pairwise expansion of all rows in a file

Suppose I have a file (call it sample.txt) that looks like this: Row1,10 Row2,20 Row3,30 Row4,40 I want to be able to work on a stream from this file that is essentially the pairwise combination of ...
7
votes
6answers
6k views

How to remove all comments from a file?

I have a file with a few lines commented out: foo bar stuff #Do not show this... morestuff evenmorestuff#Or this I just want to print all the uncommented code: foo bar stuff morestuff ...
7
votes
4answers
793 views

Sort fields inline

I'm trying to sort within a line of input over an unknown number of fields: Input: ab bc bc ab cd ef bc bc cd ef cd bc ab ef ab bc cd gh Output: ab bc ab bc bc cd ef bc cd ef ab cb cd ab bc cd ...
4
votes
5answers
96 views

Count unique associated values in awk (or perl)

I've already found "How to print incremental count of occurrences of unique values in column 1", which is similar to my question, but the answer isn't sufficient for my purposes. First let me just ...
4
votes
4answers
3k views

Average rows with same first column

Given a file with two columns: Id ht 510 69 510 67 510 65 510 62 510 59 601 29 601 26 601 21 601 20 I need a way to coalesce all rows with the same ID into one that has an average height. In this ...
4
votes
2answers
8k views

My awk program to change spaces into tabs doesn't work

How to write a shell script which uses awk to read in the data file students.txt and output the data in the tabbed format as shown: Surname Forename MSc Stream Date of Birth Smith John IT 15.01.1986 ...
3
votes
2answers
58 views

match columns and replace

I am newbie in linux. I have two very large text files with space-delimited fields: File1 527858 51 2 27.92464882 8.63E-07 570289 82 2 30.12532071 2.87E-07 571034 90 2 ...
3
votes
2answers
275 views

Assign specific line from a variable to another variable

I would like to take a specific line from a variable to another variable. I tried this but it doesn't work: c="1.apple 2.banna 3.peach" read "Please choose fruit [1-3]:" t a=$c | awk "NR==$t" echo ...
3
votes
2answers
4k views

Merging 2 files with based on field match

I want to create a file that contains columns from two input files. File1 is like: aa 32 bb 15 cc 78 File2 is: fa 19 bc 23 cc 50 de 28 aa 45 bb 31 The task is, read through File1, if the 1st field ...
3
votes
3answers
316 views

how to create a data file with a column of date from day1 to dayN?

I was wondering if I could generate a hourly or daily time step date in a column starting from some year in the past to some day in the near past or today. to be more clear, I want to create one ...
2
votes
4answers
453 views

Why isn't this awk command doing a full outer join?

Objective: Merge the contents of two files using common key present in the files file1.txt ========= key1 11 key2 12 key3 13 file2.txt ========= key2 22 key3 23 key4 24 key5 ...
0
votes
4answers
312 views

How to sum up values of each two rows across their line in linux

I have a data like this: input.txt 1 0000100101000000 1 0000010100000000 2 1110000001000000 2 1111000000001000 3 0000000111111111 3 1111111100000000 4 8888345500000000 4 0000000000000000 and I ...
7
votes
2answers
16k views

Extract file name from path in awk program

I have an awk script and I have passed a CSV file to it. awk -f script.awk /home/abc/imp/asgd.csv What am I doing is to get FILENAME within script.awk. FILENAME gives me the whole path. As I am in ...
6
votes
2answers
143 views

How to split a file according to a column (including header) and rename the generated files?

I have a .txt that can be exemplified like this: NAME | CODE name1 | 001 name2 | 001 name3 | 002 name4 | 003 name5 | 003 name6 | 003 I need to write a script to split this file according to the ...
6
votes
3answers
2k views

sum columns based on matching fields

I have encountered a problem that exceeds my basic unix knowledge and would really appreciate some help. I have a large file in the following format: 2 1019 0 12 2 1019 3 0 2 1021 0 2 ...
6
votes
3answers
2k views

How to select first occurrence between two patterns including them

How can I select first occurrence between two patterns including them. Preferably using sed or awk. I have: text something P1 something content1 content2 something P2 something text something P1 ...
6
votes
3answers
3k views

How can I delete all text between curly brackets in a multiline text file?

Example: This is { the multiline text file } that wants { to be changed } anyway. Should become: This is that wants anyway. I have found some similar threads in the forum, but they don't seem ...
6
votes
5answers
3k views

Find images by size: find / file / awk

I've been trying to find png image files a certain height (over 500px). I know that file will return image dimensions. Example: $ file TestImg1a.png TestImg1a.png: PNG image data, 764 x 200, ...
6
votes
4answers
46k views

How to use regex with AWK for string replacement?

Suppose there is some text from a file: (bookmarks ("Chapter 1 Introduction 1" "#1" ("1.1 Problem Statement and Basic Definitions 23" "#2") ("Exercises 31" "#30") ("Notes and References 42" "#34")) ) ...
5
votes
5answers
581 views

Remove lines from file up to a pattern, unless the pattern doesn't exist

I am attempting to write a filter using something like sed or awk to do the following: If a given pattern does not exist in the input, copy the entire input to the output If the pattern exists in ...
5
votes
3answers
228 views

Escape sequences passed as args to gawk not interpreted

I want to be able to pass an argument on the command line to gawk that is evaluated for escape sequences. The issue: $ gawk 'BEGIN { print ARGV[1]; }' '\t' \t Instead, I would like to get an ...
5
votes
2answers
6k views

Remove entire row in a file if first column is repeated

I have a file containing two columns and 10 million rows. The first column contains many repeated values, but there is a distinct value in column 2. I want to remove the repeated rows and want to keep ...
4
votes
3answers
116 views

String replacement using a dictionary

What is a good way to do string replacements in a file using a dictionary with a lot of substituend-substituent pairs? And by a lot, I actually mean about 20 – not much, but many enough so that I want ...
4
votes
2answers
263 views

split a large file to new file with unique file names

I need to split a file into unique file names. I can do it with sed command eg, sed -n '/scaffold135_/w 135-scaf.txt' input file.txt but it's time consuming so I need a smart way to do it faster. ...
4
votes
2answers
122 views

Awk: compare two files

I have a file1.txt USA Joe 123.123.123 Russia Marry 458.786.892 Canada Greg 151.844.165 Latvia Grace 125.895.688 and file2.txt 1 123.123.123 2 151.844.165 3 465.879.515 and I want to create a ...
4
votes
3answers
452 views

Rearrange text file by prepending data rows with header row

I have a file whose content looks like this: 2009 150 0 0 0 0000 75.316 0.0400390625 0.00007 0.00000 0.8980 76.216 0.0400390625 0.00007 1.00000 0.9046 77.217 0.0400390625 ...
4
votes
1answer
5k views

How can I use awk to (sometimes) remove the first character of a column?

I run a command and pipe it through awk as below : mycommand | awk '{print $1,tolower($3),$4} This gives me the following output: fred.o: t .abcdefg fred.o: u .rstuvwxy fred.o: t .defghi ...
3
votes
2answers
465 views

Why does $var give the value of $0?

I have written a very simple awk script that reverses every word of a file-: awk ' { for (i=1;i<=NF;i++) { x="" for(j=length($i);j>0;j--) x=x substr($0,j,1); print x }}' file1 The contents of ...
3
votes
2answers
121 views

How can I partition a set of words given pairs that must end up together?

I have a space or comma separated table with two columns, each row representing equivalence of the two words. A B B C B D C E F G What I want is a table with each row listing all mutually ...
2
votes
3answers
637 views

unix: replace one entire column in one file with a single value from another file

I want to replace a column in one file with a single value from another file using unix. File 1 is a pdb file, structured like this: HETATM 14 H4B FAD B 600 95.544 50.240 71.308 1.00 ...
2
votes
2answers
89 views

Delimiter is being changed with fields

I have a file with values organized in columns, and separated by commas, as in the file below: 324,01,1,113333600000,1,, 016,01,1,134954200000,1,, 770,01,1,109069200000,1,, 853,01,1,111518800000,1,, ...
2
votes
1answer
135 views

string search using awk

I have file-1 which has list of transaction id . File -1 201503301616|9b8791b3-f860-409f-aad0-24debf834c1b|10.233.18.29.677220869928387637969 ...
2
votes
2answers
81 views

new pattern required from an input file using sed or awk

I have data as below: MGW: VMG110 836-16 836-18 836-19 336-20 836-23 MGW: VMG120 3802-1 3802-2 3802-3 3456-1 3456-4 Required output is: VMG110:836-16&-18&&-20&-23 ...
1
vote
1answer
47 views

I want to search and replace a pattern

I want to search each record (records are defined by blank lines) in a file for the pattern NAME#AAAA. If it matches, then insert an # in front of the record's age and insert a line age NIL at the end ...
1
vote
1answer
52 views

Merge records depending on common columns and specify the values which are uncommon in the last column as comma separated

I need to merge records depending on first three common columns and place the values which are uncommon in the last column as comma separated . I have a requirement to do the following- InputFile : ...
1
vote
2answers
1k views

How to read only the second last line of a file

I have a client that updates/writes into myfile.csv arbitrarily. I've written the following code, I left the MySQL snippet out: while true do awk_variables_value_array=`tail -n2 myfile.csv | awk ...
1
vote
1answer
223 views

How to use grep/awk/unix to match all lines from one file in another file, even if they are duplicates

I'm trying to retrieve the the matching lines from a file, however the file that has the "index" or list of matches to retrieve also has duplicate entries and I want to print out the duplicates as ...
1
vote
3answers
2k views

using awk command to compare the two files and print all the columns then missing display as NA

File 1: 123, 234 345, 789 678, 900 File 2 123, abc 345, dnc Desired output (all columns in file1 and missing value in file two as NA): 123,234,abc 345,789,dnc 678,900,NA
1
vote
2answers
2k views

Extract Values from simple html file via grep/awk

I have another grep/awk/sed issue within the attached HTML code below. Given is a simple HTML file with table inside. The HTML is generated by a smart meter (household energy meter). Inside the html ...
1
vote
1answer
242 views

AWK: Nested Conditional Subset of Rows

I have a pretty specific question, and have been able to find lots on conditional subsetting using awk but none that lends enough explicit code for me to generalize to my situation. I have a file ...
1
vote
4answers
2k views

Replace text between brackets

I'm using awk '{ gsub(/BAR|WIBBLE/, "FOO"); print }' to replace text in data like: SOMETHING [BAR, WIBBLE] SOMETHING [BAR] This gives the desired result of: SOMETHING [FOO, FOO] SOMETHING [FOO] ...
0
votes
1answer
54 views

Compare 1st column of the 1st file and 2nd column of the 2nd file and matching

File1: Excel file (.xls) UN ID St M1 M2 SE DOF PV PA FC 17127159 0 -5.9 297.3 765.7 0.22 4 0.003 0.00389231 2.57536 17127163 2 ...
0
votes
1answer
46 views

How to compare the strings using < (Greater than symbol)

Below is my input file: PA43410-2,1,3,/vobs/atlas-idc/src/utils/logger/IDCLogger/IDC.cpp,48,19:16:46.523.177, 2 PS Sensor Value = -5.501000 , Min = -5.583000 , Max = -5.319000 ...
0
votes
0answers
73 views

Replace column in one file with column from another using awk? where column from another may have null values

I have two files: f1: 111|aaa|444 222|bbb|555 333|ccc|666 f2: 111 333 replacing column 2 of f1 with column 1 of f2 required output new: 111|111|444 222||555 333|333|666 I am using ...
0
votes
1answer
45 views

Search for a pattern in a file and move it to the new files one by one

I am trying to search for a particular pattern in a file and move it to a new files one by one. I am a noob and I tried sed -n 's/CREATE PROCEDURE/,/END ;/p' File It is again printing the same file. ...
0
votes
1answer
1k views

kill processes in one line with kill, awk, ps, and grep

I have a few processes that spring up and I am able to print a line of the pgid's that I would like to enter into a kill command. here is what I have: sudo ps o pgid,args | grep mininet: | sudo awk ...
0
votes
2answers
165 views

How to delete specific lines within a certain area?

I want to delete some specific lines in the YAML front matter: --- Author: aaaa Title: bbbb Time: cccc Place: dddd ... Lets say I wanted to delete only the lines in the front matter that contained ...
0
votes
2answers
137 views

remove duplicate entry from file1 if entry present in file2

I want to delete duplicate entry e from file1 if e is also present in file2. Input file1: x1 y1 x2 y2 x3 y3 x4 y4 y1 x1 x5 y5 y3 x3 x6 y6 x5 y5 Input file2: y1 x1 y2 x2 y3 x3 y4 x4 ...