awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

14
votes
9answers
6k views

How to remove multiple newlines at EOF?

I have files that end in one or more newlines and should end in only one newline. How can I do that with Bash/Unix/GNU tools? Example bad file: 1\n \n 2\n \n \n 3\n \n \n \n Example corrected file:...
3
votes
6answers
190 views

Find a pattern and insert # at the beginning of 2 line before that and 1 line after that

I have a big file (more than 2000 lines). Where I have to insert a # at the beginning of 2 above lines and at the beginning of 1 below line after finding a pattern. Also insert # at the beginning of ...
2
votes
2answers
47 views

help to construct grep output

I have a command I am using #cat test |grep "Ongoing Calls" |awk -F' ' '{print $3}' 40 2 As you can see it returns 2 values 40 & 2 I want to now add these together and put into a file as 42 ...
0
votes
0answers
18 views

How to add a # to the begining of a line, only specific lines that I grep for [duplicate]

I have a file that has many thousands of lines. I what to comment out only specific lines in the file. I want to add a "#" to the beginning of the greped for line plus 2 lines above and one line below ...
2
votes
3answers
43 views

Need to parse sections of text from repetitive “groupings” of lines within a file

I have a file that contains 3-5 lines of information pertaining to 1 instance of a Multicast group and the file contains multiple instances of Multicast groups: Group: 239.103.0.64 Source: 10.12.49....
0
votes
3answers
58 views

Want to print a table format with multiple line in shell scipt

I want to print the line in a table format using sed or awk or any other in a shell script. example Line1 Line2 Line3 Line4 Line5 Line6 Line7 Line8 Desired Output Line1 Line2 Line3 Line4 Line5 ...
0
votes
0answers
63 views

Insert missing numbers in a sequence, make corresponding column default to `0` [duplicate]

I want to include any missing numbers in a sequence, and introduce 0 as default value in the corresponding columns which would otherwise be blank. Input "afile": 1 4 2 7 3 8 4 3 10 145 ...
-1
votes
3answers
33 views

Need help on extracting bad records from flat files using awk [closed]

Need a script to extract the row in the file which contains more than specified columns using awk?
1
vote
2answers
39 views

Joining two files matching two columns with mismatches and in each matching line, substitute second column from file 1 into 6th column in file 2

I'm struggling with this task (novice in this world...): I have two files: file1 looks like: 102 13.342 103 7.456 105 6.453 107 3.567 108 4.210 file2 looks like: 0 098 0 0 0 -9 x 0 099 0 0 0 -9 x ...
-1
votes
4answers
85 views

Extracting IPs from a web list

I am trying to extract IPs from a huge web generated list which contains a mix of text, IPv4 & IPv6 IPs, newlines, IP ranges etc. Below is a snippet from the list: ; Spamhaus DROP List 2016/07/...
0
votes
0answers
28 views

Linux Shell Script Question. Comment out particular lines in a file [duplicate]

I have a very large file. I will put a short example below. I need to find a line containing a particular string, Then comment out that line, plus 2 lines above and 1 line below. also must maintain ...
1
vote
1answer
40 views

how to change contents of one column in a file?

I have a file that looks something like that: 1 mm10_rmsk exon 67108753 67108881 239.000000 + . gene_id "RLTR17B_Mm"; transcript_id "RLTR17B_Mm"; 1 mm10_rmsk exon 134217652 ...
11
votes
3answers
561 views

Merging two files, one column at a time

I have 2 large files (3000 columns, 15000 rows) of the following format file1 (tab-separated): 1/0 0/0 0/0 0/0 1/1 0/0 1/1 0/1 0/0 file2 (tab-separated): 3 5 2 1 7 10 3 4 3 I'd like to combine ...
9
votes
6answers
413 views

Rearranging columns using awk

I am trying to move 7th column of my csv file to the end by using awk -F '{print $1,$2,$3,$4,$5,$6,$8,$9,$10,$11,$7}',OFS= "$file" where $file is a .csv file in a directory. However, the output is ...
3
votes
2answers
110 views

Find each line matching a pattern but print only the line above it

I need to find a string and need to print the line above it. case1: The same line won't have the more than one matching pattern ie) consider a file containing $cat > para returns between the ...
1
vote
2answers
77 views

Adding new columns in a csv file

I want to append two new columns in the beginning of a csv file using bash scripting in a way such that the first four letters of the filename are the values of the first column and the next two ...
1
vote
3answers
64 views

awk to consolidate large tabular file?

I have a file with 21 tabular fields in columns. Tabs 14 and 15 are sets of data which are repeated several times for variables in tab 10(up to ":") and tab 11 has numeric descriptive data for tab 10 ....
2
votes
5answers
310 views

Reducing the complexity of the data in text file

I have this file: 1 2 2 7 3 4 4 7 5 3 6 7 7 1 8 2 9 4 And my desired output is 1 13 2 17 3 7 In my input i have 9 rows and i want to reduce it to three rows while ...
3
votes
3answers
400 views

Use awk to pad date and time formats

I would like to use awk to pad the date and time fields in my report. Here is my current .csv that is actually tab delimited. AA 9/7/2014 2:30:38 PM AA0000011111 08 Y A Jane, Doe AA ...
0
votes
4answers
3k views

Use SED or AWK to move rows to columns

I have a report that is generated in rows and I need to move it to columns. The number of rows per record will always be the same. This will be in a bash script on Linux, so I would like to use ...
1
vote
1answer
141 views

Move data row(s) to single column while retaining row header(s)

I have reports that are generated in the following tab-delimited format: UNIT TC CC PC TCP FTX FRX HOUSE 55 65 75 85 95 105 CAR 100 200 300 400 500 600 H2 ...
1
vote
5answers
892 views

Use SED or AWK to move string to new column and row

I have a .csv file that is tab delimited and I need to move the header row information to a "new" column at the beginning and a specified numbers of rows down. I would like to use standard tools that ...
2
votes
6answers
128 views

Comparing 3 lines and 2 lines in BASH

I have a text file which looks like this: 2014-11-24 12:59:42.169 101.0.0.0 source 2014-11-24 12:59:40.375 104.156.80.0 destination 2014-11-24 12:59:36.729 104.219.48.0 destination 2014-11-24 12:59:...
3
votes
3answers
79 views

Basic grep/awk help - extracting all lines containing a list of terms from one file into a seperate file

I have two files genelist.txt and data.txt. genelist.txt simply contains a single column of ~500 gene names, whereas data.txt is a tab delimited file that contains ~1000 columns (the samples) and ~30,...
6
votes
4answers
1k views

How to add a header and/or footer to a sed or awk stream?

I have a bunch of output going through sed and awk. How can I prefix the output with START and suffix the answer with END? For example, if I have All this code on all these lines and all these ...
10
votes
3answers
7k views

Why does awk do full buffering when reading from a pipe

I'm reading from a serial port connected to a gps device sending nmea strings. A simplified invocation to illustrate my point: $ awk '{ print $0 }' /dev/ttyPSC9 GPGGA,073651.000,6310.1043,N,...
0
votes
3answers
34 views

run a command in subfolders

I have a command rdseed -d -R -p -f filename which gives an output of response time and etc for each file in a subfolder. The problem is I want to run this command in all subfolders and the output (...
3
votes
1answer
67 views

awk or sed command to start each column of file from desired locations or Byte Position

I have a source file format like below containing 15 columns and so many rows: 23.5 31.5 5.00 255 0 255 2 1 sal unknown sa time sps meter ms 25.5 32.5 6.00 0 255 0 2 2 sal unknown sa time sps ...
14
votes
5answers
629 views

Find files that contain multiple keywords anywhere in the file

I'm looking for a way to list all files in a directory that contain the full set of keywords I'm seeking, anywhere in the file. So, the keywords need not to appear on the same line. One way to do ...
0
votes
3answers
39 views

Count the number of occurrences of specific number before each string in linux

I have this file : 90 1 120 1 Transition 150 1 Transition 165 1 Transition 180 1 225 1 240 1 255 1 270 1 Transition 285 1 Transition and I want this output: 1 2 2 1 3 1 4 5 5 ...
0
votes
1answer
36 views

awk csv file append numbers

I am have the below CSV file vol123,12/05/2016,ABC vol123,12/05/2016,ABC vol123,12/05/2016,ABC vol123,14/05/2016,ABC vol123,13/05/2016,ABC vol123,12/05/2016,ABC vol123,12/05/2016,ABC vol123,18/05/...
1
vote
2answers
46 views

Inconsistent behavior in awk

I have a text file called junk.txt: hello foo 0 -1 0 1 0 2 0 foo Cindy 00000 Lou 2 000 0 Who 0000 0 wat? 0000 00000 0 0 0000 00000 filler 00 0 00 000 0000 0 0 bye When I run the following I ...
1
vote
2answers
33 views

using gsub in awk to replace a word with parentheses

Hi how can I use gsub to replace a word which has parentheses. Here I want to replace ABC(T) with ABC/G awk ' {gsub("ABC\(T\)","ABC/G")}; Print $0' "$FILENAME" > tmp.tmp && mv tmp.tmp "$...
2
votes
1answer
387 views

Read from and append to file at the same time while preserving end-of-line

Conceptually I have an easy task... I have [loosely] structured data in a file: Testing: debug, default CXXFLAGS <100's of additional output lines> Testing: release, default CXXFLAGS <100's ...
3
votes
2answers
71 views

confused with awk and shell variables

I am trying to run a script that uses awk. There is no error it seems, but it doesn't create the file that I want. This is the script: #!/bin/bash echo "Number of domain:" read number echo "Domain ...
4
votes
4answers
354 views

How to print lines between pattern1 and 2nd match of pattern2?

Test file is given below: PATTERN1 a b c PATTERN2 d e f PATTERN2 g h I want to print line between PATTERN1 and 2nd match of PATTERN2: PATTERN1 a b c PATTERN2 d e f PATTERN2
63
votes
16answers
76k views

Is there a way to get the min, max, median, and average of a list of numbers in a single command?

I have a list of numbers in a file, one per line. How can I get the minimum, maximum, median and average values? I want to use the results in a bash script. Although my immediate situation is for ...
-3
votes
1answer
38 views

text to columns in CSV

I am having data in CSV file as follows in single column: Input: Company;C Desc;Region;R Desc; H;Hindustan;13;Maharashtra; I;Unilever;28;Dadra; I would like to change the data Text to Columns with ...
0
votes
2answers
231 views

How to update three CSV files based on a fourth file

I have four csv files, and I want to update a set of three, based on the values in the fourth file. file_1 contains names. file_2 their numerical ids. file_3 contains ids from file_1 and file_2, ...
1
vote
3answers
94 views

Combine matching columns from two files, or use default value

I am trying to rewrite a file text, main.txt with changes from changes.txt. The main.txt file is like this: word_1 genre_A word_2 genre_A word_3 genre_B word_4 genre_C word_5 genre_A And the ...
4
votes
5answers
2k views

Remove lines based on pattern but keeping first n lines that match

I need to remove lines from a text file based on pattern but I need to keep the first n lines that match the pattern. Input % 1 % 2 % 3 % 4 % 5 text1 text2 text3 output %1 %2 text1 text2 text3 ...
3
votes
1answer
76 views

Split file by pattern keeping only the pieces that contain a second pattern

I found the following awk script in a previous answer for splitting large files into smaller parts based on a pattern (replacing "header-line" with a different pattern). However, I have the need to ...
1
vote
1answer
49 views

Better way to apply formula to specific column of inputfile, and pass-through all other columns as-is

I regularly need to apply simple formulas to input data give in textual tabular form. In the beginning, I wrote specific little functions for each use-case. Then I re-factored the "common code" into ...
4
votes
2answers
87 views

How Do I Remove Duplicate Words With Suffixes?

How do I compare a list of unsorted words of variable length and remove duplicate words that have a suffix? Example word list: iron curl curled railroad curler ...
0
votes
1answer
48 views

sum of numbers located in different rows using awk

The following lines sum numbers located in the row #2 and row #3 , row #4 and row #5, row #6 and row #7 and so on (so, sum in a consecutive way). awk ' NR==1{$1=$1; print; next} !(NR%2){split($0,a);...
1
vote
2answers
10k views

How to remove a column or multiple columns from file using shell command?

My file, ARCHIVE B1_NAME B2_NAME B3_NAME ELEMENT INFO_NAM WERT PROCID -------- -------- -------- -------- -------- -------- ---- ------ ...
1
vote
1answer
32 views

Calculate hits within a timespan from log file [closed]

I need to use UNIX utilities that implement out-of-core divide-and-conquer algorithms so to decrease the amount of virtual memory used (cap of 1 mb). I have a large data set that is an apache log file....
1
vote
2answers
73 views

how to use \n( and )\n as delimiter in awk

I have a file which contains below data 7[label = "ScanStep: T_b0\n(T.a = 1)\na\nb\nc\nd\ne\ndob\ntimestamp1\nUnc: Integer(8)\nUnc: Integer(8)\nUnc: Varchar(80)\nUnc: Numeric(10,2)\nUnc: Varbinary(80)...
1
vote
1answer
63 views

Need to strip off the comments in double quotes

I have test case which contains string message in double quotes and I want to strip off data in double quotes. I used awk command to strip off comment section data but it doesn't work awk '/^"/ { ...
1
vote
3answers
51 views

Reformatting negative numbers with sed or awk

I have a text file that contains some negative numbers except, they are formatted as: 00000012- With the hyphen after the number. I am processing the file using awk to reformat it into SQL insert ...