awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

0
votes
1answer
43 views

AWK output help

I am currently writing something to parse some apache logs, yet the system command seems to be putting itself above print. Haven't done a whole lot with awk, so it could be something very simple. IFS=...
4
votes
2answers
35 views

How to split a file alternating the prefix used for the output files?

I have large file. It is made of parts of 40 lines each. There are two types of parts and they alternate. The two types of parts should be numbered independently. So the first part should be X_0001, ...
2
votes
4answers
93 views

awk / sed / etc. concatenating colums in one file

I wonder what is a simpler way to do this: awk 'NR > 1 {print $1"\t"$2"\t"$3"\t"$4"\t"$5"\t"$6"\t"$7"\t"$8"\t"$9$10$11$12$13$14$15$16}' file.in > file.out which is simply speaking " ...
0
votes
2answers
24 views

add lines for each 3 lines [duplicate]

I have a document like this 1 0.2 1 0.3 1 0.1 2 0.8 2 0.5 2 0.3 I want to add the lines if the first column are the same and output something like this 1 0.6 2 1.6 How can I do this with awk? Or ...
2
votes
2answers
40 views

Merging 2 files records by template

I am trying to use awk to merge two text files in a rather peculiar way, taking two lines from file1, a group of word(s) from file2 (but placed on a separate line), alternating ad infinitum. Groups ...
2
votes
4answers
39 views

Merge lines between keywords into one-line comma separated values

Between first occurrence of Cat to next occurrence of Cat, it should create a separate line with delimiter as ",". File input as below. Cat AA BB CC Cat AA-1 BB-1 CC-1 Output expected: Cat,AA,BB,...
1
vote
1answer
17 views

Splitting a file into multiple files based on 1st column value [duplicate]

I have seen solutions of splitting a file with respect to pattern matching and line matching but what I want is the following. The scenario is, let's say I have a file file1 - A.B|100|20 A.B|101|...
3
votes
2answers
50 views

how to get content within a specific text from a file

I have a file name demo.txt with content as follows: value -= [ "02|05|06|abc", ] /* Some other content other than value variable */ value -= [ ] value -= [ "0698|06|07|abc", ] I have lots of ...
1
vote
3answers
53 views

Appending a character in the nth position of a matching string

I need to find a sed/awk command which will insert a character at the nth position of a matching string HAVING LENGTH 10. For example, I need to search for the string starting with pattern 541 and ...
3
votes
1answer
49 views

Using bash variable with escape character in awk to extract lines from file

I am writing a bash script (just learning bash) to extract some lines from a file based on two patterns. The first pattern is just a sentence ending in a colon. The second pattern is a * repeated N (...
1
vote
4answers
83 views

Merge two files: two lines, partial line, two lines, partial line, etc [closed]

Hello I am trying to use awk to merge two text files in a rather peculiar way, taking two lines from file1, a group of word(s) from file2 (but placed on a separate line), alternating ad infinitum.  ...
-4
votes
0answers
36 views

awk matching variable from 2 files [closed]

a='1 system1 active' is file1 b=x where x is 1..20 is from file2 HOST=$(cat $a | awk {' if ($1 == $b) print $2}') If I attempt to solve for $HOST, there is no return. However, if I run as HOST=$(...
-3
votes
1answer
56 views

Extracting two lines if pattern exist in the second line + clean + remove duplicates (Whatsapp extraction) [duplicate]

I want to search through a WhatsApp conversation (dumped as txt file) for any message containing the string "I will tell to your mom" as well as the message before it. However, I need to be able to ...
3
votes
5answers
114 views

How to repeat a block of text?

How do I use awk to repeat a block of text 100 times? ie string1 string2 string3 Would become: string1 string2 string3 string1 string2 string3 string1 string2 string3...
0
votes
1answer
29 views

awk accepting variable supplied by dollar sign in cshell [duplicate]

In c shell script, I'm trying to pass bbb to awk sub, but apparently it does not work. #!/bin/csh set aaa=a set bbb=b #And I'm using it in awk sub set ccc=`echo $aaa | awk '{sub("a",$bbb); print $...
3
votes
1answer
31 views

Awk and discarding non-matching tokens in a string?

I'm having trouble getting a string tool to provide what I need (due to my ignorance). I have a string based on CPU features. The string will change as different processors offer different features: #...
4
votes
1answer
60 views

Splitting file content into other files based on parsed parameter value in file [duplicate]

A regular file contains data organized in columns, separated by space. Based on the value of the second field, I want to split that initial data file into several other files. The data file has 54,...
0
votes
0answers
30 views

How To Create Soft Links To An Entire Directory Structure Which Has Sub-Directories, Files Etc! [duplicate]

I'd like to: Create a soft link for a directory or file. But have a scenario where I need to create soft links, to a directory which has multiple sub-directories, files etc. I wanted to loop across ...
0
votes
1answer
49 views

How to calculate the standard deviation for each row?

I have file of 61 columns; I would like to calculate the standard deviation (SD) for each row with ignoring the 1st column and print the results against the 1st column, please note that the number of ...
0
votes
1answer
33 views

Variable Capture and For-Loop Arithmetic

My input files are a single column with variable length of rows. My code is supposed to count those rows, and print a specified number. The problem I'm having, is if the number of rows captured in the ...
1
vote
3answers
64 views

How to grep/awk/sed for text in a log and display the chunk that has the text?

I'm looking for something pretty similar to this. The logs look like this: [09:44:22] [main] ERROR [url/location] - A ONE LINE ERROR [09:44:22] [main] ERROR [url/location] - Another ERROR [09:44:22]...
6
votes
1answer
223 views

Each grep result to different file

I have something like grep "keyword" -B 3 log. It show a lot of results like: some trash [GET] /orders/42 one more trash keyword and usefull info [GET] /orders/43 some trash one more trash keyword ...
2
votes
1answer
48 views

how to append words of two files word by word [duplicate]

sorry for my newbie question. I have two files with the same number of words and column fields in each line. I want to append these two files word by word. For example: File 1: A1 B1 C1 D1 E1 B1 C1 ...
1
vote
1answer
35 views

remove lines that contain IP range from specific column while making exclusions to range

I have a logfile with the following format: Jul 13 21:47:41 192.168.100.254 "user from 192.168.100.101" I need to remove ALL lines that contain IP's in the 192.168.x.x range but only if they appear ...
0
votes
2answers
65 views

deleting lines between rows in a text file using awk or sed

I am wondering if there is a sed or awk command to delete all lines between the 'Query_' headers in column 1, if the number of lines between each header is less than 5. The following is an extract ...
0
votes
2answers
34 views

Extracting all columns containing a list of terms

I have a data file (data.txt) with ~320 rows and ~1800 columns (1.9 MB), and I need to extract certain columns out (for all rows). The general layout of data.txt is as follows: ID ...
5
votes
4answers
245 views

sed/awk remove newline on two pattern matches

I have a .pem file that after some formatting it shows like this: -----BEGIN RSA PRIVATE KEY----- MIIEogIBAAKCAQEAoK3D4mMIRnzPaYqHidgpBnDDzLlcYYd0GoB0pQGyGSHDW7KO3K+VeJP90GhE ZTEWJLp2N5DR/KT+5Vg7cgdx/...
1
vote
3answers
65 views

Output lines between two parameter dates INCLUDING the first parameter date but not the second

This questions has been asked before, except they were looking for text between two dates including or excluding those dates. I’ve changed one of the answers so that it selects and includes from first ...
7
votes
3answers
283 views

Side-by-side comparison of more than two files containing numerical values

I have three files containing a sorted sequence of numbers, one per line : file1 1 2 3 file2 1 3 4 file3 1 5 I want to "align" these three files side-by-side like the following : file1 ...
2
votes
1answer
126 views

Sum of columns for rows containing a specific term

I'm wondering if there is any sed or awk command that can sum all columns for the rows with an identical identifier. For example, my file data.txt is set out as below, except that it contains ~1800 ...
7
votes
7answers
354 views

grep the exact block of lines (content of file1) from file2

I have two files, file1 and file2. The sample content of file1 is: A B C D E F G H and the content of file2 is like: A B few other lines E F few more other lines A B C D E F G H few more other ...
3
votes
1answer
62 views

Parse (grepable) nmap output to print a list of IP\t[all open ports] with text utils like awk

I would like to find a way to print for each IP address found to have at least one open port, to print that IP address, followed by a list of open ports separated by commas. The ports and IP address ...
4
votes
3answers
105 views

How to print same lines multiple times with variables changed using Sed/Awk/ anything?

a. Have set of lines as given below. cr_v8_sel0 : cross cp_v8_en, cp_sel0 { ignore_bins ig_v8_sel0 = binsof(cp_v8_en) && binsof(cp_sel0) intersect {[0:3],[5:$]}; } b. I wanted to print ...
1
vote
1answer
37 views

read patterns from file1 and search them iteratively in file2

I'd like to read patterns from file1 and search them iteratively in file2, printing the matched line plus 3 rows, and print them to file3. Now I am doing: cat file1 | while read line; do awk '/$...
0
votes
1answer
53 views

Computing standard deviation with a known mean

I have many, two column files. Column 1 contains variable numbers while column two contains the 'true' of those numbers. i.e. file 1 41619.442 41586.441 41474.2539 41586.441 41587.539062 41586.441 ...
3
votes
3answers
280 views

How to print a specific column with awk on a remote ssh session?

When using df -h | grep /dev/root | awk '{print $5}' I get the usage of my SD Card in my Pi: 78% But when I use /usr/bin/ssh -i /path/to/key user@server "df -h | grep /dev/root | awk '{print $5}'" ...
0
votes
0answers
18 views

How to add a # to the begining of a line, only specific lines that I grep for [duplicate]

I have a file that has many thousands of lines. I what to comment out only specific lines in the file. I want to add a "#" to the beginning of the greped for line plus 2 lines above and one line below ...
2
votes
3answers
43 views

Need to parse sections of text from repetitive “groupings” of lines within a file

I have a file that contains 3-5 lines of information pertaining to 1 instance of a Multicast group and the file contains multiple instances of Multicast groups: Group: 239.103.0.64 Source: 10.12.49....
-1
votes
3answers
33 views

Need help on extracting bad records from flat files using awk [closed]

Need a script to extract the row in the file which contains more than specified columns using awk?
1
vote
2answers
40 views

Joining two files matching two columns with mismatches and in each matching line, substitute second column from file 1 into 6th column in file 2

I'm struggling with this task (novice in this world...): I have two files: file1 looks like: 102 13.342 103 7.456 105 6.453 107 3.567 108 4.210 file2 looks like: 0 098 0 0 0 -9 x 0 099 0 0 0 -9 x ...
2
votes
2answers
47 views

help to construct grep output

I have a command I am using #cat test |grep "Ongoing Calls" |awk -F' ' '{print $3}' 40 2 As you can see it returns 2 values 40 & 2 I want to now add these together and put into a file as 42 ...
0
votes
0answers
63 views

Insert missing numbers in a sequence, make corresponding column default to `0` [duplicate]

I want to include any missing numbers in a sequence, and introduce 0 as default value in the corresponding columns which would otherwise be blank. Input "afile": 1 4 2 7 3 8 4 3 10 145 ...
0
votes
0answers
28 views

Linux Shell Script Question. Comment out particular lines in a file [duplicate]

I have a very large file. I will put a short example below. I need to find a line containing a particular string, Then comment out that line, plus 2 lines above and 1 line below. also must maintain ...
0
votes
3answers
59 views

Want to print a table format with multiple line in shell scipt

I want to print the line in a table format using sed or awk or any other in a shell script. example Line1 Line2 Line3 Line4 Line5 Line6 Line7 Line8 Desired Output Line1 Line2 Line3 Line4 Line5 ...
1
vote
1answer
40 views

how to change contents of one column in a file?

I have a file that looks something like that: 1 mm10_rmsk exon 67108753 67108881 239.000000 + . gene_id "RLTR17B_Mm"; transcript_id "RLTR17B_Mm"; 1 mm10_rmsk exon 134217652 ...
11
votes
3answers
568 views

Merging two files, one column at a time

I have 2 large files (3000 columns, 15000 rows) of the following format file1 (tab-separated): 1/0 0/0 0/0 0/0 1/1 0/0 1/1 0/1 0/0 file2 (tab-separated): 3 5 2 1 7 10 3 4 3 I'd like to combine ...
9
votes
6answers
423 views

Rearranging columns using awk

I am trying to move 7th column of my csv file to the end by using awk -F '{print $1,$2,$3,$4,$5,$6,$8,$9,$10,$11,$7}',OFS= "$file" where $file is a .csv file in a directory. However, the output is ...
-1
votes
4answers
86 views

Extracting IPs from a web list

I am trying to extract IPs from a huge web generated list which contains a mix of text, IPv4 & IPv6 IPs, newlines, IP ranges etc. Below is a snippet from the list: ; Spamhaus DROP List 2016/07/...
2
votes
5answers
312 views

Reducing the complexity of the data in text file

I have this file: 1 2 2 7 3 4 4 7 5 3 6 7 7 1 8 2 9 4 And my desired output is 1 13 2 17 3 7 In my input i have 9 rows and i want to reduce it to three rows while ...
1
vote
2answers
84 views

Adding new columns in a csv file

I want to append two new columns in the beginning of a csv file using bash scripting in a way such that the first four letters of the filename are the values of the first column and the next two ...