awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
3answers
51 views

Appending a character in the nth position of a matching string

I need to find a sed/awk command which will insert a character at the nth position of a matching string HAVING LENGTH 10. For example, I need to search for the string starting with pattern 541 and ...
2
votes
2answers
40 views

Merging 2 files records by template

I am trying to use awk to merge two text files in a rather peculiar way, taking two lines from file1, a group of word(s) from file2 (but placed on a separate line), alternating ad infinitum. Groups ...
4
votes
2answers
3k views

grep one line before the match plus the match

zzzzzzzzz aaaaaaaaa bbbbbbbbb & ccccccccc & ddddddddd hhhhhhhhh eeeeeeeee fffffffff & ggggggggg & in the above line, what i want is to grep/sed/awk (any method is fine) line that ...
2
votes
4answers
39 views

Merge lines between keywords into one-line comma separated values

Between first occurrence of Cat to next occurrence of Cat, it should create a separate line with delimiter as ",". File input as below. Cat AA BB CC Cat AA-1 BB-1 CC-1 Output expected: Cat,AA,BB,...
1
vote
1answer
15 views

Splitting a file into multiple files based on 1st column value [duplicate]

I have seen solutions of splitting a file with respect to pattern matching and line matching but what I want is the following. The scenario is, let's say I have a file file1 - A.B|100|20 A.B|101|...
3
votes
2answers
48 views

how to get content within a specific text from a file

I have a file name demo.txt with content as follows: value -= [ "02|05|06|abc", ] /* Some other content other than value variable */ value -= [ ] value -= [ "0698|06|07|abc", ] I have lots of ...
1
vote
4answers
82 views

Merge two files: two lines, partial line, two lines, partial line, etc [closed]

Hello I am trying to use awk to merge two text files in a rather peculiar way, taking two lines from file1, a group of word(s) from file2 (but placed on a separate line), alternating ad infinitum.  ...
3
votes
1answer
48 views

Using bash variable with escape character in awk to extract lines from file

I am writing a bash script (just learning bash) to extract some lines from a file based on two patterns. The first pattern is just a sentence ending in a colon. The second pattern is a * repeated N (...
22
votes
4answers
38k views

Pass shell variable as a /pattern/ to awk

Having the following in one of my shell functions: function _process () { awk -v l="$line" ' BEGIN {p=0} /'"$1"'/ {p=1} END{ if(p) print l >> "outfile.txt" } ' } , so when called as ...
4
votes
1answer
60 views

Splitting file content into other files based on parsed parameter value in file [duplicate]

A regular file contains data organized in columns, separated by space. Based on the value of the second field, I want to split that initial data file into several other files. The data file has 54,...
-4
votes
0answers
35 views

awk matching variable from 2 files [closed]

a='1 system1 active' is file1 b=x where x is 1..20 is from file2 HOST=$(cat $a | awk {' if ($1 == $b) print $2}') If I attempt to solve for $HOST, there is no return. However, if I run as HOST=$(...
0
votes
2answers
2k views

Sum two columns from different files

I have two files like this: File1: -54.005 -67.405 0.1 -54.05 -68.7833 0.2 -54.05 -68.7333 0.3 File2: -54.005 -67.4050 70 12.7 -54.05 -68.7833 167 12.5 -54.05 -68.7333 205 12.6 ...
-3
votes
1answer
54 views

Extracting two lines if pattern exist in the second line + clean + remove duplicates (Whatsapp extraction) [duplicate]

I want to search through a WhatsApp conversation (dumped as txt file) for any message containing the string "I will tell to your mom" as well as the message before it. However, I need to be able to ...
6
votes
4answers
413 views

How can I blank the nth to mth field using the awk command?

I would like to solve the problem below using AWK. If any other solutions are possible using languages such as sed or Perl, that would also be much appreciated. Below is the input: U,N,UNIX,000,A,5 ...
15
votes
1answer
764 views

Why does awk print the whole line when I want it to print a variable?

When I type: echo "foo bar" | gawk -v var="content" '{print $var}' I expect the output to be content. Why is it foo bar?
6
votes
1answer
522 views

What is a *.sorted command and why doesn't it work in my bash?

In a Bash Guide for Beginners (sec. 3.2.1.2. Local variables) there is an example command which is supposed to list all of the local variables, that are not in enviromental variables, or so I think: ...
21
votes
1answer
13k views

Difference between gawk vs. awk

Trying to understand the differences between the two functions gawk vs. awk? When would one use gawk vs awk? Or are they the same in terms of usage? Also, could one provide an example?
3
votes
5answers
113 views

How to repeat a block of text?

How do I use awk to repeat a block of text 100 times? ie string1 string2 string3 Would become: string1 string2 string3 string1 string2 string3 string1 string2 string3...
6
votes
4answers
13k views

Using bash shell function inside AWK

Is it possible to use bash function inside AWK somehow? Example file (string, int, int, int) Mike 247808 247809 247810 Trying to convert values from decimal to hexadecimal. Function defined ...
0
votes
1answer
29 views

awk accepting variable supplied by dollar sign in cshell [duplicate]

In c shell script, I'm trying to pass bbb to awk sub, but apparently it does not work. #!/bin/csh set aaa=a set bbb=b #And I'm using it in awk sub set ccc=`echo $aaa | awk '{sub("a",$bbb); print $...
0
votes
2answers
1k views

Bash to join columns from multiple files

I have around 20 files in different folders for which I have created a separate file called pathtofiles.in which lists path information: /home/users/gray_wolf/unix/File_1.tsv /home/users/gray_wolf/...
3
votes
1answer
30 views

Awk and discarding non-matching tokens in a string?

I'm having trouble getting a string tool to provide what I need (due to my ignorance). I have a string based on CPU features. The string will change as different processors offer different features: #...
0
votes
0answers
30 views

How To Create Soft Links To An Entire Directory Structure Which Has Sub-Directories, Files Etc! [duplicate]

I'd like to: Create a soft link for a directory or file. But have a scenario where I need to create soft links, to a directory which has multiple sub-directories, files etc. I wanted to loop across ...
7
votes
1answer
5k views

remove lines where a field's value is less than or equal to 3 - sed or awk?

I need to remove every line that has a value of 2 or less in the 8th field (column). My data looks like this: 12-31 Airport 189 379 41 49.70946503 -124.91377258 2 2880 30.8 01-01 ...
1
vote
4answers
949 views

How to extract column name (header) from a CSV file which contains the max value in a row?

I am trying to extract the column name with the maximum value in each row using bash script i.e., the column header value or the value from the same column in the first row. I am using the following ...
0
votes
1answer
32 views

Variable Capture and For-Loop Arithmetic

My input files are a single column with variable length of rows. My code is supposed to count those rows, and print a specified number. The problem I'm having, is if the number of rows captured in the ...
0
votes
1answer
46 views

How to calculate the standard deviation for each row?

I have file of 61 columns; I would like to calculate the standard deviation (SD) for each row with ignoring the 1st column and print the results against the 1st column, please note that the number of ...
1
vote
3answers
64 views

How to grep/awk/sed for text in a log and display the chunk that has the text?

I'm looking for something pretty similar to this. The logs look like this: [09:44:22] [main] ERROR [url/location] - A ONE LINE ERROR [09:44:22] [main] ERROR [url/location] - Another ERROR [09:44:22]...
6
votes
1answer
223 views

Each grep result to different file

I have something like grep "keyword" -B 3 log. It show a lot of results like: some trash [GET] /orders/42 one more trash keyword and usefull info [GET] /orders/43 some trash one more trash keyword ...
2
votes
1answer
48 views

how to append words of two files word by word [duplicate]

sorry for my newbie question. I have two files with the same number of words and column fields in each line. I want to append these two files word by word. For example: File 1: A1 B1 C1 D1 E1 B1 C1 ...
1
vote
1answer
35 views

remove lines that contain IP range from specific column while making exclusions to range

I have a logfile with the following format: Jul 13 21:47:41 192.168.100.254 "user from 192.168.100.101" I need to remove ALL lines that contain IP's in the 192.168.x.x range but only if they appear ...
-1
votes
1answer
21k views

convert CSV to XLS file on linux [closed]

The following Perl script can convert CSV file to XLS file the problem is that I need to install on customer Linux machine many Perl modules in order to run this Perl script , and actually I cant ...
0
votes
2answers
64 views

deleting lines between rows in a text file using awk or sed

I am wondering if there is a sed or awk command to delete all lines between the 'Query_' headers in column 1, if the number of lines between each header is less than 5. The following is an extract ...
4
votes
3answers
732 views

UNIX paste columns and insert zeros for all missing values

I would like to merge specific columns from two txt files containing varying number of rows, but same number of columns (as shown below): file1: xyz desc1 12 uvw desc2 55 pqr desc3 ...
6
votes
3answers
38k views

Adding a Column of values in a tab delimited file

How can I add a Column of values in a file which has a certain number of rows. I have a input file like this: Input file: SPATA17 1 217947738 LYPLAL1 1 219383905 FAM47E 4 77192838 SHROOM3 4 ...
26
votes
13answers
11k views

Remove duplicate $PATH entries with awk command

I am trying to write a bash shell function that will allow me to remove duplicate copies of directories from my path environment. I was told that it is possible to achieve this with a one line ...
0
votes
2answers
33 views

Extracting all columns containing a list of terms

I have a data file (data.txt) with ~320 rows and ~1800 columns (1.9 MB), and I need to extract certain columns out (for all rows). The general layout of data.txt is as follows: ID ...
1
vote
3answers
65 views

Output lines between two parameter dates INCLUDING the first parameter date but not the second

This questions has been asked before, except they were looking for text between two dates including or excluding those dates. I’ve changed one of the answers so that it selects and includes from first ...
7
votes
7answers
350 views

grep the exact block of lines (content of file1) from file2

I have two files, file1 and file2. The sample content of file1 is: A B C D E F G H and the content of file2 is like: A B few other lines E F few more other lines A B C D E F G H few more other ...
5
votes
4answers
242 views

sed/awk remove newline on two pattern matches

I have a .pem file that after some formatting it shows like this: -----BEGIN RSA PRIVATE KEY----- MIIEogIBAAKCAQEAoK3D4mMIRnzPaYqHidgpBnDDzLlcYYd0GoB0pQGyGSHDW7KO3K+VeJP90GhE ZTEWJLp2N5DR/KT+5Vg7cgdx/...
7
votes
3answers
281 views

Side-by-side comparison of more than two files containing numerical values

I have three files containing a sorted sequence of numbers, one per line : file1 1 2 3 file2 1 3 4 file3 1 5 I want to "align" these three files side-by-side like the following : file1 ...
3
votes
1answer
62 views

Parse (grepable) nmap output to print a list of IP\t[all open ports] with text utils like awk

I would like to find a way to print for each IP address found to have at least one open port, to print that IP address, followed by a list of open ports separated by commas. The ports and IP address ...
1
vote
5answers
69 views

Getting substrings in between delimiters multiple times from one string without looping

I have looked all over for a solution to this, but I couldn't find one. I am using qstat -x to pass a large string of job information. The output from qstat -x is in XML format. The substrings I am ...
4
votes
5answers
1k views

Merge alternate lines from two files

File1: .tid.setnr := 1123 .tid.setnr := 3345 .tid.setnr := 5431 .tid.setnr := 89323 File2: .tid.info := 12 .tid.info := 3 .tid.info := 44 .tid.info := 60 Output file: .tid.info := 12 .tid....
2
votes
1answer
122 views

Sum of columns for rows containing a specific term

I'm wondering if there is any sed or awk command that can sum all columns for the rows with an identical identifier. For example, my file data.txt is set out as below, except that it contains ~1800 ...
6
votes
6answers
3k views

Find images by size: find / file / awk

I've been trying to find png image files a certain height (over 500px). I know that file will return image dimensions. Example: $ file TestImg1a.png TestImg1a.png: PNG image data, 764 x 200, 4-...
4
votes
3answers
105 views

How to print same lines multiple times with variables changed using Sed/Awk/ anything?

a. Have set of lines as given below. cr_v8_sel0 : cross cp_v8_en, cp_sel0 { ignore_bins ig_v8_sel0 = binsof(cp_v8_en) && binsof(cp_sel0) intersect {[0:3],[5:$]}; } b. I wanted to print ...
1
vote
1answer
37 views

read patterns from file1 and search them iteratively in file2

I'd like to read patterns from file1 and search them iteratively in file2, printing the matched line plus 3 rows, and print them to file3. Now I am doing: cat file1 | while read line; do awk '/$...
6
votes
4answers
10k views

How to remove duplicate lines with awk whilst keeping empty lines?

Below awk command removes all duplicate lines as explained here: awk '!seen[$0]++' If the text contains empty lines, all but one empty line will be deleted. How can I keep all empty lines whilst ...
0
votes
1answer
52 views

Computing standard deviation with a known mean

I have many, two column files. Column 1 contains variable numbers while column two contains the 'true' of those numbers. i.e. file 1 41619.442 41586.441 41474.2539 41586.441 41587.539062 41586.441 ...