awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

10
votes
7answers
17k views

How to read first and last line from cat output?

I have text file. Task - get first and last line from file after $ cat file | grep -E "1|2|3|4" | commandtoprint $ cat file 1 2 3 4 5 Need this without cat output (only 1 and 5). ~$ cat file | ...
0
votes
3answers
85 views

Find all words beginning with Q

How do I search a text file and list all the words begining with Q? The file I am searching is Python source code. The closest I have found to a soulution is this question but that is not what I ...
0
votes
2answers
24 views

get specific variable using awk

I have this file.txt kespaul pts/14 2001:778:200:180 Fri May 29 11:33 - 16:40 (00:07) kespaul sshd 2001:778:200:180 Fri May 29 16:33 - 16:40 (00:07) algvai pts/2 ...
1
vote
1answer
39 views

finding lines in a file matching patterns defined in another file

I have two files file1.txt kepaul jack mike david and file2.txt 1 london smith 12564 2 miami jack 51616 3 la mike 165156 4 london kepaul 54654 5 miami grag 46546 and I need to print to a new ...
1
vote
2answers
46 views

Specify String Lenth in Awk Pattern Search

I'm not sure what I'm doing wrong. Given a test.log of: 123.123.123.123 321.321.321.321 GET /test1234/asdfasdfpioasdfjkhweris,9 123.123.123.123 321.321.321.321 GET ...
2
votes
2answers
123 views

how to print field between the double quote

I have the following file conf.txt: perl /home/site/gen.pl "GG - W C1 Test - sai1" "15072" && ssh 13.4.93.103 perl /home/site/gen.pl "GG - W C1 Test - sai2" "15072" && ssh ...
0
votes
0answers
34 views

awk search for string containing forward slash in variable [duplicate]

I'm searching for a string in a massive XML file. Snip: <sv:node sv:name="notfound"> <sv:property sv:name="jcr:primaryType" sv:type="Name"> ...
1
vote
1answer
125 views

How can I combine information of three files to create a fourth one?

I have three csv files, each of them containing the information I need to get a fourth one. I'm describing toy files but the actual ones contain much more columns and rows, still on the same models. ...
2
votes
2answers
54 views

Add a number in a huge ASCII file

Background: (1) Here is a screen capture of a part of my ascii file (over 600Mb): (1.1) Here is a part of the code: 0, 0, 0, 0, 0, 0, 0, 0, 3.043678e-05, 3.661498e-05, 2.070347e-05, ...
3
votes
3answers
74 views

Specific column level substitution

If I have a file with columnar entries, is it possible to substitute specific entries from a specific columns only and save changes to the same file?? Example a file abc.txt: SR|FRUITS|COLOR| ...
1
vote
2answers
4k views

Type URL and show HTTP status

I want to make a shell script where I type an url and it returns me the status code.So far I've tested wget is there any other cause it has the tendency to save whatever you run. Then I tried to some ...
0
votes
1answer
37 views

Move a range of words to the end of the string

I have the following string: "5 S comment=whatever Unspecific text fruit=apple animal=mouse sky=blue" I need to move to end of the string, all that begin with comment to the word text. The start ...
34
votes
13answers
51k views

Is there a way to get the min, max, median, and average of a list of numbers in a single command?

I have a list of numbers in a file, one per line. How can I get the minimum, maximum, median and average values? I want to use the results in a bash script. Although my immediate situation is for ...
2
votes
2answers
86 views

Delete blank columns

This must be a common problem, but I couldn't find a solution. I have several files which have different sets of columns which are completely blank (for all rows except header) , how can I delete ...
3
votes
1answer
462 views

What is an array in awk?

I'm very new to Unix and am trying to learn basic awk scripting - during which I keep coming across the word "array", and "associative arrays". I have no clue as to what this is; and don't tell me to ...
2
votes
4answers
149 views

Awk - Print row if number is between column 1 and column 2

I have a file with data formatted like this. Input file "1930","1946","FOO","BAR","GREEN" "1949","1962","FOO","BAR","BLUE" "1970","1980","FOO","BAR","YELLOW" Numbers are not continuous, with some ...
1
vote
3answers
52 views

Data extraction from text file

I have a text file in the following format:- $DATA1 SOURCE='HSPICE' VERSION='I-2013.12-SP2-1 32-BIT' .TITLE 'cmos inverter transfer characteristics' index t_r t_f ...
-1
votes
2answers
55 views

Error in file comparison - awk [duplicate]

I have two files file1.txt, file2.txt. I want to select records which are common to both file1.txt and file2.txtbased on column 2 and column 3 and copy them to out.txt with additional column(column 4 ...
0
votes
1answer
50 views

How do I add a \n to the end of the record on file using AWK

I have a file that can contain 1 or many lines oracle1.txt oracle2.txt oracle3.txt How do I using AWK put a \n after the last line only so it will show oracle1.txt oracle2.txt oracle3.txt\n I ...
9
votes
9answers
21k views

How to print only last column?

echo -e 'one two three\nfour five six\nseven eight nine' one two three four five six seven eight nine how can I do some "MAGIC" do get this output?: three six nine UPDATE: I don't need it in this ...
0
votes
3answers
60 views

awk or sed or perl: remove only characters on specific position

I have multiple files to rename leng-1494-001 leng-1464-002 leng-2414-004 leng-7894-005 leng-1323-006 I want to rename it ferr-1494-001 ferr-1464-002 ferr-2414-004 ...
-1
votes
1answer
44 views

Webpage content - search for pattern & save to file

I have a web page, say https://test.salesforce.com. I would like to search for a pattern, e.g. "src" at the beginning of a line in the entire webpage content. If the pattern is found, I need to ...
0
votes
2answers
77 views

Print text between two patterns not containing a particular word

I wanted to print some text between two patterns which doesn't contain a particular word input text is HEADER asdf asd asd COW assd TAIL sdfsdfs HEADER asdf asd sdfsd DOG sdfsdfsdf TAIL sdfsdfs ...
4
votes
3answers
1k views

awk + print lines from the first line until match word

I want to print all lines from file until the match word please advice how to do that with awk for example I want to print all lines until word PPP remark the first line chuld be diff from AAA ( ...
2
votes
2answers
38 views

Copy files with parent folder to a new directory

I want to copy some files and include their parent directory in the process. The current structure is similar to: /Module1/A/downloads/m13312/file.pdf /Module1/A/downloads/m17312/file.pdf ...
5
votes
2answers
174 views

Transpose a file, and replace missing values

I have Readings for Names, from Machines and sometimes these readings are replicated. Where a reading was not found, it is left as blank. Name Instrument Rep R1 R2 R3 N1 I1 1 1 2 3 N2 I1 1 1 3 4 ...
6
votes
2answers
3k views

awk: forcing a return status?

This is a followup to my earlier question. I am validating the number of fields in /etc/passwd using this handy snippit. In the following example, the users 'fieldcount1' and 'fieldcount2' have the ...
2
votes
1answer
31 views

Adding consecutive characters to duplicate words according to order they appear

So I have files like this but for hundreds of different genes per file. >lcl|accession_cds_proteinaccession [gene=dnaA] [protein=description] [protein_id=accession] [location=location] ...
1
vote
2answers
55 views

Use awk to get two specific columns from third line of file based on value in first line

I'm looking to conditionally select two columns from the third line of a file based on the first column of the first line of the file. Here is the file format: v1 shortdesc value1 value2 value3 ...
1
vote
2answers
58 views

Compare three files: lines present in one, but not in the other using column comparison

I have three files file1.txt, file2.txt, file3.txt and they are of same format(same number of columns with same data type). I want to select records from file1.txt which are not present in file2.txt, ...
0
votes
2answers
35 views

Variable in for loop to be used as an array for a csv output

I've been struggling to come up with a solution for this, the goal is to organize the results into comma separated columns as we have to recable our entire datacenter from cat6 to cat6a. $DBFILE ...
0
votes
3answers
39 views

Awk formatting issue for XML

There is an issue with the output XML format due to Record separator (RS = "\n \n"). The group of data is separated by an empty line. Any suggestion would be very much helpful to obtain the desired ...
1
vote
3answers
151 views

Find records present in file 1 also in both file 2 and file 3

I have three files file1.txt, file2.txt, file3.txt and they are of same format. I want to select records from file1.txt which are present in both file2.txt and file3.txt based on column 2 and column ...
1
vote
1answer
66 views

sort file based one colum and group by another column

I have a file of three columns: 1 A 0.5 2 B 0.7 3 A 10 4 C 4 5 B 4 I want to sort the file by increasing order of column 3 and group by column 2 1 A 0.5 3 A 10 ...
3
votes
3answers
269 views

Extracting unique piece of text from file with awk, sed or grep

From the output of pactl list sink-inputs, I need to grab the sink input number for VLC. Before that, I'm trying to extract the piece that contains the output for only VLC. All the methods that I ...
0
votes
0answers
46 views

Output of wc | awk is different in shell script than from cli

When issued via command line, this returns "2": pgrep someprocess | wc | awk '{print $1}' When run in a script like this, it returns "4" #!/bin/bash SOME_PROCESS=`pgrep someprocess | wc | awk ...
2
votes
4answers
171 views

Column editing using awk or sed. how to print the following pattern?

Sample Input: title role subject name-JOHN student math school state NY county street Phone name-TOM student math school ...
1
vote
1answer
30 views

AWK : the ' ' as field separator

I have those lines from db dump INSERT INTO db1 VALUES ('2', '33', '3', '1975', '3', '5', '1'); INSERT INTO db1 VALUES ('4', '23544', '1', '1987', '3', '1', '1'); INSERT INTO db1 VALUES ('2', '343', ...
0
votes
1answer
24 views

awk change one column with another

I want to substitute column 2 os a db file like this INSERT INTO db1 VALUES ('2', '33', '3', '1975', '3', '5', '1'); INSERT INTO db1 VALUES ('4', '23544', '1', '1987', '3', '1', '1'); INSERT INTO db1 ...
-2
votes
1answer
74 views

How to properly use awk in ksh?

I have: 1 #!/bin/ksh 2 BEGIN { print "START" } 3 { print "Hello World" } 4 END { print "STOP" } 5 6 7 exit 0 then when I try to run the file the following errors appears: ...
0
votes
2answers
44 views

.csv Keep only certain email addresses?

I have a large .csv file with my customers Name:Email I have another large list of emails that are part of this list and only want to keep those. How can I remove all contacts from the .csv file that ...
1
vote
3answers
47 views

How to use sed or awk to obtain just a directory

3:ââsdb1 8:17 1 984M 0 part /media/(NAMEOFDRIVEHERE) I want to just have the /media/(NAMEOFDRIVEHERE) location so I can store it in a variable. I've tried some sed commands and have ...
0
votes
1answer
65 views

Cutting logs (and not copying) from a log file between two time stamps

I have a very huge access.log file of 2 Gb containing apache access logs. The date time format in that is as below : 04/May/2015:02:15:25 I want to cut (and not just copy) the logs from this ...
0
votes
2answers
34 views

awk replace specific column in specific line

My data file is sdfsdfsdf 1 0.5000 0.5000 0.5000 8 7 90 135 90 135 1 2 0.5000 0.5000 0.0000 8 7 90 -45 90 -45 1 3 0.0000 0.5000 0.5000 8 7 90 225 90 225 1 4 0.5000 0.0000 0.5000 8 7 90 45 90 ...
1
vote
2answers
49 views

Add string to list using either AWK or SED?

Hi I want to be able to edit the list of name like this using either AWK or SED. Sample input list file: john paul rose lily Desired output: I am john of earth; I am paul of earth; I am rose of ...
3
votes
1answer
56 views

Counting number of entries in column 2 where column 1 is similar

I am trying to count the number of entries in column 2 for the same column1 and also number of times the first entry in column2 for similar column1 occur. For example my input TSV file is: C1 NC01 ...
8
votes
5answers
634 views

Remove duplicate lines from a file that contains a timestamp

This question/answer has some good solutions for deleting identical lines in a file, but won't work in my case since the otherwise duplicate lines have a timestamp. Is it possible to tell awk to ...
1
vote
1answer
37 views

Conditional awk across 4 columns

I have a tab-delimited file with 9 columns as follows: 1 25555818 T 14 0 0 0 14 0:0:0:14 2 25555819 T 14 7 0 0 7 7:0:0:7 3 25555820 G 13 0 0 13 0 ...
0
votes
2answers
54 views

Awk Using NR to output iteration number

I am looping through a list of files, extracting the final line, and printing out columns 8, 9, and 10. I need to also print to the output the 'event number', which is essentially the total number of ...
4
votes
2answers
76 views

Awk inplace option with print statement

In recent version of awk there is a inplace option that can be used to do in place operations similar to sed's -i option. However I can't make it work with print statement. Let's see my example. The ...