awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

7
votes
8answers
159 views

Split text file into lines with fixed number of words

Related, but no satisfactory answers: How can I split a large text file into chunks of 500 words or so? I'm trying to take a text file (http://mattmahoney.net/dc/text8.zip) with > 10^7 words all in ...
2
votes
3answers
47 views

How to add “wrappers” around methods in source code based on a pattern, using sed, awk, grep and friends

How can I add wrappers to a file based on a pattern? For instance I have the following: ... find(css_policy_form_stage3).click with_ajax_wait expect(css_coverage_type_everquote).to ...
1
vote
2answers
36 views

With awk, how to Insert a number only for lines starting with “Toto” when they are between a specific couple of patterns

With awk, I would like to insert a numbering like Record n°i# in front of each line starting with Toto only when they are located in-between two specific patterns start=ABCD and stop=EFGH. Input file ...
2
votes
8answers
3k views

How to print lines with blank 5th field in CSV

I have to check if the fifth field is empty in a CSV file. This is my file: 1,abc,543,87,DATA,fsg; 1,abc,543,87,,fsg; 1,abc,543,87,DATA,fsg; 1,abc,543,88,,fsg; 1,abc,543,,DATA,fsg; As you can ...
5
votes
8answers
341 views

awk or sed command to match regex at specific line, exit true if success, false otherwise

I need to determine if a file contains a certain regex at a certain line and to return true (exit 0) if found, and otherwise false. Maybe I'm overthinking this, but my attempts proved a tad unwieldy. ...
0
votes
3answers
23 views

extract url between 2 strings in a file

I have a file in which each line is like this "372"^""^"2015-09-03 06:59:44.475"^"NEW"^"N/A"^""^0^"105592"^"https://example-url.com"^"example-domain < MEN'S ULTRA < UltraSeriesViewAll ...
0
votes
2answers
19 views

Replace values on the basis of matching ID variable

I have two files. file1 has 6 columns, file2 has 2 columns. I want to replace the values in the 6th column of file1 by the values in column 2 of file2, based on matching ID variables (column 1 in both ...
0
votes
2answers
15 views

Find line within file matching pattern and make a substitution

I'm trying to simplify my life here but I'm stuck. I have a file containing multiple lines. Each line has a user and a number. I'd like to know a bash one-liner to search for a user and change the ...
1
vote
3answers
42 views

Replacing a “time” column with corresponding values

I have a file in the following format: "2004-04-19 12:25:57" 44 44 "2004-04-19 13:39:32" 36 36 "2004-04-19 14:00:53" 34 34 I need 2 new files: a) A file which will replace "time" values of the ...
6
votes
6answers
3k views

Convert string into date time stamp in gawk or awk

I want to convert following string (20140805234656) into date time stamp (2014-08-05 23:46:56).I am new to gawk and I don't know the exact syntax,how can I put - at every 5,8 and : at every 14,17 and ...
105
votes
3answers
225k views

How can I replace a string in a file(s)?

Replacing strings in files based on certain search criteria is a very common task. How can I replace string foo with bar in all files in the current directory? do the same recursively for sub ...
-1
votes
3answers
38 views

How to delete certain line from file?

I have a text file with line containing only comma. How can I delete this particular line? Consider following example: CREATE TABLE ENTERPRISE.TSDAILYACCTSUMMARY ( TSDAILYACCTID FOR COLUMN ...
-1
votes
1answer
48 views

How to add double quotes to a various strings in a same pattern in a file

I have a file like this CREATE TABLE ENTDTA.$$BAPWD ( PERSID NUMBER(10) DEFAULT 0 NOT NULL , "PASSWORD" CHAR(10) DEFAULT '' NOT NULL , UPDATE_IDENT NUMBER(7, 0) DEFAULT 0 NOT NULL ) ...
0
votes
1answer
20 views

Replace the first word of each line with its MD5

I have file with two space separated word in each line. I am looking for shell scrip in Unix to replace the first word of each row with its md5 hash representation. I have tried to adjust what is ...
1
vote
1answer
23 views

OSX: Delete all matching certificates by command-line

Using shell script to delete all matching certificates. I found a list of certificates using: security find-certificate -c "certificatename" -a -Z|grep SHA-1|awk '{ print $NF }' Output is like ...
-8
votes
0answers
23 views

Need to replace random words with “X” from 13th filed till a “|” is encountered [on hold]

Sample Input Aug 27 07:23:46 ABIT-AB545-Default [ABIT-PMC][0x80000341][xsltmsg][info] wsgw(abc_v2): trans(123456789)[request][10.99.99.4]: RespOperation Name: ABCV2 | RespConsumerRequest: ABC12345C3 ...
1
vote
1answer
27 views

How can I use awk to create a variable that is used in the next line edit

I have a file with a list of info. Each line has either 12 or 13 fields. For example I will use a file will 3 and 4 fields. e.g. myfile.txt user password location date john passwd1 new_york today ...
1
vote
1answer
36 views

Merge lines with matching first column

I've been trying to use awk to merge two files when they have the same first column. Here are my example files: FileA.txt A2M 1 A4GALT 11 AAAS 35 AAGAB 7 FileB.txt A4GALT 2 AAAS ...
4
votes
1answer
56 views

How to use awk to indent a source file based on simple rules?

How can I indent source code based on a couple of simple rules? As an example, I've used sed and ask to transform a selenium HTML source table to the following rspec like code. How could I ...
3
votes
2answers
53 views

Why awk says “syntax error” for the comma I placed between the two patterns?

I run the following script under Linux (Mint on a USB key) with GNU Awk 4.0.1 installed by default. Error returned by awk is the following : awk: cmd. line:7: /^Observation/,/^@@@/ ...
-1
votes
1answer
22 views

Look for a particular pattern and do a swap on that line

I have a set of files containing VIEW_NAME FOR COLUMN DNAME , VIEW_OWNER FOR COLUMN DCREATOR , OBJECT_NAME FOR COLUMN ONAME , OBJECT_SCHEMA FOR COLUMN OSCHEMA So, ...
0
votes
1answer
21 views

replacing first link of a text “awk equivalent of sed command”

I wanted to replace the first line of a text with sed it should be like sed -i.bak "1 s/^.*$/"CompoundState\tMethod\tApproach\tS^2\tEnergy\tPath"/" awk.xls but I also need to run it on OSX but it ...
5
votes
4answers
5k views

How to reduce the greed of a regular expression in AWK?

I have an example: echo "@article{gjn2010jucs, Author = {Grzegorz J. Nalepa}, " | awk '{ sub(/@.*,/,""); print }' Is it possible to write a regular expression that selects the shorter pattern? ...
0
votes
1answer
21 views

sum over floating points of the form 3.016e-06 using awk

I want to sum over floats of the form: 3.016e-06 The floats are saved in a files called S_3x3.txt S_4x4.txt ... S_8x8.txt My code sums only the first integer. Instead of 3.016e-06 + 4.022e-06 it ...
2
votes
2answers
48 views

adding column in CSV file using awk

I need to a add a column in CSV file from an array using awk. For example, input.csv a,10 b,11 array = (100 200) output.csv should look like a,10,100 b,11,200 I tried this command but it does ...
13
votes
2answers
222 views

Fastest way to delete duplicates in large wordlist?

I need to deduplicate a large wordlist. I tried several commands and did some research here and here where they explain that the fastest way to deduplicate a wordlist seems to be using awk. awk ...
1
vote
0answers
43 views

How to replace words in a string which has “|” in it [closed]

For example line, in this loop in a shell script: for ((k=13;`awk -v f="$k" -F " " '{print $f}' line$i.txt` != "|";k++)) I want to replace the words in the line from 13th position until a "|" is ...
1
vote
5answers
52 views

Making a *nicely* formatted list of number of files in directories - awk & sed probably

I have lots and lots of files in a series of directories, and would like a nicely formatted list of how many files are in each directory. The directories are ordered thus: Parent directories named ...
1
vote
2answers
2k views

awk - get input from both file and STDIN?

Is it possible to get input from both file and the terminal? I wish to ask the user something in the END portion of the awk script. That input should be typed in by the user and not read from the ...
6
votes
4answers
199 views

How to add a header and/or footer to a sed or awk stream?

I have a bunch of output going through sed and awk. How can I prefix the output with START and suffix the answer with END? For example, if I have All this code on all these lines and all these ...
0
votes
1answer
37 views

Print concat vars inside loop

I have a code like this: cat input | while read p; do lynx -dump "www.xyz.com/p?=$p" | grep -m 1 -A 1 "Hello" | awk -F: '{print $2}' >> output done Where the content looks like this: [$p ...
1
vote
5answers
113 views

lines to columns with awk

I have to following sample output: <HARDWARE> <NAME>WIN1</NAME> <OS>Windows 7</OS> <IP>1.2.3.4</IP> <DOMAIN>contoso.com</DOMAIN> ...
5
votes
2answers
620 views

Fastest `uniq` tool in linux

I have large text file (1.5 G), I want to know what the fastest and more reliable tool in linux is. I usualy use: awk '!x[$0]++' file.txt But when I use htop command I see my memory usage is ...
2
votes
3answers
93 views

FNR and NR variables problem

I have two files, one of them has 10 records, and the other one with 15 record, the problem is when I cat them together, always FNR=NR. For example, consider below files: File1: 1,boo 2,foo 3,boo ...
0
votes
0answers
21 views

Sort Checkpoint Firewall log into easier output

I am trying to sort out this log: Aug 27 01:23:38 1.1.1.1/1.1.1.1/1.1.1.1 fw1-loggrabber[0000]: loc=12569|time=2015-08-27 ...
0
votes
1answer
31 views

Big XML - copy data from nth Occurrences to another file

<XML> <Employee> <firstname></firstname><lastname></lastname> </Employee> <Employee> ...
15
votes
11answers
47k views

Show sum of file sizes in directory listing

The Windows dir directory listing command has a line at the end showing the total amount of space taken up by the files listed. For example, dir *.exe shows all the .exe files in the current ...
3
votes
4answers
222 views

Conditional if clause in awk

Please consider below file: foo,1000,boo,A foo,1000,boo,B foo,1001,boo,B foo,1002,boo,D And we have below rules: If $2 equal 1000, $4 should be equal A If $2 equal 1001, $4 should be equal B If ...
0
votes
1answer
37 views

How to add new column, where the value is based on existing columns with awk

I have a file of multiple columns. I would like to make an additional column based on the values of 2 columns from this file. Example input: A B C D E F 1 2 T TACA A 3 2 3 4 I R 8 2 9 3 A C 9 3 ...
3
votes
3answers
48 views

Find a pattern and insert # at the beginning of 2 line befoe that and 1 line after that

I have a big file (more than 2000 lines). Where i have to insert a # at the beginning of 2 above lines and at the beginning of 1 below line after finding a pattern. Also insert # at the beginning of ...
2
votes
3answers
42 views

Assigning “awk” or “nawk” to a variable in a shell script [closed]

I have a script to be run on both Solaris and Red Hat systems. I want to use "nawk" in Solaris and "awk" in Red Hat systems as text processing tool. For that, I have a code snippet as below in my ...
0
votes
1answer
61 views

SED - Find and replace with special characters (#, , % )

I'm aware that this is a really simple question, but I'm struggling to find a solution to this . I need to automatically find and replace in the /etc/aliases file the following section: # Person ...
4
votes
4answers
134 views

Current date in awk

Is there a constant variable in awk, that store today's date? if no, is there a way that can store today's date for daily use? let's say we have below file: boo,foo,2016-08-30 foo,boo,2016-07-31 ...
0
votes
5answers
68 views

Appending One File to the Second Until End of File

I have two files. The first is just a single number. This number is calculated and changes every time. The second file has a bunch of columns, and I do not know the number of rows (which can change). ...
1
vote
1answer
49 views

How to print the line numbers with corresponding line that matches a pattern using shell command? [duplicate]

File, .false alarm is bla no. 11 .no alarm generated is bla .application will be killed is bla wall command .doc file is document file with .doc extension .no authority for this selection bla Now ...
1
vote
1answer
39 views

Identifying duplicate fields and print both with awk

I have a file with multiple columns and want to identify those where specific column values (cols 3-6) have been duplicated. The following code finds the duplicates but I want to display both ...
0
votes
2answers
25 views

awk edit column value if numeric

What I would like to do is search for lines where the first column does not begin with 'rs' or 'chr' THEN if those lines begin with a number, append 'chr' to the first column value, otherwise leave as ...
0
votes
2answers
40 views

awk if-then-else statement error

I am writing an if-then-else statement using awk in a bash script. What I would like to do is identify lines with col 1 values not matching a particular string (rs or chr) and append a prefix (chr) ...
0
votes
1answer
33 views

Repeatedly extract every second and third line of a file with 3-line data blocks

I have a file where I have data in 3-line sets. I want the 2nd and 3rd line from each set. How do I get that? For an example from: [set no][no of line in set] 11 12 13 21 22 23 31 32 33 I want: ...
4
votes
2answers
2k views

grep one line before the match plus the match

zzzzzzzzz aaaaaaaaa bbbbbbbbb & ccccccccc & ddddddddd hhhhhhhhh eeeeeeeee fffffffff & ggggggggg & in the above line, what i want is to grep/sed/awk (any method is fine) line that ...