awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

7
votes
8answers
128 views

Split text file into lines with fixed number of words

Related, but no satisfactory answers: How can I split a large text file into chunks of 500 words or so? I'm trying to take a text file (http://mattmahoney.net/dc/text8.zip) with > 10^7 words all in ...
1
vote
2answers
35 views

With awk, how to Insert a number only for lines starting with “Toto” when they are between a specific couple of patterns

With awk, I would like to insert a numbering like Record n°i# in front of each line starting with Toto only when they are located in-between two specific patterns start=ABCD and stop=EFGH. Input file ...
0
votes
3answers
22 views

extract url between 2 strings in a file

I have a file in which each line is like this "372"^""^"2015-09-03 06:59:44.475"^"NEW"^"N/A"^""^0^"105592"^"https://example-url.com"^"example-domain < MEN'S ULTRA < UltraSeriesViewAll ...
2
votes
3answers
45 views

How to add “wrappers” around methods in source code based on a pattern, using sed, awk, grep and friends

How can I add wrappers to a file based on a pattern? For instance I have the following: ... find(css_policy_form_stage3).click with_ajax_wait expect(css_coverage_type_everquote).to ...
0
votes
2answers
19 views

Replace values on the basis of matching ID variable

I have two files. file1 has 6 columns, file2 has 2 columns. I want to replace the values in the 6th column of file1 by the values in column 2 of file2, based on matching ID variables (column 1 in both ...
1
vote
3answers
42 views

Replacing a “time” column with corresponding values

I have a file in the following format: "2004-04-19 12:25:57" 44 44 "2004-04-19 13:39:32" 36 36 "2004-04-19 14:00:53" 34 34 I need 2 new files: a) A file which will replace "time" values of the ...
5
votes
8answers
341 views

awk or sed command to match regex at specific line, exit true if success, false otherwise

I need to determine if a file contains a certain regex at a certain line and to return true (exit 0) if found, and otherwise false. Maybe I'm overthinking this, but my attempts proved a tad unwieldy. ...
0
votes
1answer
20 views

Replace the first word of each line with its MD5

I have file with two space separated word in each line. I am looking for shell scrip in Unix to replace the first word of each row with its md5 hash representation. I have tried to adjust what is ...
-1
votes
3answers
37 views

How to delete certain line from file?

I have a text file with line containing only comma. How can I delete this particular line? Consider following example: CREATE TABLE ENTERPRISE.TSDAILYACCTSUMMARY ( TSDAILYACCTID FOR COLUMN ...
-1
votes
1answer
48 views

How to add double quotes to a various strings in a same pattern in a file

I have a file like this CREATE TABLE ENTDTA.$$BAPWD ( PERSID NUMBER(10) DEFAULT 0 NOT NULL , "PASSWORD" CHAR(10) DEFAULT '' NOT NULL , UPDATE_IDENT NUMBER(7, 0) DEFAULT 0 NOT NULL ) ...
1
vote
1answer
22 views

OSX: Delete all matching certificates by command-line

Using shell script to delete all matching certificates. I found a list of certificates using: security find-certificate -c "certificatename" -a -Z|grep SHA-1|awk '{ print $NF }' Output is like ...
1
vote
1answer
27 views

How can I use awk to create a variable that is used in the next line edit

I have a file with a list of info. Each line has either 12 or 13 fields. For example I will use a file will 3 and 4 fields. e.g. myfile.txt user password location date john passwd1 new_york today ...
-8
votes
0answers
22 views

Need to replace random words with “X” from 13th filed till a “|” is encountered [on hold]

Sample Input Aug 27 07:23:46 ABIT-AB545-Default [ABIT-PMC][0x80000341][xsltmsg][info] wsgw(abc_v2): trans(123456789)[request][10.99.99.4]: RespOperation Name: ABCV2 | RespConsumerRequest: ABC12345C3 ...
1
vote
1answer
36 views

Merge lines with matching first column

I've been trying to use awk to merge two files when they have the same first column. Here are my example files: FileA.txt A2M 1 A4GALT 11 AAAS 35 AAGAB 7 FileB.txt A4GALT 2 AAAS ...
3
votes
2answers
53 views

Why awk says “syntax error” for the comma I placed between the two patterns?

I run the following script under Linux (Mint on a USB key) with GNU Awk 4.0.1 installed by default. Error returned by awk is the following : awk: cmd. line:7: /^Observation/,/^@@@/ ...
-1
votes
1answer
22 views

Look for a particular pattern and do a swap on that line

I have a set of files containing VIEW_NAME FOR COLUMN DNAME , VIEW_OWNER FOR COLUMN DCREATOR , OBJECT_NAME FOR COLUMN ONAME , OBJECT_SCHEMA FOR COLUMN OSCHEMA So, ...
0
votes
1answer
21 views

replacing first link of a text “awk equivalent of sed command”

I wanted to replace the first line of a text with sed it should be like sed -i.bak "1 s/^.*$/"CompoundState\tMethod\tApproach\tS^2\tEnergy\tPath"/" awk.xls but I also need to run it on OSX but it ...
0
votes
1answer
21 views

sum over floating points of the form 3.016e-06 using awk

I want to sum over floats of the form: 3.016e-06 The floats are saved in a files called S_3x3.txt S_4x4.txt ... S_8x8.txt My code sums only the first integer. Instead of 3.016e-06 + 4.022e-06 it ...
1
vote
0answers
43 views

How to replace words in a string which has “|” in it [closed]

For example line, in this loop in a shell script: for ((k=13;`awk -v f="$k" -F " " '{print $f}' line$i.txt` != "|";k++)) I want to replace the words in the line from 13th position until a "|" is ...
6
votes
4answers
199 views

How to add a header and/or footer to a sed or awk stream?

I have a bunch of output going through sed and awk. How can I prefix the output with START and suffix the answer with END? For example, if I have All this code on all these lines and all these ...
4
votes
1answer
56 views

How to use awk to indent a source file based on simple rules?

How can I indent source code based on a couple of simple rules? As an example, I've used sed and ask to transform a selenium HTML source table to the following rspec like code. How could I ...
0
votes
1answer
37 views

Print concat vars inside loop

I have a code like this: cat input | while read p; do lynx -dump "www.xyz.com/p?=$p" | grep -m 1 -A 1 "Hello" | awk -F: '{print $2}' >> output done Where the content looks like this: [$p ...
1
vote
5answers
52 views

Making a *nicely* formatted list of number of files in directories - awk & sed probably

I have lots and lots of files in a series of directories, and would like a nicely formatted list of how many files are in each directory. The directories are ordered thus: Parent directories named ...
2
votes
2answers
48 views

adding column in CSV file using awk

I need to a add a column in CSV file from an array using awk. For example, input.csv a,10 b,11 array = (100 200) output.csv should look like a,10,100 b,11,200 I tried this command but it does ...
13
votes
2answers
222 views

Fastest way to delete duplicates in large wordlist?

I need to deduplicate a large wordlist. I tried several commands and did some research here and here where they explain that the fastest way to deduplicate a wordlist seems to be using awk. awk ...
2
votes
3answers
93 views

FNR and NR variables problem

I have two files, one of them has 10 records, and the other one with 15 record, the problem is when I cat them together, always FNR=NR. For example, consider below files: File1: 1,boo 2,foo 3,boo ...
0
votes
0answers
21 views

Sort Checkpoint Firewall log into easier output

I am trying to sort out this log: Aug 27 01:23:38 1.1.1.1/1.1.1.1/1.1.1.1 fw1-loggrabber[0000]: loc=12569|time=2015-08-27 ...
0
votes
1answer
31 views

Big XML - copy data from nth Occurrences to another file

<XML> <Employee> <firstname></firstname><lastname></lastname> </Employee> <Employee> ...
1
vote
5answers
113 views

lines to columns with awk

I have to following sample output: <HARDWARE> <NAME>WIN1</NAME> <OS>Windows 7</OS> <IP>1.2.3.4</IP> <DOMAIN>contoso.com</DOMAIN> ...
3
votes
4answers
222 views

Conditional if clause in awk

Please consider below file: foo,1000,boo,A foo,1000,boo,B foo,1001,boo,B foo,1002,boo,D And we have below rules: If $2 equal 1000, $4 should be equal A If $2 equal 1001, $4 should be equal B If ...
0
votes
1answer
37 views

How to add new column, where the value is based on existing columns with awk

I have a file of multiple columns. I would like to make an additional column based on the values of 2 columns from this file. Example input: A B C D E F 1 2 T TACA A 3 2 3 4 I R 8 2 9 3 A C 9 3 ...
2
votes
3answers
42 views

Assigning “awk” or “nawk” to a variable in a shell script [closed]

I have a script to be run on both Solaris and Red Hat systems. I want to use "nawk" in Solaris and "awk" in Red Hat systems as text processing tool. For that, I have a code snippet as below in my ...
0
votes
1answer
61 views

SED - Find and replace with special characters (#, , % )

I'm aware that this is a really simple question, but I'm struggling to find a solution to this . I need to automatically find and replace in the /etc/aliases file the following section: # Person ...
3
votes
3answers
48 views

Find a pattern and insert # at the beginning of 2 line befoe that and 1 line after that

I have a big file (more than 2000 lines). Where i have to insert a # at the beginning of 2 above lines and at the beginning of 1 below line after finding a pattern. Also insert # at the beginning of ...
1
vote
1answer
49 views

How to print the line numbers with corresponding line that matches a pattern using shell command? [duplicate]

File, .false alarm is bla no. 11 .no alarm generated is bla .application will be killed is bla wall command .doc file is document file with .doc extension .no authority for this selection bla Now ...
1
vote
1answer
39 views

Identifying duplicate fields and print both with awk

I have a file with multiple columns and want to identify those where specific column values (cols 3-6) have been duplicated. The following code finds the duplicates but I want to display both ...
0
votes
5answers
68 views

Appending One File to the Second Until End of File

I have two files. The first is just a single number. This number is calculated and changes every time. The second file has a bunch of columns, and I do not know the number of rows (which can change). ...
0
votes
2answers
25 views

awk edit column value if numeric

What I would like to do is search for lines where the first column does not begin with 'rs' or 'chr' THEN if those lines begin with a number, append 'chr' to the first column value, otherwise leave as ...
0
votes
2answers
40 views

awk if-then-else statement error

I am writing an if-then-else statement using awk in a bash script. What I would like to do is identify lines with col 1 values not matching a particular string (rs or chr) and append a prefix (chr) ...
0
votes
1answer
33 views

Repeatedly extract every second and third line of a file with 3-line data blocks

I have a file where I have data in 3-line sets. I want the 2nd and 3rd line from each set. How do I get that? For an example from: [set no][no of line in set] 11 12 13 21 22 23 31 32 33 I want: ...
0
votes
2answers
14 views

Find line within file matching pattern and make a substitution

I'm trying to simplify my life here but I'm stuck. I have a file containing multiple lines. Each line has a user and a number. I'd like to know a bash one-liner to search for a user and change the ...
4
votes
4answers
134 views

Current date in awk

Is there a constant variable in awk, that store today's date? if no, is there a way that can store today's date for daily use? let's say we have below file: boo,foo,2016-08-30 foo,boo,2016-07-31 ...
3
votes
1answer
27 views

Executing awk on each file found with find, then redirecting the result to a new filename

I am trying to take the results from find and then execute awk to print the first two columns. Then I want to redirect those results to a new file name for each file, using the existing file name with ...
1
vote
0answers
51 views

Delete the data based on the output of awk command [duplicate]

I have a file test.dat with delimited text. Delimiter is | containing 4 columns in a file. But due to some issue, there is some discrepancy in the number of columns for some of the rows. I have found ...
1
vote
2answers
24 views

Script that only counts Established, TIME_WAIT, and Closed Wait connections using Netstat

I'm trying to make a script (using Perl, but it isn't necessary) that will only count the number of Established, Time_Wait, and Closed_Wait connections on a system and print them in terminal. So far ...
-2
votes
2answers
98 views

Grouping input using awk,sed,grep

I have an input file as below: 1001 Ivanov I.I: chess 2021 Petrov P.P: chess, football 3352 Sidorov S.S: 1000 Putin V.V: judo 8773 Schwarzenegger A: judo, chess, football I expect ...
0
votes
0answers
30 views

Convert vnstat log file to csv

My goal is to log daily network activity (rx/tx) from each ethernet interface. I installed vnstat, and while it reports daily and monthly stats, it doesn't offer a nice way to convert to csv to ...
1
vote
2answers
79 views

grep first column uniq values

Two columns A and B: Column A have repeated ids and column B has different values for each corresponding repeated value. I would like to read every uniq value from Column A with only the first hit ...
1
vote
2answers
49 views

Awk & regular expression to process pgn chess games file

Below an extract of the sample.pgn file [Event "rated blitz match"] [Site "Internet Chess Server"] [Date "2015.08.20"] [Round "?"] [White "Villain"] [Black "Hero"] [Result "0-1"] [WhiteElo "1643"] ...
0
votes
1answer
42 views

Read all the values of a particular column of a file and store it in another file

I have a file.txt which contains data like below: col11 col12 col13 col21 col22 col23 I fetch 2nd column of all the available rows (no. of rows may vary) using awk '{print $2}' file.txt Now i ...