awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

3
votes
4answers
380 views

how to change one column and add other columns in a row by awk

I have a file MDD.TXT with following records. file : MDD.TXT 0|ABC|055309|20120929053309|11965150 0|ABC|061434|20120929063434|11155537 0|ABC|070228|20120929073228|11965150 ...
0
votes
1answer
38 views

Please explain this awk statement [duplicate]

Yesterday I was googling how to merge two files and came across an awk snippet. I need a simple merge, so sort -u is not the way to go, but the code below works. Could some one please explain what ...
0
votes
3answers
44 views

How to get filename in awk

I have a zip file with below name and record: Filename : OMS_Profile_20150922.list.Z Sample Record: 1234568194|301776931|301344831|301745924|4000247920|5|1|14| ...
6
votes
4answers
298 views

Count number of a substring repetition in a string

I have a file which contains a gene sequence such as: ...
3
votes
2answers
232 views

split a large file to new file with unique file names

I appreciate your help if some one could help me with a script that could split a file into unique file names. I can do it with sed command eg, sed -n '/scaffold135_/w 135-scaf.txt' input file.txt ...
3
votes
7answers
110 views

How to replace line with file contents and around the content with some text?

I would like to combine VPN certificates into a single file. This is a template file: # some unrelated stuff ca ca.crt # other unrelated stuff And I want to replace line ca ca.crt with the ...
0
votes
2answers
56 views

Problem with data comparison in if loop

I have 2 variables in bash. Using test i found out that one of them is integer and other one is string. I want to compare them and see if they are equal. I tried the following options. if [ "$a" -eq ...
6
votes
4answers
257 views

Delete column that includes a specific text

I was looking for an option to delete a column with a specific text like "ddd" aaa bbb ccc ddd eee fff 1 2 3 4 5 6 2 3 4 5 6 0 So that the output looks likes this: aaa bbb ccc ...
3
votes
2answers
63 views

How to format a file while moving?

I have 2 directories. SOURCE and DESTINATION. I am moving the .csv files from source to destination as mv -f SOURCE/*.csv DESTINATION/ While moving, I want to remove the first and last line from ...
2
votes
1answer
41 views

How to change trailing numbers to incrementing number

I'm using ArchLinux 4.1.6-1 and bash 4.3. I need to tidy up a pile of programs in 200+ industrial robots and so trying to automate the leg work. I can strip a file down to a just a list of ...
1
vote
1answer
27 views

Modify string if original does not match particular string

I have a file with millions of rows and want to modify values in column A only if the string is '.'. The modification would be to add "chr:"$2":"$3 to the start of the string. All other rows would be ...
0
votes
1answer
78 views

Wanted: nonassociative array in AWK

I am looking at speeding up some code in AWK by serially processing an internal string representation instead of serially processing an array representation. (For Examples 1 and 2 below assume ...
4
votes
2answers
144 views

Keeping first instance of duplicates

I have a file with multiple columns and have identified lines where specific column values (cols 3-6) have been duplicated using a bash script. Example input: A B C D E F G 1 2 T TACA A 3 2 Q 3 4 I ...
2
votes
1answer
141 views

replace nth column of a csv file by the help of lookup table

I have a csv file (a.csv) as below (with thousands of lines): NMS_NE,,,,,,,,,,,,,,,,,,,,,,, D,NE Name,IP Address,Site Name,Product UPEI,Product Name,Product Model,NE Status,Serial Number,SW Major ...
-1
votes
1answer
33 views

Get the respective tablenames if it matches particular pattern

I need to list the table names if it finds particular pattern from the below file. CREATE TABLE IMS.ACTIVITY ( "ID" INTEGER NOT NULL , OWNING_TYPE FOR COLUMN OWNIN00001 VARCHAR(20) ...
0
votes
1answer
58 views

Split a set of data to massives and find maximum for each

I have a set of data looking like this: 1 0.1 2 0.23 3 0.05 5 0.73 6 0.40 7 0.34 10 0.11 11 0.45 12 0.68 I need to split the set of data to massives using the rule that the next ...
1
vote
3answers
136 views

how to extract a value from a string in xml using grep or awk in Linux

I have a file containing: DEVICE="7" PGMNR="24" VCONF="800" In this file, I want to find PGMNR="24" and I need the value 24 without the ". The PGMNR="24" may be located anywhere in the line. The ...
2
votes
3answers
85 views

How to substite a " in awk?

For example, I need to change a " into the word quote in to change a string with a " at some point into a string with a quote at some point I have tried: $ echo 'a string with a " at some ...
1
vote
2answers
68 views

use an export variable as part of filename in awk program

I have set an export X25DATAF=/home/Bob/X25/Datafiles in my profile and that's what is outputted when I do a echo $X25DATAF. I want to replace /home/Bob/X25/Datafiles in the awk below with the ...
-1
votes
1answer
42 views

Replace a string with “string|string”

I have a pretty weird problem. I did some calculations for bioinformatics, but my downstream apps won't accept my header information in its current format. In order to circumvent this I modified my ...
0
votes
1answer
28 views

create a structured table

I have a file which is as follows Tran No Date Vchr No Debit Credit 1239 05/06/2015 115 750.00 ...
1
vote
2answers
33 views

Reading a variable-sized line and splitting its content into separate lines with text

I have a file with the header (first line) with a variable length; and it comes as: "One"|"Two"|"Three" or "One"|"Two"|"Three"|...|"One Hundred" and want to get do a loop that transforms it ...
0
votes
1answer
47 views

Fetch second and fourth column from a file

Problem Statement I am not able to get the values in second and fourth column. Tried grep -i "AUGC\|AACN\|AACY" output.csv | cut -d ',' 2,4 DATA ...
0
votes
1answer
45 views

How to remove the first instance of something in a file with awk?

If I have a file with multiple lines with ... end ... end ... end ... How could I remove only the first instance? So far I have tried awk 'BEGIN {found=false} / end/ { d; found=true} else ...
4
votes
3answers
279 views

Convert genotypes to 0/1

I have a file that looks like: 1 rs6687776 1020428 T C T C T C C C T C C C T C The 4th and 5th column are the two different possible alleles at that ...
1
vote
2answers
91 views

Print the first and last match of a field with awk

I have a tab delimited file: qrs John tuv abcd Sam efgh ijk Sam lmnp abcd Sam efgh ijk Sam lmnp qrs John tuv I am trying to print the line in which the ...
3
votes
5answers
326 views

Find text between tab (\t) as a delimiter

I thought this will be simple, but can't find out how to do it. Scenario I have a single .csv file with id_user,text,id_group columns where each column is delimited by tabs such like: "123456789" ...
4
votes
1answer
103 views

Thousand separator in awk

I would like to print thousand separator with awk's printf. In bash, it is pretty straightforward: > printf "a %'d b \n" 1234567 a 1,234,567 b > LC_NUMERIC=fr_FR.utf8 > printf "a %'d b \n" ...
1
vote
2answers
41 views

Split file in two on line before pattern

This is an almost identical question to split file into two parts, at a pattern but instead of having the lines up to XYZ in file1 in the example file.txt: ABC EFG XYZ HIJ KNL I want to have the ...
6
votes
1answer
158 views

Handling 3 Files using awk

Consider following files: file1: boo,8,1024 foo,7,2048 file2: foo,0,24,154 noo,0,10,561 file3: 24,154,7,1024,0 What I need is to go to File1 and check if $2==7; if true, take $1, $2 and $3 ...
0
votes
1answer
31 views

Print specific FULL column only by range

I have a file which have contents like this. Column1 Column2 Column3 Column4 USER1 25 Years xyz zyz 123 abc ...
10
votes
5answers
406 views

How to print own script name in mawk?

In bash $0 contains the name of the script, but in awk if I make a script named myscript.awk with the following content: #!/usr/bin/awk -f BEGIN{ print ARGV[0] } and run it, it will only print ...
1
vote
2answers
54 views

Remove last comma within a pattern

If I have the following text within a file (somefile.txt): CREATE TABLE "table_name" ( "id" int(11) unsigned NOT NULL, "some_field" varchar(10), ); CREATE TABLE "another_table" ( "id" ...
0
votes
2answers
24 views

Delete multiple lines with text and curly braces

I have a multiple files that contains something like: this is a test1 { test 123 test 456 test 789 } this is a test2 { test 123 test 456 test 789 } this is a test3 { test 123 test 456 test 789 } ...
0
votes
2answers
31 views

Merge labels from lines with duplicate fields

I have a rather large text file(thousands of lines) in following format: 123.123 5 123.123 91 156.456 45 233.185 85 233.185 4 257.448 1 455.456 60 455.456 2 The first column are ...
0
votes
3answers
104 views

How to replace contents of each line depending on some formula?

I've been working with bash to get some data output to a multiline file with the data stored in this format for each row: text1 "text2" text3 integer1,integer2 A "space" separates the first four ...
8
votes
8answers
715 views

Split text file into lines with fixed number of words

Related, but no satisfactory answers: How can I split a large text file into chunks of 500 words or so? I'm trying to take a text file (http://mattmahoney.net/dc/text8.zip) with > 10^7 words all in ...
1
vote
2answers
45 views

With awk, how to Insert a number only for lines starting with “Toto” when they are between a specific couple of patterns

With awk, I would like to insert a numbering like Record n°i# in front of each line starting with Toto only when they are located in-between two specific patterns start=ABCD and stop=EFGH. Input file ...
2
votes
4answers
138 views

extract url between 2 strings in a file

I have a file in which each line is like this "372"^""^"2015-09-03 06:59:44.475"^"NEW"^"N/A"^""^0^"105592"^"https://example-url.com"^"example-domain < MEN'S ULTRA < UltraSeriesViewAll ...
2
votes
3answers
69 views

How to add “wrappers” around methods in source code based on a pattern, using sed, awk, grep and friends

How can I add wrappers to a file based on a pattern? For instance I have the following: ... find(css_policy_form_stage3).click with_ajax_wait expect(css_coverage_type_everquote).to ...
0
votes
2answers
51 views

Replace values on the basis of matching ID variable

I have two files. file1 has 6 columns, file2 has 2 columns. I want to replace the values in the 6th column of file1 by the values in column 2 of file2, based on matching ID variables (column 1 in both ...
1
vote
4answers
116 views

Replacing a “time” column with corresponding values

I have a file in the following format: "2004-04-19 12:25:57" 44 44 "2004-04-19 13:39:32" 36 36 "2004-04-19 14:00:53" 34 34 I need 2 new files: a) A file which will replace "time" values of the ...
5
votes
8answers
551 views

awk or sed command to match regex at specific line, exit true if success, false otherwise

I need to determine if a file contains a certain regex at a certain line and to return true (exit 0) if found, and otherwise false. Maybe I'm overthinking this, but my attempts proved a tad unwieldy. ...
0
votes
1answer
63 views

Replace the first word of each line with its MD5

I have file with two space separated word in each line. I am looking for shell scrip in Unix to replace the first word of each row with its md5 hash representation. I have tried to adjust what is ...
-1
votes
3answers
54 views

How to delete certain line from file?

I have a text file with line containing only comma. How can I delete this particular line? Consider following example: CREATE TABLE ENTERPRISE.TSDAILYACCTSUMMARY ( TSDAILYACCTID FOR COLUMN ...
-1
votes
1answer
70 views

How to add double quotes to a various strings in a same pattern in a file

I have a file like this CREATE TABLE ENTDTA.$$BAPWD ( PERSID NUMBER(10) DEFAULT 0 NOT NULL , "PASSWORD" CHAR(10) DEFAULT '' NOT NULL , UPDATE_IDENT NUMBER(7, 0) DEFAULT 0 NOT NULL ) ...
1
vote
1answer
76 views

OSX: Delete all matching certificates by command-line

Using shell script to delete all matching certificates. I found a list of certificates using: security find-certificate -c "certificatename" -a -Z|grep SHA-1|awk '{ print $NF }' Output is like ...
1
vote
1answer
32 views

How can I use awk to create a variable that is used in the next line edit

I have a file with a list of info. Each line has either 12 or 13 fields. For example I will use a file will 3 and 4 fields. e.g. myfile.txt user password location date john passwd1 new_york today ...
2
votes
1answer
70 views

Merge lines with matching first column

I've been trying to use awk to merge two files when they have the same first column. Here are my example files: FileA.txt A2M 1 A4GALT 11 AAAS 35 AAGAB 7 FileB.txt A4GALT 2 AAAS ...
3
votes
2answers
112 views

Why awk says “syntax error” for the comma I placed between the two patterns?

I run the following script under Linux (Mint on a USB key) with GNU Awk 4.0.1 installed by default. Error returned by awk is the following : awk: cmd. line:7: /^Observation/,/^@@@/ ...