Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [awk]

A pattern-directed scanning and processing language.

5
votes
4answers
170 views

Is there an efficient way to replace text matching the entire content of one file with the entire content of another file?

I have three files ~/naive-file.txt ~/old-text.txt ~/new-text.txt I want to find every instance where the contents of ~/old-text.txt occurs in ~/naive-file.txt and replace those occurrences with ...
1
vote
2answers
2k views

remove lines from a vcf.gz file with awk command

I just asked a question about filtering out lines with a specific value in a specific column. If I now want to delete lines with a specific value in a specific column. How do I do that? E.g. delete ...
0
votes
2answers
80 views

Linux awk grep from file 1 into file 2

I have regex in file 1 .*8912.*.*.*.*81415444.* .*8912.*.*.*.*81415444.* .*8912.*.*.*.*81415444.* .*8912.*.*.*.*81415444.* .*8912.*.*.*.*81415444.* .*8912.*.*.*.*81415444.* .*8912.*.*.*.*81415444.* .*...
22
votes
8answers
21k views

How to grep-inverse-match and exclude “before” and “after” lines

Consider a text file with the following entries: aaa bbb ccc ddd eee fff ggg hhh iii Given a pattern (e.g. fff), I would like to grep the file above to get in the output: all_lines except (...
1
vote
1answer
31 views

How to replace substrings with values taken from a dictionary indexed by the substrings

I need to parse a big file replacing possibly occurring substrings, that are part of possibly occurring strings that match a regex, with a value taken from an array whose indexes are the substrings in ...
4
votes
2answers
149 views

Use backslash or single-quotes for field separation

I have always been using single-quotes for the field separation like: awk -F';' ... Quite new to me is the way of using a backslash like: awk -F\; ... is there a technical difference for either, or ...
3
votes
5answers
159 views

Delete whole line if after “:” less than 4 chars

I have a file with many lines, such as : blue:fish red:tomato black:moon I would like the whole line to be deleted if after ":" there is less than 4 chars.
22
votes
6answers
17k views

How to select first occurrence between two patterns including them

How can I select first occurrence between two patterns including them. Preferably using sed or awk. I have: text something P1 something content1 content2 something P2 something text something P1 ...
2
votes
1answer
346 views

Is there any other method to uncomment in awk?

This command works perfect for me to uncomment a line. awk -i inplace '/# MyLINE/ { sub ("^ *#","") } { print }' file Just for curiosity, I am wondering if there are any other methods for "awk" to ...
2
votes
4answers
70 views

How to not display a column that has all NULL values in a file

I have a file that has let's say, 5 columns(obtained by redirecting the results of a Sybase select query). Each column is separated by a tab. I need to filter out the columns that have all NULLs. Any ...
1
vote
1answer
31 views

How to convert a space-separated file into an Excel file in Linux?

I have a file which looks like this: ID avedebv debv2019 ebv2018 gebv2018 number avecvlean 7063993 1.06109 0.5325 0.7420 1.5084 206 66.5962 7076247 1.67947 2.1066 2.8630 3.0629 460 67.6856 7076881 -0....
2
votes
1answer
45 views

Replace VI macro with Linux equivalent

My file arrives on a Linux server and contains ^M characters when seen in VI and the file format = unix, (ff=unix) I want to turn lines 1-4 into line lines 9 - 10. My vi macro solves my problem. /\r/...
29
votes
10answers
53k views

How to insert the content of a file into another file before a pattern (marker)?

File1 contents: line1-file1 "1" line2-file1 "2" line3-file1 "3" line4-file1 "4" File2 contents: line1-file2 "25" line2-file2 "24" Pointer-file2 "23" line4-...
0
votes
1answer
39 views

Skip lines with matching pattern and delete lines with condition matched

I have file with following format : Q L A D 1 10 2 21 3 22 4 5 5 9 I need to skip lines which starts with 'Q' or 'L' and delete lines where second column has value greater than 10 and save the ...
5
votes
4answers
308 views

First Match - awk

consider file having values: foo boo too and another one: foo,1 foo,2 boo,1 soo,1 How to get only first match from the 2nd file, where output will be: foo,1 boo,1
0
votes
1answer
18 views

syntax error near unexpected token `done' with 2 files generated

I am encountering syntax error when I am trying the below code to obtain the 2 output files namely input1,input2 as described below. test.txt Q L A D 1 10 2 21 3 22 4 5 5 9 while read -r line; ...
0
votes
2answers
375 views

Creating a file from another file on the basis of a numeric value in each line

We will generate a csv file with below values. yp1234,577,1,3 yp5678,570,3,5 yp9012,132,8,9 I need to extract data and create files based on second column. If it's value greater than 577 then the ...
0
votes
3answers
46 views

AWK - sum plus and minus values into separate variables

I have a CSV file like this: 2019.04.15;3.75; 2019.04.29;-5.17; 2019.05.01;7.5; 2019.05.06;0.5; 2019.05.13;0.25; 2019.05.20;-8.5; I want to get the sum of the plus and the minus values off the ...
0
votes
2answers
40 views

AWK - Need to display total value of field and field number

I have data below that needed to count or total the value start with specific field. Sample below the needed field to compute is from Field5. For sample Field5 total is 42, field6 total is 107, etc.. ...
0
votes
0answers
54 views

What is a great way to list files found in a directory that add up to a certain file size

This list will then be used to create tar balls for easy movement. This is what I came up with so far. Add up files that are just greater than 10GB list them to file. find . -type f -exec ls -al {} ...
1
vote
6answers
2k views

How to print nth line after match / empty lines

I'm trying to write a short script that finds empty lines and then prints the nth line after the empty lines. For foo1 foo2 foo3 bar1 bar2 bar3 spam1 spam2 spam3 eggs1 eggs2 eggs3 printing every ...
0
votes
1answer
52 views

Why is my bash shell script changing spaces to linefeeds?

I started out writing a simple bash shell script to find duplicate files within a given directory using SHA1 to detect matches. Everything worked until I encountered file names with spaces. The ...
2
votes
3answers
5k views

Extracting subset from fasta file

I have a fasta file which looks like this: >chr1 ACGGTGTAGTCG >chr2 ACGTGTATAGCT >chrUn ACGTGGATATTT >chr21 ACGTTGATGAAA >chrX GTACGGGGGTGG >chrUn5 TGATAGCTGTTG I just want to ...
2
votes
3answers
37 views

How to delete lines containing string in column n using awk?

I have a tab-separated file that looks like this: $ cat in_file NC_013132.1 7260299 7261429 WP_012793281.1 NC_013132.1 7270674 7270862 NC_013132.1 7270674 7270862 ID=cds5678 NC_013132.1 ...
1
vote
3answers
59 views

AWK - How to count stored or index on array

I have data that need to be count how many occurrences. 1 2 1 2 3 1 And used the awk command below and work properly. awk -F'","' '{ a[$1]++ } END { for (n in a) print n "=" a[n] } ' text.txt And ...
0
votes
2answers
37 views

bash + regular expression + capture java version [on hold]

how to capture the jdk1.8.0_112 under /usr/jdk64 /usr/jdk64/jdk1.8.0_112/ while jdk1.8.0_112 is example and could be any other version as jdk1.7.0_112 jdk3.8.2_112 jdk1.8.0_117 jdk1.2.0_112 jdk2....
1
vote
1answer
34 views

Printing a variable in a file along with some existing columns

I have a long file with three columns. I want to add a column (as the first column) with '5' for the first 200 lines and then '6' for the next 200 lines and so on...
0
votes
1answer
17 views

Remote carriage return if there is only one column

A proprietary command similar to "df" returns me this : FOO_vol6 1714236984 1685700840 28536144 98% /root_vdm_10/FOO_vol6 FOO_vol4 253851840 64818424 189033416 26%...
2
votes
4answers
62 views

Filter YAML file content using sed/awk

I have a text file with the following content in it. $ cat hosts.yml [prod_env] foo.example.com bar.example.com [stage_env] foo_stage.example.com bar_stage.example.com [dev_env] foo_dev1.example.com ...
0
votes
1answer
22 views

How to sum each column and print column name and column sum using awk?

I have a file with a table of numbers; the first line is a header line. I want to sum each column and print out sum and column name for each column using awk. How can I do it?
0
votes
1answer
22 views

To subtract a text from the last value

I have a text file like below. I want the check the type of file based on the extension and perform the action. Example.2019.txt Example.test.2019.txt Example.test.2019.txt.gz Example.test.2019.txt.Z ...
0
votes
1answer
27 views

Print numbers from table

How to get four numbers from this table please? The original input is much longer, so it is necessary to restricted the area. Input: chil 2 44843.084886752418 44843.080972494339 ...
0
votes
2answers
20 views

SED or AWK to remove everything before the first dash and after the last

I have a private key file with some extra nonsense in there, and want just the text of the key. so: nonsense -----Begin Key----- keep this1 keep this2 keep this3 -----End Key----- nonsense should ...
1
vote
5answers
55 views

How to grep/awk/sed for a random pattern

I have a log file that I am attempting to pull a string out of. The numerical string is randomly generated and so far all of my attempts to use grep/sed/awk have been unsuccessful. I have an entry ...
0
votes
2answers
55 views

Unix command to get 0 count , when no files present for the day along with count

How to do I get the list days were no files been received. Have used the below command to get the file count along with the date: find . -maxdepth 1 -type f -printf '%TY-%Tm-%Td\n' | awk '{array[$0]+=...
5
votes
8answers
364 views

Separate the element after every 2nd ',' and push into next row in bash [duplicate]

An example of my file looks like this: 201012,720,201011,119,201710,16 Output I want: 201012,720 201011,119 201710,16
0
votes
4answers
44 views

Read the data page wise and store in seperate files

I have a huge text file where data in seperated pagewise.for example The Text is in below format i want to read lines page wise and store in seperate files Page 1 line 1 line 2 line 3 line 4 ...
2
votes
3answers
77 views

How to remove duplicate values based on multiple columns

I have a file tmp.log with fields like description ID valueA valueB valueC xxx x 1 1 1 yyy y 3 100 23 zzz z 0 0 0 aaa a 4 ...
-1
votes
2answers
36 views

Specifying the range of the arguments to print in every record line

We have a sample data as below EMP_ID|EMP_NAME|AGE|SALARAy|DepId|DepName 123456|XXXXXXXXX|30|10000000|1234|XYZWE I have used below code to split the data as below awk -F\| ' BEGIN { print "EMP_ID|...
1
vote
1answer
31 views

how to print column names as well while reading the values for that variable

I have below sample data EMP_ID|EMP_NAME|AGE|SALARAy 123456|XXXXXXXXX|30|10000000 Is there a way I can split the record into multiple records and also get header. Example output should be like ...
1
vote
3answers
1k views

Printing Characters after a Colon

How do I print anything after the colon? Input: color:white,name:green so I would like to print anything after : Output: white,green
1
vote
2answers
33 views

print a sequence in a file

I would like to print in a column in a new file the following 1_number_density_per_atom_1.xvg 1_number_density_per_atom_2.xvg 1_number_density_per_atom_3.xvg . . . 1_number_density_per_atom_653.xvg ...
5
votes
7answers
8k views

grep first n and last n characters from a line in a file

I have a log file which looks like: Mar 23 08:20:23 New file got created in sec: 235 Mar 23 08:21:45 New file got created in sec: 127 Mar 23 08:22:34 New file got created in sec: 875 Mar 23 08:25:46 ...
2
votes
1answer
45 views

CSV - Converting SOME Columns to Rows with duplicated data

I am working with an unresponsive vendor whose CSV files for inventory are completely unsearcheable by Excel. My team has no knowledge or use of Unix, but I know that a simple IF loop or two with awk ...
0
votes
5answers
96 views

AWK from text with modification

I have the following file: hello there. let's try&ACCT=1&bla bla bla&EX=1118&anything ss &SERIAL=1011&DEAL=NO BLA BLA BLA&TOM=1&ACCT=2&LSLSLSLSL&none=1&EX=...
2
votes
3answers
278 views

Awk: Count occurrences of a string in one column, between a range of lines starting 2 lines below pattern 1 and ending with a condition

I have an input file with several thousand lines. I am interested in one section of it, which contains the first instance of pattern /m.o./ in the file. I need to search for this pattern, run my code, ...
8
votes
5answers
55k views

How to convert from one date format to other format?

I have one problem with my date format. I want to change from one format to the other and vice versa. My date formats are Format 1 YYYY-MM-DD Format 2 MM/DD/YYYY I want to change format 1 to ...
2
votes
4answers
172 views

Grep value of a specific key from a String, concatenated of key : value pairs

I have a string which is concatenation of "key":"value" pairs separated by "," like- KEY1:VALUE1, KEY2:VALUE2, KEY3:VALUE3 From this string, I have to grep for a specific string -- let's say KEY2 -- ...
0
votes
1answer
36 views

List lines that are the same in the second part but differ on the first part

Suppose that you have a big file with lines of this format: A, B, C, D (each line has 4 parts comma separated). I need a list of all the lines which have common fourth part (D is the same across those ...
3
votes
1answer
33 views

Iteratively adding value to column of text file

I have a 3 column text file (XYZ coordinates) and I need to iteratively add constants to the first column while appending to the end of the original file. I have tried several options but this is the ...