Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [awk]

A pattern-directed scanning and processing language.

4
votes
8answers
193 views

Separate the element after every 2nd ',' and push into next row in bash

An example of my file looks like this: 201012,720,201011,119,201710,16 Output I want: 201012,720 201011,119 201710,16
2
votes
2answers
58 views

How to remove duplicate values based on multiple columns

I have a file tmp.log with fields like description ID valueA valueB valueC xxx x 1 1 1 yyy y 3 100 23 zzz z 0 0 0 aaa a 4 ...
0
votes
1answer
27 views

List lines that are the same in the second part but differ on the first part

Suppose that you have a big file with lines of this format: A, B, C, D (each line has 4 parts comma separated). I need a list of all the lines which have common fourth part (D is the same across those ...
3
votes
1answer
32 views

Iteratively adding value to column of text file

I have a 3 column text file (XYZ coordinates) and I need to iteratively add constants to the first column while appending to the end of the original file. I have tried several options but this is the ...
0
votes
1answer
31 views

How to group a string that has multiple words with awk? (Prints lines between 2 patterns)

I have an issue in which my string its not a single string; it is a string with multiple words; in which using sed works great; but not with awk; How can I group all that string with awk? I tried to ...
1
vote
1answer
21 views

print a sequence in a file

I would like to print in a column in a new file the following 1_number_density_per_atom_1.xvg 1_number_density_per_atom_2.xvg 1_number_density_per_atom_3.xvg . . . 1_number_density_per_atom_653.xvg ...
5
votes
4answers
75 views

how to insert new line if a line has more than 60 characters

I need to turn a text file that has a single line per file name and output separated by a single space into specific blocks that have lines that are equal to 60 characters in length. Like this: >...
0
votes
4answers
45 views

how to capture IP address from info

by the following api we can get the info about master1/2 machines curl -sH "X-Requested-By: ambari" -u "admin"":""admin" -i http://192.23.39.2:8080/api/v1/hosts?fields=Hosts/host_name,Hosts/ip | ...
1
vote
2answers
48 views

how can I merge multiple column in one column and separated by '-'?

I have a table as below: 1 10 15 2 2 25 1 10 26 I like to merge them and make a new column in linux, like below: 1 10 15 1:10-15 2 2 25 2:2-25 1 10 26 1:10-26
0
votes
2answers
24 views

Read the data page wise and store in seperate files

I have a huge text file where data in seperated pagewise.for example The Text is in below format i want to read lines page wise and store in seperate files Page 1 line 1 line 2 line 3 line 4 ...
0
votes
1answer
49 views

Issue while creating csv with awk

I am trying to create a csv file with AWK. I am using output file separator to add commas for columns. But there is one column whose name is separated by space and Its also getting processed in the ...
0
votes
4answers
35 views

change format of column

I have in the input file a column with the following format H6 H7 O4 C4 H8 H9 O5 C5 H10 H11 I want the output to have the following format "a H6" "a H7" "a O4" "a C4" "a H8" "a H9" "a O5" "a C5" "a ...
10
votes
6answers
13k views

How do you check if a file exists within awk? [-d 'filename'] failing

I'm trying to generate a list of users who have a home directory set which does not exist. It seems I should be able to do this with awk, but something is wrong with my syntax. It keeps telling me "...
1
vote
2answers
39 views

Update rows by referring another file

I have two comma separated files as mentioned below, cat source.txt A,1000 B,3000 C,2500 D,5000 E,4000 F,3000 cat test.txt A,1000 C D,5000 B E How can I update file test.txt by referring file ...
1
vote
2answers
32 views

How to add a new column in between existing columns?

File1.txt: 1 Mark USA 2 Amir UK 3 Pant USA 4 Veer IND I have to add a new column 657890 768908 435678 342343 in 3rd position. So Output must be, Output.txt 1 Mark 657890 USA 2 Amir 768908 UK 3 ...
1
vote
1answer
15 views

awk to add to field based on condition

In the awk below I am trying to add adjust $2 in the ouput by adding +1 if the original value from file that was used in $2 had a - in it. Line 2 of in is an example of this. In my current awk I my ...
5
votes
5answers
379 views

Extract the characters before last colon

I need to extract the characters before last colon : and also remove the square brackets [] in the last line. My file structure is 256.XXX.XXX.X:20234 214.XXX.XXX.X:7249 [2200:XXXX:XXXX:XXX:XXXX:XXXX:...
1
vote
3answers
36 views

read file and based on criterion print desire output

I have input file which looks like this: H1 O1 C1 H2 H3 O2 C2 Br1 I want to have an output file which to have the following output H1 = 1 O1 = 8 C1 = 6 H2 = 1 H3 = 1 O2 = 8 C2 = 6 Br1 = 35 So when ...
0
votes
1answer
2k views

wget: Retrieving a list of URLs when modifying input data file on the fly

This issue is currently driving me up the wall. It just does not work as it should. I have a file inp with audio samples to download where I preserved the internal ID number by parsing some other ...
2
votes
4answers
780 views

how to separate line using awk

How can I separate the line as below in a csv file: (12,'hello','this girl,is lovely(adorable \r\n actually)',goodbye),(13,'hello','this fruit,is super tasty (sweet actually)',goodbye) to the below ...
2
votes
3answers
158 views

Grep value of a specific key from a String, concatenated of key : value pairs

I have a string which is concatenation of "key":"value" pairs seperated by "," like- KEY1:VALUE1, KEY2:VALUE2, KEY3:VALUE3 And from this string I have to grep value for a specific string,lets say ...
1
vote
2answers
132 views

Rearrange columns separated by : in LINUX [closed]

The question is how to rearrange the column along with values in desired order. input "a":"val1","c":"val2","b":"val3","d":"val4" "a":"val1","b":[],"c":"val3","d":"val4" "a":"val1","d":["val2","...
1
vote
1answer
66 views

Improve performance when using “system” call (shell escape) processing large files in awk

I have an awk script that processes very large files that look something like this: K1353 SF3987.7PD833391.4 KARE K1353 SF3987.2KD832231.4 MEAKE K1332 IF4987.7RP832231.2 LEAOS K1329 SF2787.7KD362619....
0
votes
2answers
45 views

Rearranging columns in UNIX/Linux

I have to move the unique key which is in 3rd column of each record to first column. Now based on this key each record has different total columns in it (means # of fields) File content is 10,,FH,...
0
votes
4answers
152 views

Conditional grepping

I have a config file and its content looks something like below: Jobname|Type|Silo|Description #comment1 #comment2 job1|me|silo1|test_job1 job1|me|silo1|test_job2 job1|prod|...
-1
votes
0answers
28 views

Compare 2 days and MTM on current day should not be zero when the deal value doesn't exist [on hold]

There are two files: one for each day. I need to sum the MTM (column6) value based on the Deal (column2). Like adding the MTM value for the same deal values for each file. After adding and getting ...
1
vote
1answer
58 views

Rsync , cp or any utility add specific filtered data back from original file to analyzed file [closed]

I have mistakenly deleted some useful data from my backup file but the problem is I have so far analyzed huge amount of backup file and now i cannot take a another backup and start analyzing from ...
1
vote
2answers
395 views

Using awk to process multiple files need to count occurance of variable after pattern. How can I stop array resetting after each file?

I want to count all occurances of any variable in parenthesis after patterns, for files in a directory that have a specific extension. Each file may contain the pattern on multiple records/line. So ...
0
votes
3answers
128 views

Deciphering this AWK script

xev | awk -F'[ )]+' '/^KeyPress/ { a[NR+2] } NR in a { printf "%-3s %s\n", %5, %8} When I use xev there is only a certain bit of information I want. The natural response of using xev to get keycode ...
3
votes
1answer
43 views

AWK compare to produce desired merged output

I am attempting to use AWK to compare two csv files and generate a merged output based on the compare results. Both files have a matched value and I can get it to display the desired value in data2 ...
0
votes
2answers
65 views

Rectifying awk statement

I have a file as below to do date and number validation. File: 006063416.01|USD|1| |00.00000|00.00000|O| |20100802|20160119| |D|+0000006063416|0000000000| |060.634164000| 06063416.001|...
0
votes
1answer
42 views

grepping HPC jobs

let say, I have table in the imagewhich is basically "qstat" of HPC jobs: I like to print out columns 2 and 6 which match NDS==1 and TSK==1 I am using command: qstat -a |grep ' $4=="1" $5=="1" ' |...
-1
votes
2answers
192 views

awk - replace pattern

how do I search the following pattern by awk in a file. '\b' The pattern is one of the lines in a file. I have to add text next to the pattern. Like this: '\b' TEXT Any of these commands are not ...
4
votes
3answers
124 views

Shell script to extract data from a list of files and saving it as a csv

I'm on CentOS. I have a list of files that I want to read, extract data from them and organize it as a csv file. The log files text format is: ... {"name":"test-api","hostname":"ci47","pid":3202,"...
0
votes
1answer
33 views

print row if number from file is between two columns

I want to match a list of numeric ids with two columns (each line has a different number/id) in file 1 >cat file1 1 23444 3 422255223 1 35541 and print only those lines from a larger file (file2)...
2
votes
4answers
5k views

How to extract column name (header) from a CSV file which contains the max value in a row?

I am trying to extract the column name with the maximum value in each row using bash script i.e., the column header value or the value from the same column in the first row. I am using the following ...
0
votes
2answers
49 views

AWK command to display Header, Transform and Footer?

I have used the below command, awk 'BEGIN{print "Header"};{FIELDWIDTHS ="3 4 3"}{print $1"|"$2"|"$3" |"};END{print "Footer"}' file8 Actual - Header 1234567890||| 234|5678|901| 345|6789|012| 098|...
0
votes
0answers
25 views

unix shell scripting :compare 2 days values and on current day the value should not be zero

I have two different files in two different location, like one file is generated on today and another one is generated on previous day.Now, i want to compare the data of column B for both the files ...
1
vote
1answer
882 views

How to automatically detect and write to usb with variable spaces in its name

I am doing the second BASH exercise from TLDP Bash-Scripting Guide, and I have most of it figured out up until the part when it comes time to copy the compressed files to an inserted USB. Home ...
0
votes
2answers
55 views

Which regular expression string ignores single comment lines from a source code file? [on hold]

I'm trying to parse a plain text file or a source code file for which I need to build a regular expression to detect single line comments. I'm trying the following one without success: re = sprintf("...
0
votes
1answer
876 views

how to print only Difference in two files using NR=FNR?

we have to print the old entries only Delta(Difference) in File-1 and File-2. newly added enties in files-1 no need to print. on my unix server everyday new report is extracted from system which is ...
0
votes
2answers
43 views

How to delete string other than pattern?

I have a m3u file as below. I want to delete all strings except the pattern and next first line. or save the pattern and the next first line as another file. Patterns group-title="TR ULUSAL" , group-...
0
votes
0answers
33 views

syntax error awk usage [closed]

I want to use the following code: for ((j=1; j<=20; j++)); do varioutput=$(awk -F, -v m=$j '{print $m}' data/damper.test_temp1.csv) k=$(($j+1)) awk -F, '{$k='$...
0
votes
1answer
28 views

Splitting tab-separated file depending on column value

I have a large file that I am trying to split into columns based on the values of the second column. I would like a series of output files that contains the data at the 'switching point'. The ...
0
votes
1answer
28 views

awk save value as variable

I use the command varioutput=$(awk '{print $j}' OFS=, data/damper.test_temp1.csv) because I want to extract the j th value of a line of many values seperated by , . But when I want to use $...
0
votes
0answers
23 views

Awk save the output to the same file [duplicate]

I am using the command awk -F, '{$12='$varioutput'; print}' OFS="," damper.test.csv to replace the 12th column with a value. The output is correct, but it does not save the output in damper.test....
2
votes
2answers
54 views

goto a specific line in awk to start processing [closed]

I want the awk to go to the beginning of the block of lines to start processing it again once it hit the bottom of that block and processed its last line. Basically, if block of text is <START ...
0
votes
3answers
58 views

change “\nX” into “Y”

I need to remove returns at the end of lines whose following lines begin with certain strings, and insert something, e.g. change "and " at the beginning of a line into " and " while removing the ...
0
votes
2answers
30 views

Replace pattern in all columns except in specific column

I would like to replace "colon" from each columns except from first column as it is. ]chr9:67290502]N SUPP_VEC=011 SVLEN=0 SVTYPE=TRA 0 0 0 0/1:NA:58745598:23,0:--:.:TRA:BND00068094:NA:NA:...
4
votes
3answers
144 views

Add Columns to .csv from Multiple Files

I'm trying to add columns to a csv file from multiple files. This is what I have tried so far, but keep ending up with either an empty file or just the column from the last file. while read i; do ...