awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
1answer
48 views

Linux log file structure change for usage with kst

My file (tcp dump from networked sensors) has this shape: 20:06:57.049686 IP (tos 0x0, ttl 64, id 26871, offset 0, flags [DF], proto UDP (17), length 68) e108-193.eduroam.tugraz.ac.at.34225 > ...
0
votes
1answer
85 views

How to update files based on another file

I have 4 csv files and I want to update set of three based on values in the other one. The first 2 files (file_1 and file_2) contains names and their numerical ids. The third file (file_3) contains ...
4
votes
4answers
195 views

How to remove all comments from a file?

I have a file with a few lines commented out: foo bar stuff #Do not show this... morestuff evenmorestuff#Or this I just want to print all the uncommented code: foo bar stuff morestuff ...
1
vote
2answers
59 views

Extract and format data with `cut` and `awk`

I have the below file in txt: "Firstname","Lastname","Company" "Alex","Johnson","Alex Corp." "Baran","Atasoy","Gülerler Ltd." I would like to extract and format data by using awk and/or cut and ...
1
vote
1answer
38 views

Insert missing string in multiple ordered columns

I have a multi-column file that looks like this: 6 93129B 177 306.98 350.20 8.84 0.23 -0.79 0.22 0.46 m 7 8 192 269.08 413.35 9.29 0.15 -0.81 ...
2
votes
2answers
63 views

How to wrap the comma delimited values to the next line without disturbing the other fields using awk or ksh

I have the following input, in which some of the values are comma delimited. I would like decouple each value and print it in the new line. I tried many ways in awk, but no luck. If I have three ...
2
votes
3answers
224 views

Using grep/awk/sed to sort and combine 2 files

Ok I am running into a problem. I am taking a wifi log that has MAC addresses listed in it. I want to find out what MAC vendors/manufactors for the devices that are connected to my router. I have 2 ...
3
votes
2answers
178 views

Shell script to assign values from a lookup table is too slow

I'm trying to assign some variables from a lookup file with a shell script. I have something working but it seems unnecessary slow. Script: while read line do ...
0
votes
1answer
49 views

How to copy lines in awk command? (ksh scripting is preferred)

I have the below index file key1|1|1001 key1|1|2001 key2|2|3001 key2|2|4001 using this index file, I have to update my main file key1|1000|2000|3000|4000 key2|1000|2000|3000|4000 The ...
0
votes
1answer
29 views

Using find to search and report files that have one but not two keywords inside

I'd like to use find to report files under the following conditions: If the file contains the word "Serializable" Then report the file if it does NOT contain the word "serialVersionUID" Something ...
1
vote
4answers
175 views

How to convert fixed length fields in a file to space delimited

I have a fixed length matrix type file where each field is 20 characters long. How can I convert this to a space delimited file? Also there are some missing values, which I would like to ...
1
vote
1answer
55 views

count matches and mismatches group by

Please help with a shell script of the following. I need count number of consistent variables in each lane (col1) across samples (col2). For example, since all the values (col4) of lane1 variable 1 ...
3
votes
2answers
42 views

Help with iterating fieldA in CSV file, where fieldB has specified value

I've been trying to setup a script to run on gitbash, to process a CSV file so each record that has a specific value, the last field is changed from empty ("") to a value that iterates from 1 to 16. ...
0
votes
3answers
49 views

How to print loop index to a file using awk?

I'm a new in bash scripting. I need a help in how to print the loop index to a file using awk? Here is the code I used. echo "Write the start and end file number:" read sta end echo "$sta" "$end" for ...
2
votes
2answers
58 views

How to exclude lines that has given columns?

There is an input file that has TAB delimited columns. We need to remove the lines which has "NA" for the fourth AND the eleventh column. Question: how can we do this in awk?
0
votes
1answer
47 views

Perl/awk/bash: multiply and sum values putting values inline instead original text [closed]

I have a text like this: ;--------------------------------------- JOURNEY_ARRAY_01 .BYTE <JOURNEY_00 .BYTE <JOURNEY_01 ;--------------------------------------- JOURNEY_ARRAY_03 ...
1
vote
2answers
83 views

Print a line in stdout that matches an expression if the output contains another expression

This might be a common/easy task but I couldn't figure out from examples on the web or manuals of awk/sed/grep. So, here is the scenario: There is an internal command line tool that prints out a ...
3
votes
2answers
109 views

How can I partition a set of words given pairs that must end up together?

I have a space or comma separated table with two columns, each row representing equivalence of the two words. A B B C B D C E F G What I want is a table with each row listing all mutually ...
3
votes
3answers
397 views

Count records matching pattern with Awk

So I was told to do last > lastloggedin which creates a file that shows the classes last login since the last system reboot, and now I am asked to write an Awk script which is named myawk that ...
0
votes
2answers
48 views

Why awk does not show the correct record length?

I'm concerned regarding what awk shows as the record length. I'm checking some files for a specific record length - awk shows the result I wanted, but the file size shows that each record in the file ...
0
votes
0answers
25 views

to check fields in configuration file

i have configuration file in which first field is filename, second field is the column if we find the value of that column is null in the filename that is given in first column. if any of value of ...
5
votes
1answer
2k views

Replacing multiple lines in sed or awk [closed]

I'm trying to use sed or awk to replace 5 lines in a smb file but I just don't have any idea how to deal with the newlines.
1
vote
4answers
160 views

Sed/awk/perl: reversing order of comma separated values, preserving other text

I have this text: LABEL1 .BYTE 01, 02, 03, 04, 05 .BYTE 01, 02, 03 I need to reverse only comma separated values order: LABEL1 .BYTE 05, 04, 03, 02, 01 .BYTE 03, 02, 01 I need ...
1
vote
3answers
79 views

How to List files which contain white space in filename? [duplicate]

I am trying to read files one by one by using the for loop like below, by passing argument to the functions. Argument 3 makes lot of problem due to space in file names. Example: ...
1
vote
2answers
34 views

Adding entries to the end of a row via awk

I was wondering if anyone would be able to point out what is going wrong here. I have a txt file of data, delimited by a space, of 7 columns by 10000 rows. I want to add two extra columns one with the ...
0
votes
1answer
61 views

Using ARGV to get user input in Awk script [closed]

I know ARGV[i] can be used for storing user input. However, I want to use it in awk script and get the ARGV[i] to compare the field from another text file. If the ARGV[i] matches the field or the ...
1
vote
3answers
91 views

How to align a table of display nicely

I am trying to create a script that act as a ps command with it's own properties that I want to display. Say this is how processes would look like in ps command: sas 24431 1 0 Oct10 ? ...
2
votes
1answer
53 views

How to replace the first line using sed?

Hi i am trying to replace the header line from my file using the sed command as mentioned below, sed "1s/.*/$new_header/" Test_file.csv The above line replaces the header and prints in stdout , ...
2
votes
2answers
73 views

Environment variable not expanded inside the command line argument

I have a file user-pid.out2 which has "usernumber" and "process id" as two columns. based on usernumber I want to find corresponding process id. the first two lines below does not show the output ...
0
votes
2answers
16 views

Replicate lines in text fine according to a column value

I have a text file with a column containing text/data followed by an abundance column, e.g. Car 3 Apple 2 In order to subsample these columns I would like to replicate each text column by ...
2
votes
2answers
53 views

Sed/awk/perl: modify text preserving parts and align to a column

I have a text like this: A1JOURNEY0TO1 .BYTE 00, 00, 00 A2JOURNEY0TO2 .BYTE 00, 01, 00 A3JOURNEY1TO0 .BYTE 00, 01, 01 I need to have: JOURNEY_01 ; 00 TO ...
1
vote
2answers
30 views

When interpret awk as a command or a programming language?

I know that awk is a script programming language, but sometimes I get confused about when I interpret as command or as a program. Eg. 1 - Here I interpret as a command: awk '{print $2}' file.txt ...
1
vote
1answer
46 views

redirect output of awk gsub command to file

I found an answer on how to remove commas between quotes of a comma delimited csv file over here on stackexchange (cannot comment there as I don't have the needed rep). awk -F'"' -v OFS='' '{ for ...
0
votes
1answer
100 views

Extract values from filename and add it into a text file

I have a gridded model data (separated by different depth) saved in text files. The structure is like this: Format of every column in a row in the text file: x_coordinate y_coordinate density ...
2
votes
2answers
94 views

Compare 2 delimited files and output differences

I have asked similar questions here a couple times in the past with great success, but now my needs have slightly changed and I am struggling to get the exact output I am looking for. I would like to ...
6
votes
5answers
178 views

Using awk to identify the number identical columns

I am new to all this, and this is probably a rather simple question, but I am stuck: I have a large number of individual files that contain six columns each (number of rows can vary). As a simple ...
-1
votes
3answers
55 views

Filtering a .CSV file based on consecutive numbers in a line

I have a CSV file as follows: 1st 2nd 3rd 4th ID ... Res Res Res Res (other columns) ... RZ_AUTO_1, 1cx0, C118, B, C119, B, A165, ...
2
votes
3answers
140 views

Use AWK to pad date and time formats

I would like to use AWK to pad the date and time fields in my report. Here is my current .csv that is actually tab delimited. AA 9/7/2014 2:30:38 PM AA0000011111 08 Y A Jane, Doe AA ...
2
votes
4answers
94 views

replace several comma out side one or more sets of curly braces and exception that in one or more sets of curly braces

In the text file I have multi records. Every record has multi column with comma separated, some column have one sets of curly brace and other have more than one curly brace. I need: If a comma is ...
0
votes
1answer
78 views

Cut a specific part of a column from a csv file [closed]

I have to process a csv containing about 15 columns one of which contains a particular timestamp that includes milliseconds. When I go to insert this csv into a DB, I get an error because of this ...
1
vote
1answer
84 views

Summing by common strings in different files

I have a file file1 with the amount of times a user shows up in the files, something like this: 4 userC 2 userA 1 userB and I have another file file2 with users and other info like: userC, degree2 ...
1
vote
1answer
108 views

splitting up a MySQL dump using awk

I have a large-ish (250G) MySQL dump file from which I only want / need one database. I was hoping to use awk to split the file up so I did some searching and found this expression: awk ...
2
votes
1answer
113 views

Sed to Modfiy Hosts File

I have a text file, modifyhostslist.txt, which contains entries that correspond to entries found in my hosts file. Not every entry in my hosts file needs to be modified, only entries also found in ...
1
vote
1answer
73 views

Sum values in the other columns for the 1st column with the same ID

The input txt file looks like (there are more columns in the actual txt file): target_id length eff_length tot_counts uniq_counts est_counts eff_counts mthl7 61 0 0 0 0 0 loqs 72 ...
12
votes
2answers
608 views

How does awk '!a[$0]++' work?

This one-liner removes duplicate lines from text input without pre-sorting. For example: $ cat >f q w e w r $ awk '!a[$0]++' <f q w e r $ The original code I have found on the internets ...
1
vote
2answers
126 views

Merging 2 files with based on field match

I want to create a file that contains columns from two input files. File1 is like: aa 32 bb 15 cc 78 File2 is: fa 19 bc 23 cc 50 de 28 aa 45 bb 31 The task is, read through File1, if the 1st field ...
1
vote
3answers
184 views

keep only certain part of a string in a certain column

I have a file like this: id target_id length eff_length 1 intron_FBgn0000721:20_FBgn0000721:18 1136 243.944268 1 intron_FBgn0000721:19_FBgn0000721:18 1122 ...
1
vote
1answer
72 views

Replace pattern in file with pattern in second file

I want to replace part of a file's data, with data from another file. Suppose file1 has data as is written below and file2 has some data I want to store and replace file1 data from file2 from ...
12
votes
8answers
2k views

How to perform a multi line grep

How would you perform a grep for text that appears on two lines? For example: pbsnodes is a command I use that returns the utilization of a linux cluster root$ pbsnodes node1 state = free ...
7
votes
3answers
423 views

Distributing a script: Should I use /bin/gawk or /usr/bin/gawk for shebang?

Is gawk in /bin or /usr/bin usually? I would go with #!/usr/bin/env gawk but then I can't use arguments. Right now I'm using #!/bin/gawk -f. The script is very long and contains a lot of single ...