awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

1
vote
2answers
137 views

Search 2nd line of files and print filenames to stdout

I have a few directories of ~10,000 files. What's the quickest way to search each file and return the filename if the second line contains a specific string? Edited for clarity
2
votes
6answers
1k views

Handling comma in string values in a CSV file [duplicate]

I have a comma-separated file which has numeric and string columns. String columns are quoted and can have comma in between the quotes. How do I identify the columns with FS =","? Sample records ...
13
votes
8answers
9k views

How to print the longest line in a file?

I'm looking for the simplest method to print the longest line in a file. I did some googling and surprisingly couldn't seem to find an answer. I frequently print the length of the longest line in a ...
4
votes
2answers
383 views

How do I populate end tags using sed, awk or any other command?

I have a txt file that looks like this: <sss>ss< or <firstword>anotherword< I want it to look like this <sss>ss</sss> or ...
3
votes
3answers
457 views

How can I delete all text between curly brackets in a multiline text file?

Example: This is { the multiline text file } that wants { to be changed } anyway. Should become: This is that wants anyway. I have found some similar threads in the forum, but they don't seem ...
8
votes
4answers
441 views

Remove duplicate lines from a file that contains a timestamp

This question/answer has some good solutions for deleting identical lines in a file, but won't work in my case since the otherwise duplicate lines have a timestamp. Is it possible to tell awk to ...
2
votes
1answer
127 views

Calculate median value for a column and say if this value is “OK” or “ERROR” if falls within acceptable range

I would like to calculate the median value of a particular column in a .txt file, output that value and also say if that output value is ok. I can do the initial steps and output the median value, ...
2
votes
2answers
67 views

Parse several thousand lines of txt into lines and columns

The file at hand lists names, credentials, roles, and privileges for thousands of users. It was passed onto me as .xls but the lines and columns are not arranged in proper form. I have reformated ...
14
votes
10answers
3k views

There must be a better way to replace single newlines only?

I am in the habit of writing one line per sentence because I typically compile things to LaTex, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the ...
2
votes
3answers
50 views

Removing lines from one file based on non-existence in another

I need to remove lines from fileA which has two columns if fileB (one column only) does not contain the string from column1 in fileA. Any linux command would help. Thanks. Example. Here is fileA: a ...
0
votes
1answer
47 views

Need to compare the two files and print all the columns then missing display as NA, then join another file [duplicate]

File 1: 123, 234 345, 789 678, 900 File 2 123, abc 345, dnc output (all columns in file1 and missing value in file two as NA): 123,234,abc 345,789,dnc 678,900,NA Output file add with another ...
1
vote
3answers
298 views

using awk command to compare the two files and print all the columns then missing display as NA

File 1: 123, 234 345, 789 678, 900 File 2 123, abc 345, dnc Desired output (all columns in file1 and missing value in file two as NA): 123,234,abc 345,789,dnc 678,900,NA
7
votes
3answers
158 views

Is there a way to make this one-liner faster?

Context I have a directory of thousands of zip files that are dated in the form YYYYMMDD_hhmmss.zip and each about 300K. Within each zip file is about 400 xml files each about 3K. The problem I ...
2
votes
2answers
124 views

How to capture the port number from this `lsof -F` output using awk (or something better)?

I've got output coming from lsof -F that looks like this: p7646 g7646 R8300 csocat u1000 Labe f3 au l tIPv4 G0x80002;0x0 d640391 o0t0 PTCP n*:51352 TST=LISTEN TQR=0 TQS=0 I am trying to capture the ...
0
votes
1answer
19 views

add fields from different lines in one line

I'm trying to add the .data and .bss sections of the size -A output size -A mylib.so | grep "\.data" | awk '{print $2} this will print the value of the size of the .data section. I guess I could ...
0
votes
2answers
62 views

Grouping data and assigning alphabet and numeric values for new categories

I have 4 columns in my tab delimited data, I need to perform 2 steps. Group the data according to combinations of first 2 columns. Every time a new combination of values appear in the concatenated ...
0
votes
2answers
79 views

Delete consecutive lines if patterns match in each line

Good day, I was wondering how to delete to consecutive lines when this pattern is found: /^pt [a-z]\npy [0-9][0-9][0-9][0-9]\n// Expected Input pt a py 01234 pt b cd abc py 5678 Expected Output ...
1
vote
2answers
68 views

Print only the second field of the second line

I have to extract from the command hcitool dev only the mac address of the BT dongle. Output of hcitool dev is Devices: hci0 xx:xx:xx:xx:xx:xx I write this output on a file and try to get the ...
4
votes
4answers
359 views

How do I remove every number that's surrounded by <>

I have tried this sed -i '' 's/[0-9]*<>/g' But it didn't work. Example file: <Number1> </Number8> output: <Number> </Number>
0
votes
5answers
84 views

Subset data with awk

I want to subset data with awk. Lets say I have this file called test: IP MAC Bandwidth etc etc 192.1.1.1 ff:ff:ff:ff 5.421M 192.1.2.3 ff:ff:ff:f3 5.120M 192.1.2.5 ff:ff:ff:f1 5.100M stuff I dont ...
0
votes
2answers
74 views

How to list only filenames removing directories and subdirectories

How can I get the list of only filenames and remove directory and subdirectories using find, sed, awk, cut or any. For e.g : I get list of all media files using find as such: find /var/warehouse/* ...
2
votes
4answers
65 views

Awk remove field only if numeric

ls /var/log/packages/*SBo|cut -d / -f 5 return webcore-fonts-3.0-noarch-3_SBo winetricks-20120923-noarch-1_SBo wireshark-1.12.1-x86_64-1_SBo wxGTK-2.8.12-x86_64-3_SBo I want to substitute from ...
1
vote
1answer
63 views

Multiply rows of a column in different chunk of data, add them together and get the average?

I have a chunk of data that looks like this: 924 777 777 777 8445.65349 P 9.44657 -0.840283 -0.394467 C 4.0555 -0.683104 -1.70917 P 7.5169 3.54172 2.77681 C 3.21536 2.67146 2.20242 P 4.33641 7.62012 ...
1
vote
1answer
52 views

Linux log file structure change for usage with kst

My file (tcp dump from networked sensors) has this shape: 20:06:57.049686 IP (tos 0x0, ttl 64, id 26871, offset 0, flags [DF], proto UDP (17), length 68) e108-193.eduroam.tugraz.ac.at.34225 > ...
0
votes
1answer
98 views

How to update files based on another file

I have 4 csv files and I want to update set of three based on values in the other one. The first 2 files (file_1 and file_2) contains names and their numerical ids. The third file (file_3) contains ...
4
votes
4answers
306 views

How to remove all comments from a file?

I have a file with a few lines commented out: foo bar stuff #Do not show this... morestuff evenmorestuff#Or this I just want to print all the uncommented code: foo bar stuff morestuff ...
1
vote
2answers
65 views

Extract and format data with `cut` and `awk`

I have the below file in txt: "Firstname","Lastname","Company" "Alex","Johnson","Alex Corp." "Baran","Atasoy","Gülerler Ltd." I would like to extract and format data by using awk and/or cut and ...
1
vote
1answer
41 views

Insert missing string in multiple ordered columns

I have a multi-column file that looks like this: 6 93129B 177 306.98 350.20 8.84 0.23 -0.79 0.22 0.46 m 7 8 192 269.08 413.35 9.29 0.15 -0.81 ...
2
votes
2answers
66 views

How to wrap the comma delimited values to the next line without disturbing the other fields using awk or ksh

I have the following input, in which some of the values are comma delimited. I would like decouple each value and print it in the new line. I tried many ways in awk, but no luck. If I have three ...
2
votes
3answers
428 views

Using grep/awk/sed to sort and combine 2 files

Ok I am running into a problem. I am taking a wifi log that has MAC addresses listed in it. I want to find out what MAC vendors/manufactors for the devices that are connected to my router. I have 2 ...
3
votes
2answers
288 views

Shell script to assign values from a lookup table is too slow

I'm trying to assign some variables from a lookup file with a shell script. I have something working but it seems unnecessary slow. Script: while read line do ...
0
votes
1answer
65 views

How to copy lines in awk command? (ksh scripting is preferred)

I have the below index file key1|1|1001 key1|1|2001 key2|2|3001 key2|2|4001 using this index file, I have to update my main file key1|1000|2000|3000|4000 key2|1000|2000|3000|4000 The ...
0
votes
1answer
30 views

Using find to search and report files that have one but not two keywords inside

I'd like to use find to report files under the following conditions: If the file contains the word "Serializable" Then report the file if it does NOT contain the word "serialVersionUID" Something ...
1
vote
4answers
273 views

How to convert fixed length fields in a file to space delimited

I have a fixed length matrix type file where each field is 20 characters long. How can I convert this to a space delimited file? Also there are some missing values, which I would like to ...
1
vote
1answer
58 views

count matches and mismatches group by

Please help with a shell script of the following. I need count number of consistent variables in each lane (col1) across samples (col2). For example, since all the values (col4) of lane1 variable 1 ...
3
votes
2answers
44 views

Help with iterating fieldA in CSV file, where fieldB has specified value

I've been trying to setup a script to run on gitbash, to process a CSV file so each record that has a specific value, the last field is changed from empty ("") to a value that iterates from 1 to 16. ...
0
votes
3answers
76 views

How to print loop index to a file using awk?

I'm a new in bash scripting. I need a help in how to print the loop index to a file using awk? Here is the code I used. echo "Write the start and end file number:" read sta end echo "$sta" "$end" for ...
2
votes
2answers
63 views

How to exclude lines that has given columns?

There is an input file that has TAB delimited columns. We need to remove the lines which has "NA" for the fourth AND the eleventh column. Question: how can we do this in awk?
0
votes
1answer
65 views

Perl/awk/bash: multiply and sum values putting values inline instead original text [closed]

I have a text like this: ;--------------------------------------- JOURNEY_ARRAY_01 .BYTE <JOURNEY_00 .BYTE <JOURNEY_01 ;--------------------------------------- JOURNEY_ARRAY_03 ...
1
vote
2answers
89 views

Print a line in stdout that matches an expression if the output contains another expression

This might be a common/easy task but I couldn't figure out from examples on the web or manuals of awk/sed/grep. So, here is the scenario: There is an internal command line tool that prints out a ...
3
votes
2answers
113 views

How can I partition a set of words given pairs that must end up together?

I have a space or comma separated table with two columns, each row representing equivalence of the two words. A B B C B D C E F G What I want is a table with each row listing all mutually ...
3
votes
3answers
2k views

Count records matching pattern with Awk

So I was told to do last > lastloggedin which creates a file that shows the classes last login since the last system reboot, and now I am asked to write an Awk script which is named myawk that ...
0
votes
2answers
70 views

Why awk does not show the correct record length?

I'm concerned regarding what awk shows as the record length. I'm checking some files for a specific record length - awk shows the result I wanted, but the file size shows that each record in the file ...
0
votes
0answers
26 views

to check fields in configuration file

i have configuration file in which first field is filename, second field is the column if we find the value of that column is null in the filename that is given in first column. if any of value of ...
5
votes
1answer
2k views

Replacing multiple lines in sed or awk [closed]

I'm trying to use sed or awk to replace 5 lines in a smb file but I just don't have any idea how to deal with the newlines.
1
vote
4answers
190 views

Sed/awk/perl: reversing order of comma separated values, preserving other text

I have this text: LABEL1 .BYTE 01, 02, 03, 04, 05 .BYTE 01, 02, 03 I need to reverse only comma separated values order: LABEL1 .BYTE 05, 04, 03, 02, 01 .BYTE 03, 02, 01 I need ...
1
vote
3answers
123 views

How to List files which contain white space in filename? [duplicate]

I am trying to read files one by one by using the for loop like below, by passing argument to the functions. Argument 3 makes lot of problem due to space in file names. Example: ...
1
vote
2answers
43 views

Adding entries to the end of a row via awk

I was wondering if anyone would be able to point out what is going wrong here. I have a txt file of data, delimited by a space, of 7 columns by 10000 rows. I want to add two extra columns one with the ...
0
votes
1answer
80 views

Using ARGV to get user input in Awk script [closed]

I know ARGV[i] can be used for storing user input. However, I want to use it in awk script and get the ARGV[i] to compare the field from another text file. If the ARGV[i] matches the field or the ...
1
vote
3answers
127 views

How to align a table of display nicely

I am trying to create a script that act as a ps command with it's own properties that I want to display. Say this is how processes would look like in ps command: sas 24431 1 0 Oct10 ? ...