Tagged Questions

awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

4
votes
6answers
263 views

Command line tool to “cat” pairwise expansion of all rows in a file

Suppose I have a file (call it sample.txt) that looks like this: Row1,10 Row2,20 Row3,30 Row4,40 I want to be able to work on a stream from this file that is essentially the pairwise combination of ...
2
votes
2answers
176 views

Why does $var give the value of $0?

I have written a very simple awk script that reverses every word of a file-: awk ' { for (i=1;i<=NF;i++) { x="" for(j=length($i);j>0;j--) x=x substr($0,j,1); print x }}' file1 The contents of ...
0
votes
0answers
35 views

awk - how does the for loop work when counting words? [duplicate]

I have got the following awk script from here that counts the frequency of the words in a file-: # Print list of word frequencies { for (i = 1; i <= NF; i++) freq[$i]++ } END { ...
0
votes
2answers
36 views

Read file and find all occurances and generate hash for the content in between the occurance

I have a file like this : 1 Record|1111|ABC 2 text in between for record 1 3 text in between for record 1 4 Record|2222|XYZ 5 text in between for record 2 6 Record|3333|XYZ 7 text ...
3
votes
1answer
73 views

How do I use awk to create an index of words in file?

I have a problem with my homework I need to create a shell script using awk. The script will output only the words in the file in an alphabetical order, and after the word is : after that a space then ...
1
vote
3answers
47 views

Merging multiple rows into a single one using awk

I'm working with a very large data set in the format: chr1 45162 . . C T chr1 45162 . . C T chr1 45162 A . . T chr1 45162 . . C T chr1 45257 A . . T chr1 45257 A . . T chr1 45413 A . . T chr1 46844 ...
3
votes
3answers
118 views

Split in awk not not printing array values

I have a string tstArr2 which has the following content '3 5 8' Now in awk I want to parse a flat file test my array which array is better array INDIA USA SA NZ AUS ARG GER BRA US AUS INDIA ENG ...
1
vote
3answers
39 views

making 4 out of 1 file with awk

i have large list - text file with 1 column of numbers (length=14). Numbers have 4 difrent prefixes 00001-00004. how do i split 1 list to 4 diffrend files. 1 file shoud contain list of numbers with ...
1
vote
2answers
40 views

Save AWK FNR as a Bash variable to continue search from the line

I'm trying to get to run this on a sorted file. This returns the last line which is less than or equal to the given value comparing the first 9 characters. This would be done multiple times and VALUE ...
3
votes
5answers
163 views

removing extensions in a column

I have a file like this ILMN_1343291 TGTGTTGAGAGCTTCTCAGACTATCCACCTTTGGGTCGCTTTGCTGTTCG NM_001402.5 ILMN_1343295 CTTCAACAGCGACACCCACTCCTCCACCTTTGACGCTGGGGCTGGCATTG NM_002046.3 ILMN_1651209 ...
0
votes
3answers
34 views

How can I convert timestamps in a column to a date?

I have a file containing this: 1415602803,LOGIN SUCCESS,AUTH,user2,192.168.203.63,10.146.124.73,59996,22 1415602807,LOGIN SUCCESS,AUTH,user1,172.24.31.10,172.32.1.1,48191,22 1415602811,LOGIN ...
1
vote
0answers
30 views

Unable to run awk script [closed]

I'm running a tcl script which is going to perform an network performance analysis(running it in NS2) I'm unable to run the awk script and it is throwing me an error "improper conversion" if {$argc ...
3
votes
4answers
60 views

Replace a marker in a text file with arbitrary new text

I have a file which contains some text and - on a single line - a marker which indicates where new content should be added foo bar %SUBSTITUTE% foo The line line substitute should be replaced by a ...
3
votes
2answers
197 views

Delete every Nth line in shell

I'm trying to delete a line after N lines using awk and I can't seem to get it right. The file format is like this YYYYYY XXXXXX XXXXXX YYYYYY XXXXXX XXXXXX The real example would be office3 3 1 ...
8
votes
6answers
401 views

How to append Line to previous Line?

I have a Log file which need to be parsed and analysed. File contains something similar like below: File: 20141101 server contain dump 20141101 server contain nothing {uekdmsam ikdas jwdjamc ...
6
votes
4answers
293 views

Output part of each line to a separate file

I have a file like this: a AGTACTTCCAGGAACGGTGCACTCTCC b ATGGATTTTTGGAGCAGGGAGATGGAATAGGAGCATGCTCCAT c ATATTAAATGGATTTTTGGAGCAGGGAGATGGAATAGGAGCATGCTCCATCCACTCCACAC d ...
1
vote
2answers
21 views

Comparing value in same column and directing output to new column

I have my input file as DPortal=ECCN RemoteFile=4004_130122 0256 A02 141111 0940 29343414 11 110005 DPortal=ECCN RemoteFile=4004_130122 0256 A02 141111 2336 29843714 11 110006 DPortal=ECCN ...
3
votes
3answers
158 views

Extracting part of lines with specific pattern using awk,sed

I have a question regarding the awk/sed operators. I have a big file which has following set of lines repeated Expression loweWallrhoPhi : sum=-6.97168e-09 Expression leftWallrhoPhi : ...
1
vote
3answers
33 views

How to add a column from a file to the last column of another file using awk?

I want to add the second column of file2 to file1, that contains a number an unknown number of columns, so becoming the last column of file1. In this aim, I tried to create a file3 composed by pasting ...
1
vote
2answers
61 views

awk - Group by and sum column values

I have command to list system process by memory usage: ps -A --sort -rss -o comm,pmem Which list a table like COMMAND %MEM firefox 28.2 chrome 5.4 compiz 4.8 ...
1
vote
1answer
29 views

How to print odd numbered fields from a text file using awk in Unix? [closed]

I have posted the same question before but as I an just a beginner in Unix. I am unable to understand the answer given to it though certainly that helps to fulfil my requirement. awk -F\| '{s="";for ...
0
votes
0answers
20 views

jumbled character set

In my Linux env, for some unknown! reason when I process a raw unformatted file which I copied from Win7prof (was encoded in utf-16), the resulting file is all jumbled. It will print OK in Linux ...
1
vote
2answers
87 views

Search 2nd line of files and print filenames to stdout

I have a few directories of ~10,000 files. What's the quickest way to search each file and return the filename if the second line contains a specific string? Edited for clarity
1
vote
4answers
98 views

How to subtract two times in shell

I nee to subtract two lines which is in the format of time in shell. Th time format is hh:mm:ss I used the code below to get the time. cat /var/log/kern.log |grep usb |tail -2| awk '{print $3}' The ...
3
votes
3answers
196 views

How can I delete all text between curly brackets in a multiline text file?

Example: This is { the multiline text file } that wants { to be changed } anyway. Should become: This is that wants anyway. I have found some similar threads in the forum, but they don't seem ...
2
votes
1answer
26 views

Calculate median value for a column and say if this value is “OK” or “ERROR” if falls within acceptable range

I would like to calculate the median value of a particular column in a .txt file, output that value and also say if that output value is ok. I can do the initial steps and output the median value, ...
2
votes
2answers
57 views

Parse several thousand lines of txt into lines and columns

The file at hand lists names, credentials, roles, and privileges for thousands of users. It was passed onto me as .xls but the lines and columns are not arranged in proper form. I have reformated ...
0
votes
0answers
23 views

Replacing code between two JavaScript comments

I need to set a value of JavaScript variable from bash. Variable lives in index.html file and I'd like to use unix SED command to do that. Inside index.html I have it lake this: <script> ...
-2
votes
0answers
48 views

How to merge two files using awk and based on specific pattern

I want to merge 2 files (report modifications) 1 of them is the reference file (report modifications from ) and the other is the target (report modifications to) . File 1 : int a; int foo() ...
0
votes
1answer
25 views

Need to compare the two files and print all the columns then missing display as NA, then join another file [duplicate]

File 1: 123, 234 345, 789 678, 900 File 2 123, abc 345, dnc output (all columns in file1 and missing value in file two as NA): 123,234,abc 345,789,dnc 678,900,NA Output file add with another ...
1
vote
3answers
44 views

using awk command to compare the two files and print all the columns then missing display as NA

File 1: 123, 234 345, 789 678, 900 File 2 123, abc 345, dnc Desired output (all columns in file1 and missing value in file two as NA): 123,234,abc 345,789,dnc 678,900,NA
2
votes
3answers
48 views

Removing lines from one file based on non-existence in another

I need to remove lines from fileA which has two columns if fileB (one column only) does not contain the string from column1 in fileA. Any linux command would help. Thanks. Example. Here is fileA: a ...
0
votes
1answer
13 views

add fields from different lines in one line

I'm trying to add the .data and .bss sections of the size -A output size -A mylib.so | grep "\.data" | awk '{print $2} this will print the value of the size of the .data section. I guess I could ...
8
votes
4answers
300 views

Remove duplicate lines from a file that contains a timestamp

This question/answer has some good solutions for deleting identical lines in a file, but won't work in my case since the otherwise duplicate lines have a timestamp. Is it possible to tell awk to ...
2
votes
2answers
88 views

How to capture the port number from this `lsof -F` output using awk (or something better)?

I've got output coming from lsof -F that looks like this: p7646 g7646 R8300 csocat u1000 Labe f3 au l tIPv4 G0x80002;0x0 d640391 o0t0 PTCP n*:51352 TST=LISTEN TQR=0 TQS=0 I am trying to capture the ...
0
votes
2answers
32 views

Delete consecutive lines if patterns match in each line

Good day, I was wondering how to delete to consecutive lines when this pattern is found: /^pt [a-z]\npy [0-9][0-9][0-9][0-9]\n// Expected Input pt a py 01234 pt b cd abc py 5678 Expected Output ...
0
votes
2answers
45 views

Grouping data and assigning alphabet and numeric values for new categories

I have 4 columns in my tab delimited data, I need to perform 2 steps. Group the data according to combinations of first 2 columns. Every time a new combination of values appear in the concatenated ...
1
vote
2answers
46 views

Print only the second field of the second line

I have to extract from the command hcitool dev only the mac address of the BT dongle. Output of hcitool dev is Devices: hci0 xx:xx:xx:xx:xx:xx I write this output on a file and try to get the ...
0
votes
5answers
45 views

Subset data with awk

I want to subset data with awk. Lets say I have this file called test: IP MAC Bandwidth etc etc 192.1.1.1 ff:ff:ff:ff 5.421M 192.1.2.3 ff:ff:ff:f3 5.120M 192.1.2.5 ff:ff:ff:f1 5.100M stuff I dont ...
1
vote
1answer
50 views

Multiply rows of a column in different chunk of data, add them together and get the average?

I have a chunk of data that looks like this: 924 777 777 777 8445.65349 P 9.44657 -0.840283 -0.394467 C 4.0555 -0.683104 -1.70917 P 7.5169 3.54172 2.77681 C 3.21536 2.67146 2.20242 P 4.33641 7.62012 ...
0
votes
2answers
52 views

How to list only filenames removing directories and subdirectories

How can I get the list of only filenames and remove directory and subdirectories using find, sed, awk, cut or any. For e.g : I get list of all media files using find as such: find /var/warehouse/* ...
2
votes
4answers
53 views

Awk remove field only if numeric

ls /var/log/packages/*SBo|cut -d / -f 5 return webcore-fonts-3.0-noarch-3_SBo winetricks-20120923-noarch-1_SBo wireshark-1.12.1-x86_64-1_SBo wxGTK-2.8.12-x86_64-3_SBo I want to substitute from ...
4
votes
4answers
343 views

How do I remove every number that's surrounded by <>

I have tried this sed -i '' 's/[0-9]*<>/g' But it didn't work. Example file: <Number1> </Number8> output: <Number> </Number>
4
votes
2answers
353 views

How do I populate end tags using sed, awk or any other command?

I have a txt file that looks like this: <sss>ss< or <firstword>anotherword< I want it to look like this <sss>ss</sss> or ...
0
votes
1answer
78 views

How to update files based on another file

I have 4 csv files and I want to update set of three based on values in the other one. The first 2 files (file_1 and file_2) contains names and their numerical ids. The third file (file_3) contains ...
1
vote
2answers
58 views

Extract and format data with `cut` and `awk`

I have the below file in txt: "Firstname","Lastname","Company" "Alex","Johnson","Alex Corp." "Baran","Atasoy","Gülerler Ltd." I would like to extract and format data by using awk and/or cut and ...
1
vote
1answer
35 views

Insert missing string in multiple ordered columns

I have a multi-column file that looks like this: 6 93129B 177 306.98 350.20 8.84 0.23 -0.79 0.22 0.46 m 7 8 192 269.08 413.35 9.29 0.15 -0.81 ...
2
votes
3answers
159 views

Using grep/awk/sed to sort and combine 2 files

Ok I am running into a problem. I am taking a wifi log that has MAC addresses listed in it. I want to find out what MAC vendors/manufactors for the devices that are connected to my router. I have 2 ...
2
votes
2answers
61 views

How to wrap the comma delimited values to the next line without disturbing the other fields using awk or ksh

I have the following input, in which some of the values are comma delimited. I would like decouple each value and print it in the new line. I tried many ways in awk, but no luck. If I have three ...
3
votes
2answers
147 views

Shell script to assign values from a lookup table is too slow

I'm trying to assign some variables from a lookup file with a shell script. I have something working but it seems unnecessary slow. Script: while read line do ...