awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

-1
votes
1answer
76 views

how to extract lines between 2 patterns, and save them to different files

hej all, Let' say that we have this file: rrows 35 cols 51 spin 2 %band5 efermi=0.012538 0.004352 0.004317 0.004213 0.004047 0.003828 0.003570 0.003291 0.003011 0.002754 0.002544 0.002408 ...
1
vote
3answers
53 views

compare multiple files(more than two) with two different columns

I have four different files: file1, file2, file3, file4. Each file has 2 different columns separated by tab. I want to match first column of file1 (as reference) with the first column of second file, ...
4
votes
2answers
3k views

Remove entire row in a file if first column is repeated

I have a file containing two columns and 10 million rows. The first column contains many repeated values, but there is a distinct value in column 2. I want to remove the repeated rows and want to keep ...
4
votes
2answers
1k views

Merging columns from two separate files

How to create a new file merging selective columns from two separate files using awk? Without messing up the elements orders of BOTH files. Exemple: File 3 may contain column 1,2,3 from File 1 and ...
1
vote
1answer
203 views

How to copy rows of different files and paste column-wise?

I want to create a new file by copying rows of each file and pasting columnwise according to their file name orders. Here is the sample input and required output files below. file1.txt 1234 5678 ...
2
votes
4answers
241 views

Replace values in column that contain numbers

I have a file that looks as so: chrom start stop strand isoform mu_codon mut_codon2 more_info chr22 43089055 43089055 - NM_017436 C 300 903delC chr22 43089715 43089717 ...
3
votes
4answers
202 views

How to compare two dates without using Stat?

I'm trying to compare the last date modified of a file to today's date. If the modification date is before today's date then my script will do one thing, if the date matches todays date i need it to ...
0
votes
1answer
54 views

Python output in XML format

I have a python script which compare IP list with IP Database and gives output of matching criteria. (I got the script from this forum). convert.py #!/usr/bin/python import socket,struct ...
4
votes
4answers
2k views

merge 2 rows based on the same column values

I have a file like below. 47196436 47723284 name1 1.77273 42672249 52856963 name2 1.06061 52856963 430695 name2 1.16667 55094959 380983 name3 1.55613 17926380 55584836 name4 1.02461 3213456 34211 ...
0
votes
3answers
60 views

How to sort/reformat text information in a file?

I have a file (called customers) with information that needs to be sorted; example: qweasd,asdqwd@asd.org,256.62394383 qweasdczasd,asdghjghqwd@assssd.org,256.62394383 ...
3
votes
5answers
120 views

parsing a huge file with awk and extracting a subset

I have a huge file which looks like this chr10 98072 1 chr10 98073 1 chr10 98074 1 chr10 98075 2 chr10 98076 2 chr10 98077 3 chr10 98078 5 chr10 98079 5 chr11 98080 ...
1
vote
3answers
128 views

Remove characters in a specific column before a specific character

I have a file that looks like so: chr22 43089055 43089055 - NM_017436 C 300 903delC chr22 43089715-43089717 43089715-43089717 - NM_017436 CTT 79 I want to remove all the ...
0
votes
3answers
82 views

Bash to match two variables for time difference

I have a bash command which gives me following result (Fetch user account details from the mysql using command select creationdate from users; Name Account Creation Date Mobile User1 ...
0
votes
1answer
152 views

kill processes in one line with kill, awk, ps, and grep

I have a few processes that spring up and I am able to print a line of the pgid's that I would like to enter into a kill command. here is what I have: sudo ps o pgid,args | grep mininet: | sudo awk ...
3
votes
2answers
353 views

Extract only chinese characters

I am working on a text written in Italian and Chinese and I need to extract only Chinese characters using AWK. How can I do this? I tried: [The range of Chinese Unicode chars is 4E00 thru 9FFF (344 ...
0
votes
1answer
27 views

Awk split big file conditionally in multi output file

I have to split a 90 GB file in multiple file conditionally (the file is a multiple table file to split by single table). I have create a solution in ksh and awk that cycle on the table rules and ...
1
vote
3answers
243 views

Put big data of heterogenous byte offset into arrays by AWK

Assume the data consists of byte offset which is not fixed i.e. the distance of two subsequent file headers varies. The point of this thread is to go through each size of events separately in arrays. ...
2
votes
1answer
89 views

Convert IP Address to Coordinates using database

I have a file which contains IP and I wanted to compare it to Coordinates using a local database (instead of API). I tried following a process and break down to pieces and created scripts referring ...
1
vote
1answer
68 views

Extracting trace information using gawk

I am new to awk programming and am using gawk to extract trace information from a mrt file in order to use it further for analysis. I have successfully been able to extract trace information from a ...
0
votes
1answer
54 views

How to extract particular fields from lines using shell commands?

File1: 27/07/2014 10.10.03 Breaker1 was closed by Mr. A.Basu Spectrum ui6dms 29/07/2014 04.50.53 Breaker1 was opened by Mr. D.Gupta Spectrum ui6dms and one note added 30/08/2014 03.20.01 EarthSwitch ...
2
votes
2answers
78 views

How to exclude lines that don't include a string from the output of an awk script?

Completely new to scripting in general, and have been working on a script to go through our logs which contain entries for a variety of different things. What I'm interested in, is the entries that ...
1
vote
1answer
59 views

How to count the frequency of a word >20

Using exclusively AWK, I need to count the occurrence of a word only when greater than 20 characters. I've already found the way to count the frequency of a word, but I have some problems in setting ...
5
votes
1answer
77 views

awk Command Loop

awk '{print $1 ,": " $3}' Src.txt | column -t awk '{for(x=2;$x;++x) print $1, $x "\n"}' Src.txt | column -t These commands are working but formatting is not proper either I can only print column ...
3
votes
3answers
173 views

awk and newlines in the input data

I want to find files in a directory and identify by their mimetype, not by the extension of the files. I'am using this command to determine the mime type: % find . -type f -print0 | xargs -0 -I{} ...
2
votes
6answers
778 views

Adding a character to every other text line

I have a huge (ca. 20G) text file which contains millions of passages (a.k.a. paragraphs) with headers. Headers and paragraphs are always one line each, e.g., Sunshine This is a sunny day. ...
1
vote
1answer
82 views

Impute data by majority value per group

I have instruments in col1, their variables in col2 and calls in col3. In this example, there are 17 INS and 3 M types. Potentially there will be 17 x 3 rows. But some are missing. INS1 M1 AA ...
1
vote
2answers
70 views

Make awk and get columns

I have next issue: Look at the structure information: 0 PATRAC_1 False Normal Normal 3 Not configure We have seven columns, seven values, so, I need to get that values using awk, at ...
2
votes
1answer
27 views

Extract values from block of data

I have the below block of data. Heap after GC invocations=31 (full 3): par new generation total 131008K, used 0K [0x00000000eac00000, 0x00000000f2c00000, 0x00000000f2c00000) eden space ...
3
votes
2answers
85 views

Compare 2 Files if it falls between a range of numbers

I have 2 files, File 1: 4 7 2 File 2: 1 5 Alpha 6 10 Bravo 11 15 Charlie 16 21 Delta 22 30 Echo I want to compare every entry in file 1 if it lies between col 1 and col 2 of file 2 (should be ...
1
vote
2answers
153 views

Another awk(ward) question - mapping substrings with multi-keys

I have two files that look like this: mapping.txt: 100178 L08/2015 100180 L08/2015 103822 J03/2012 103823 A06/2012 103823 J02/2012 103823 J03/2012 ... input.txt: 100180011|225| ...
1
vote
2answers
61 views

How do i search for certain sections of a file and then extract only relevant information within that section?

So I have a text file that is a server report and has about 1000 lines of information in it. I'm trying to write a script that can search the report for only certain pieces of information I'm looking ...
3
votes
3answers
103 views

Single record of a file getting splitted over multiple lines

We are using flat text files to store in teradata table after transformations from Informatica. The file contains 14 fields (~ separated). But as the records are getting scattered over multiple lines ...
2
votes
3answers
73 views

Removing duplicates every X number of lines

Is it possible to remove only the AWD015F11 duplicate using Awk? I have a large file with one column. This happens every 24 rows. I have tried counting the rows in search for a pattern that I could ...
8
votes
9answers
607 views

awk or sed to lowercase/uppercase only one character in string?

Is there a way how to uppercase/lowercase only one character in some string? Input Example: syslog_apr_24_30 syslog_mar_01_17 Desired output: syslog_Apr_24_30 syslog_Mar_01_17 Note please the ...
3
votes
5answers
156 views

How to add column in the beginning of file using perl?

I want a Perl one-liner that checks whether the first fields of an input file is the file's name and, if it isn't, adds the file name as the first column on every line. Example written in shell : ...
9
votes
4answers
6k views

Concatenate lines by first column by awk or sed

How can I use awk in the following situation? I want to concatenate lines that start with the same column. Only the first column is kept after the join (in this case aaa, www, hhh). The file may be ...
6
votes
1answer
115 views

How can I extract/change lines in a text file whose data are separated into fields?

How can I manipulate field-based data from the commandline? For example How can I print only lines whose Nth field is foo? How can I print only lines whose Nth field isn't foo? How can I print only ...
0
votes
2answers
343 views

Compare two csv files and append value using awk [closed]

I have two files as follows: file1.csv +------------+----------+--------+---------+ | Account_ID | Asset_ID | LOT_ID | FLAG_F1 | +------------+----------+--------+---------+ | 10000 | 20000 | ...
1
vote
2answers
57 views

Using df and trying to awk a specific column

Among other problems with this script I am trying to make (this is the first unix/linux script I have ever made), I am trying to use df to get the % of disk space used and email it to the root. Our ...
7
votes
7answers
402 views

Removing numeric values in certain columns whilst keeping minus signs?

I put my hands up to being a complete beginner, so apologies if I'm doing this wrong. Say I have the following data frame that continues indefinitely horizontally and vertically with negative numbers ...
0
votes
2answers
54 views

Best way to iterate through the following awk command

I have the following awk statement awk '/^string$/{i=1;next} i && i++ <= 2' file This will search for string and then print 2 lines down. So as an example $ cat file string aaa bbb ccc ...
-3
votes
2answers
43 views

sed or awk to get a string

I have a shell script with string=<deploymentTargets xmi:type="appdeployment:ClusteredTarget" xmi:id="ClusteredTarget_143378365 7353" name="cluster1"/> I want the value between name=" and ...
0
votes
2answers
119 views

Using multiple awk commands within single line

Here is what am trying to do: My data is in sets of rows within same file (the variables vaa vbb are looping over it and am able to control it as required) What i need is to add an extra column before ...
1
vote
1answer
23 views

calculate numbers on end and merge to one line

So in a file i have lines like these: thisdata:thisother:1337 thisdata:thisother:1800 thisdata:thisother:1500 thisdata:thisother:46984 I would like to output: thisdata:thisother:51621 for all ...
-1
votes
1answer
19 views

Matching 2 different files by first column

So I have 2 different files. file1.txt is: abcdefjeiireiiioe:data:otherdata rijirjeiwrieeoooe:datamore:otherdatamore file2.txt is: abcdefjeiireiiioe:data:otherotherdata ...
1
vote
1answer
39 views

Obtaining unique lines with matching strings [duplicate]

I have a JSON like text file of records with duplicate ids that look like: {"ID":"93" , "ST":[42,77,51]} {"ID":"21" , "ST":[43,4]} {"ID":"94" , "ST":[65,11,4]} {"ID":"93" , "ST":[42,77,51,29,63]} ...
2
votes
4answers
68 views

Sum of alternate values in a column using either sed or nawk

foo.txt: 1 10 11 2 20 22 3 30 32 4 40 42 5 50 52 6 60 62 7 70 72 8 80 82 9 90 92 10 100 110 Desired Out.txt : 1 10 11 2 20 22 3 ...
1
vote
2answers
58 views

Concatenate multiple fields separately based on one (key) column

I have a tab-separated test data like this: A 1 2 B 3 4 B 5 6 C 7 8 I want to concatenate (aggregate) both column#2 and #3 based on duplicate values ...
0
votes
1answer
60 views

Convert particular lines in a text file to bold [closed]

I have a text file with 100 lines in it. I want to convert particular lines, lets say line1, line6 , line27, line 48, line 86, line99 to bold. I am using solaris 5.10, with KSH. Is there a way I can ...
-1
votes
2answers
51 views

can i do this or how can i do this awk + sort?

cat file_outunique.out | awk '{ print $1, $2}' | sort $2 -u > ledernier.out I want to output $1 and $2 for the 1st iteration of $2 Do you have any suggestions about this? I've tried to change ...