awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

0
votes
1answer
18 views

Awk split big file conditionally in multi output file

I have to split a 90 GB file in multiple file conditionally (the file is a multiple table file to split by single table). I have create a solution in ksh and awk that cycle on the table rules and ...
0
votes
1answer
33 views

How to extract particular fields from lines using shell commands?

File1: 27/07/2014 10.10.03 Breaker1 was closed by Mr. A.Basu Spectrum ui6dms 29/07/2014 04.50.53 Breaker1 was opened by Mr. D.Gupta Spectrum ui6dms and one note added 30/08/2014 03.20.01 EarthSwitch ...
1
vote
1answer
48 views

How to count the frequency of a word >20

Using exclusively AWK, I need to count the occurrence of a word only when greater than 20 characters. I've already found the way to count the frequency of a word, but I have some problems in setting ...
-3
votes
1answer
38 views

sed/awk - replace first occurance of a string with array of strings [on hold]

I want to find the first occurrence of a word and replace it by 3 strings using sed/awk. for example , I have the below entry in my /etc/hosts file 10.4.16.11 dispatch1 10.4.16.12 dispatch2 ...
4
votes
3answers
155 views

awk and newlines in the input data

I want to find files in a directory and identify by their mimetype, not by the extension of the files. I'am using this command to determine the mime type: % find . -type f -print0 | xargs -0 -I{} ...
1
vote
1answer
76 views

Impute data by majority value per group

I have instruments in col1, their variables in col2 and calls in col3. In this example, there are 17 INS and 3 M types. Potentially there will be 17 x 3 rows. But some are missing. INS1 M1 AA ...
-1
votes
0answers
25 views

unexpected newline or end of string [on hold]

I am trying to run the following command. This works fine with Ubuntu 12.04 but when I try running it on CentOS 6 I get the error awk: cmd. line:1: ^ unexpected ...
1
vote
2answers
60 views

Make awk and get columns

I have next issue: Look at the structure information: 0 PATRAC_1 False Normal Normal 3 Not configure We have seven columns, seven values, so, I need to get that values using awk, at ...
3
votes
2answers
306 views

Extract only chinese characters

I am working on a text written in Italian and Chinese and I need to extract only Chinese characters using AWK. How can I do this? I tried: [The range of Chinese Unicode chars is 4E00 thru 9FFF (344 ...
2
votes
1answer
62 views

Convert IP Address to Coordinates using database

I have a file which contains IP and I wanted to compare it to Coordinates using a local database (instead of API). I tried following a process and break down to pieces and created scripts referring ...
5
votes
1answer
58 views

awk Command Loop

awk '{print $1 ,": " $3}' Src.txt | column -t awk '{for(x=2;$x;++x) print $1, $x "\n"}' Src.txt | column -t These commands are working but formatting is not proper either I can only print column ...
2
votes
1answer
16 views

Extract values from block of data

I have the below block of data. Heap after GC invocations=31 (full 3): par new generation total 131008K, used 0K [0x00000000eac00000, 0x00000000f2c00000, 0x00000000f2c00000) eden space ...
1
vote
2answers
53 views

How do i search for certain sections of a file and then extract only relevant information within that section?

So I have a text file that is a server report and has about 1000 lines of information in it. I'm trying to write a script that can search the report for only certain pieces of information I'm looking ...
2
votes
2answers
64 views

Compare 2 Files if it falls between a range of numbers

I have 2 files, File 1: 4 7 2 File 2: 1 5 Alpha 6 10 Bravo 11 15 Charlie 16 21 Delta 22 30 Echo I want to compare every entry in file 1 if it lies between col 1 and col 2 of file 2 (should be ...
1
vote
3answers
190 views

Put big data of heterogenous byte offset into arrays by AWK

Assume the data consists of byte offset which is not fixed i.e. the distance of two subsequent file headers varies. The point of this thread is to go through each size of events separately in arrays. ...
3
votes
5answers
103 views

How to add column in the beginning of file using perl?

I want a Perl one-liner that checks whether the first fields of an input file is the file's name and, if it isn't, adds the file name as the first column on every line. Example written in shell : ...
8
votes
9answers
415 views

awk or sed to lowercase/uppercase only one character in string?

Is there a way how to uppercase/lowercase only one character in some string? Input Example: syslog_apr_24_30 syslog_mar_01_17 Desired output: syslog_Apr_24_30 syslog_Mar_01_17 Note please the ...
0
votes
2answers
43 views

Best way to iterate through the following awk command

I have the following awk statement awk '/^string$/{i=1;next} i && i++ <= 2' file This will search for string and then print 2 lines down. So as an example $ cat file string aaa bbb ccc ...
-3
votes
2answers
39 views

sed or awk to get a string

I have a shell script with string=<deploymentTargets xmi:type="appdeployment:ClusteredTarget" xmi:id="ClusteredTarget_143378365 7353" name="cluster1"/> I want the value between name=" and ...
1
vote
1answer
20 views

calculate numbers on end and merge to one line

So in a file i have lines like these: thisdata:thisother:1337 thisdata:thisother:1800 thisdata:thisother:1500 thisdata:thisother:46984 I would like to output: thisdata:thisother:51621 for all ...
-1
votes
1answer
17 views

Matching 2 different files by first column

So I have 2 different files. file1.txt is: abcdefjeiireiiioe:data:otherdata rijirjeiwrieeoooe:datamore:otherdatamore file2.txt is: abcdefjeiireiiioe:data:otherotherdata ...
0
votes
2answers
65 views

Using multiple awk commands within single line

Here is what am trying to do: My data is in sets of rows within same file (the variables vaa vbb are looping over it and am able to control it as required) What i need is to add an extra column before ...
1
vote
2answers
44 views

Using df and trying to awk a specific column

Among other problems with this script I am trying to make (this is the first unix/linux script I have ever made), I am trying to use df to get the % of disk space used and email it to the root. Our ...
7
votes
7answers
344 views

Removing numeric values in certain columns whilst keeping minus signs?

I put my hands up to being a complete beginner, so apologies if I'm doing this wrong. Say I have the following data frame that continues indefinitely horizontally and vertically with negative numbers ...
1
vote
1answer
31 views

Obtaining unique lines with matching strings [duplicate]

I have a JSON like text file of records with duplicate ids that look like: {"ID":"93" , "ST":[42,77,51]} {"ID":"21" , "ST":[43,4]} {"ID":"94" , "ST":[65,11,4]} {"ID":"93" , "ST":[42,77,51,29,63]} ...
1
vote
2answers
32 views

Concatenate multiple fields separately based on one (key) column

I have a tab-separated test data like this: A 1 2 B 3 4 B 5 6 C 7 8 I want to concatenate (aggregate) both column#2 and #3 based on duplicate values ...
2
votes
4answers
44 views

Sum of alternate values in a column using either sed or nawk

foo.txt: 1 10 11 2 20 22 3 30 32 4 40 42 5 50 52 6 60 62 7 70 72 8 80 82 9 90 92 10 100 110 Desired Out.txt : 1 10 11 2 20 22 3 ...
1
vote
2answers
141 views

Another awk(ward) question - mapping substrings with multi-keys

I have two files that look like this: mapping.txt: 100178 L08/2015 100180 L08/2015 103822 J03/2012 103823 A06/2012 103823 J02/2012 103823 J03/2012 ... input.txt: 100180011|225| ...
2
votes
3answers
68 views

Removing duplicates every X number of lines

Is it possible to remove only the AWD015F11 duplicate using Awk? I have a large file with one column. This happens every 24 rows. I have tried counting the rows in search for a pattern that I could ...
0
votes
1answer
29 views

Convert particular lines in a text file to bold [closed]

I have a text file with 100 lines in it. I want to convert particular lines, lets say line1, line6 , line27, line 48, line 86, line99 to bold. I am using solaris 5.10, with KSH. Is there a way I can ...
2
votes
2answers
36 views

Awk - keep word “example” and “EXAMPLE” with one input

I am attempting to to only keep customers that contain certain words. Here is an example < newname.txt awk 'BEGIN {FS=","} $7~/USA|United States/' > usa.txt The problem I am having is if they ...
0
votes
2answers
57 views

Bash to match two variables for time difference

I have a bash command which gives me following result (Fetch user account details from the mysql using command select creationdate from users; Name Account Creation Date Mobile User1 ...
6
votes
1answer
105 views

How can I extract/change lines in a text file whose data are separated into fields?

How can I manipulate field-based data from the commandline? For example How can I print only lines whose Nth field is foo? How can I print only lines whose Nth field isn't foo? How can I print only ...
3
votes
1answer
41 views

Using awk to process a mapping file with dynamic number of columns

I'm a complete newbie to awk and am wondering how to process information as following: mapping.txt: 80 001 002 81 011 012 013 014 82 021 022 ... input.txt: 81 103823044 80 103823054 81 ...
-1
votes
2answers
50 views

can i do this or how can i do this awk + sort?

cat file_outunique.out | awk '{ print $1, $2}' | sort $2 -u > ledernier.out I want to output $1 and $2 for the 1st iteration of $2 Do you have any suggestions about this? I've tried to change ...
3
votes
6answers
114 views

Case-insensitive count duplicate line, remove duplicate by choose case with highest duplicate

The duplicate is combination of different case text. I need to count number of duplicate (case-insensitive) and then I need to remove duplicate by choose case with highest duplicate. Below example: ...
4
votes
4answers
479 views

How to insert all lines from one file to another file?

File1: .tid.setnr := 1123 .tid.setnr := 3345 .tid.setnr := 5431 .tid.setnr := 89323 File2: .tid.info := 12 .tid.info := 3 .tid.info := 44 .tid.info := 60 Output file: .tid.info := 12 ...
1
vote
3answers
77 views

awk manipulation of fasta file

I have a file like this >chr1 ACGTGGC TGCCGTT ATCCTTG >chr2 ACTTTTA CTCATAA I want to convert the seq into 1 string. This should be the output: >chr1 ACGTGGCTGCCGTTATCCTTG >chr2 ...
3
votes
1answer
31 views

Match pattern, do calculation and print only 3 or 4 floating values max

I would like to convert below to GB and print only 3 floating values v copvol2 fsgen ENABLED 204800 - ACTIVE - - v copvol3 fsgen ENABLED 204800 - ...
0
votes
2answers
23 views

Unable to perform a simple multiplication in script

I am havingf trouble performing a simple multiplication in my script. while read A B C do tmp=$A\*$C/100 echo $tmp >> out1.txt done < foo.txt foo.txt: 13721725 99 100 ...
3
votes
1answer
34 views

Sending signals to long-running processes (AWK, DD)

I'm familiar that DD command doesn't produce output while running. However, if I send a USR1 signal to it, it can print current state. Can I do the same with AWK? Prints its variables, register ...
7
votes
6answers
180 views

How to efficiently split up a large text file wihout splitting multiline records?

I have a big text file (~50Gb when gz'ed). The file contains 4*N lines or N records; that is every record consists of 4 lines. I would like to split this file into 4 smaller files each sized roughly ...
2
votes
1answer
23 views

Unset all ENV variables matching _PROXY

I'm trying to unset all environment variables that match _PROXY: env | grep -i _proxy | column -t -s '=' | awk '{ print $1 }' | grep -iv 'no_proxy' | xargs -0 -I variable unset variable but it's ...
2
votes
4answers
70 views

Change text in tag, but only tag is contained within a certain XML block

Using Git Bash, I'm trying to conditionally replace what's in the yrot tag in hundreds of files, but only if it belongs with a part name tag that pertains to wheel. // YES, change <part ...
0
votes
2answers
41 views

How to find and replace only a particular word (which is at the end of the file) using shell command?

File1: Dms 01.01.2015 feeder1 6kv close at 04:30 Updated Dms 01.01.2015 feeder1 6kv open at 06:20 Updated Dms 04.02.2015 feeder10 6kv close at 17:23 Updated Dms 04.02.2015 feeder12 6kv ...
1
vote
1answer
54 views

How to write data one by one from tmp files to the final output file using for loop?

I have two tmp files named tmp1 and tmp2 which contains some lines. tmp1 file, 1c\ datafile no. 23 2c\ datafile is ok tmp2 file, 3c\ datafile no. 24 4c\ datafile is ok I have a ...
4
votes
2answers
161 views

how to extract parts of file into separate files in linux using awk or grep or sed commands

I have text file as shown below: Input file1.txt ....................... cha21 1 3 5 nar cha21 2 3 6 piy cha23 2 3 5 ram cha23 3 ...
2
votes
5answers
113 views

Adjust gap between 2 columns to make them look straight

file1.txt: hi wonderful amazing sorry superman superhumanwith loss file2.txt : 1 2 3 4 5 6 7 When i try to combine using paste -d" " file1.txt file2.txt > actualout.txt actualout.txt : hi 1 ...
1
vote
2answers
26 views

Addition of values

foo.txt: Lillypaul_sg 204803 204803 204803 Ammy_sg 3 ramaswamy_sg 3 33 tommy_sg 3 3 137374 100 Desired Out.txt : Lillypaul_sg 614409 Ammy_sg 3 ramaswamy_sg 36 tommy_sg 137480 The output is a sum ...
2
votes
3answers
30 views

Truncating the precision of a date

I'm working with a bash script which polls build progress from several different systems. If a build is complete, it stats a log file to find out the last time it was edited, then feeds that time into ...