awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

2
votes
2answers
493 views

Merging multiple lines based on column 1

I have a file like below.. abc, 12345 def, text and nos ghi, something else jkl, words and numbers abc, 56345 def, text and nos ghi, something else jkl, words and numbers abc, 15475 def, ...
3
votes
4answers
722 views

merge 2 rows based on the same column values

I have a file like below. 47196436 47723284 name1 1.77273 42672249 52856963 name2 1.06061 52856963 430695 name2 1.16667 55094959 380983 name3 1.55613 17926380 55584836 name4 1.02461 3213456 34211 ...
2
votes
3answers
125 views

awk - to add a column after matching the remaining columns

I have a file (file1.txt) with the contents as below. 8145216 3377090 1.5 1405541 53595498 1.53637 I need to generate all the possible combinations for the first 2 columns in the above file. I use ...
1
vote
1answer
42 views

linux if else fails for some IP's

i am using the following code with if then to mount. It works for some IP's, it does not work for some IP's especially for FAILURE statement. Here is the code is_mounted() { mount | grep -qohw ...
6
votes
4answers
3k views

How to extract logs between two time stamps

I want to extract all logs between two timestamps. Some lines may not have the timestamp, but I want those lines also. In short, I want every line that falls under two time stamps. My log structure ...
2
votes
2answers
319 views

How to do alphanumeric validation for a particular column (say $2) of a file using awk?

How do I perform alphanumeric validation for a particular column (say $2) of a file using awk? For example I've tried: awk -F, '{if ($10==(/[0-9a-zA-Z]/)) count+=1;} END {print count}' final.csv I ...
0
votes
2answers
129 views

Extracting with grep/sed a first pattern and then a second pattern that occured some lines before and related to the first

I'm trying to extract a specific line from a .htm file when a pattern occur ("Event 100" in my example below) but at the same time I need to retrieve also another pattern which is related to the first ...
1
vote
1answer
68 views

AWK - every thing in one line? [closed]

Hello can I make several changes in a file using just on line with awk? For example, transform this: awk < towhee_input5 -v k="$nstep" 'NR==18 {$0=k}{print}'> towhee_input6 awk < ...
3
votes
3answers
769 views

Merge 2 files based on all values of the first column of the first file

I need to merge below 2 files: file1: TABLES REF-IO HEAD-IO DIFF-IO test 200 500 -300 exam 2 3 -1 final 2 1 1 mail 4 2 2 TOTAL 208 506 -298 ...
1
vote
2answers
562 views

How to replace a custom date string in a text file

I've looked through several tutorials and a couple of StackOverflow posts, but I've not seen any examples using sed or awk which use dates. I have a text file that contains this as the first line, I ...
4
votes
7answers
562 views

How to split a filename list in 5GB sets?

I have a pretty big list of files which I like to group in smaller list of 5GB each for further processing. The file list is in the following form: "size" (from du) "tab" "absolute path", like this ...
3
votes
4answers
346 views

extract the characters before () using grep

I need to extract the characters before the character '('. My file structure is: rose(good)ook daisy()no lilly(bad)fine sunflower()nice I need output file as in the form of: rose ...
1
vote
3answers
70 views

awk want to total by hour and variable

I am able to run this and it works fine - note: field $1 is a time/date stamped field.. gawk -F ":" "{ print $1 }" /cygdrive/c/counting/ourlog | sort | uniq -c | sort -r 57339 2014-03-21 09 54290 ...
1
vote
1answer
69 views

Remove the string before the whitespace

I want to remove a word before the whitespace. My file structure is: ant tiger static rabbit lion dynamic monkey donkey I need the output as: ant tiger rabbit lion monkey donkey ...
3
votes
2answers
86 views

introduce line break at the same position based on another file

I have 3 input files all having the same number of lines. I have come up with a movie example to make the question look interesting. file1.txt -- Essentially contains the actor names. ...
1
vote
2answers
156 views

Union of two files and unique in new file

I have two files with few similar column one, two and third column with small variation and few other columns as follows. file 1: AT1 AT22 24 1 ATAGATA ATTATAT AT2 AT24 22 0 ATAGATA ATTATAT ...
5
votes
4answers
479 views

Reduce the over use of files in text processing

Currently, I have a file (Results.txt) which has 27k rows. The file content is as below. 45008657 | A. | | Long | 0 49210987 | A. | ...
3
votes
1answer
46 views

update line based on previous line

I have a issue since I'm not familiar with awk. I have csv file generated from sar -d output converted to csv style: 12:33:41 unix,restarts 12:35:00,lofi4096,0,0.0,0,0,0.0,0.0 ...
1
vote
3answers
1k views

Extract part of string using grep

I have 3 files in a directory MYO144064T MYO144064TA MYO144064TX Where the digits and 11th character will change. In my csh script, I want to extract the letters 'A' and 'X' probably using grep ...
3
votes
3answers
9k views

How to parse JSON with shell scripting in Linux?

I have a JSON output from which I need to extract a few parameters in Linux. This is the JSON output: { "OwnerId": "121456789127", "ReservationId": "r-48465168", "Groups": ...
3
votes
4answers
280 views

Split the input file with line breaks

I am trying to insert line breaks on a file based on the same names which happens to be the second field in my file. So, my input file is like below. 17412193|name1|organization ...
2
votes
2answers
766 views

How to find and replace a field column value in UNIX

I have file with the value say which is delimited with | 123-|aaa|bbb|123|123.0|123-|123.01-|-123.02|123.03-|aaa|bbb|123-|aaa-|-bbb|cc-cc|123.04-|aa123-|123.05- I need to pick the column values ...
3
votes
4answers
167 views

Identifying genes from a list of genes

I have a gene list file. Some thing like this SWT21 SSA1 NRP1 EFB1 TFC3 MDM10 I have another file which also contains the names of these genes in my list along with other ...
-2
votes
1answer
79 views

concatenate or remove multiple files of a directory

In general for file concatenation with the same extension, we use cat *.txt. But in case we are in a different directory and trying to concatenate another directory's files how to do that ? I have ...
1
vote
2answers
263 views

gawk: Summing floats formatted with comma separator

I'm trying to sum floats which use the comma as separator, and gawk does integer calculations. Here's the script: BEGIN { FS=";" OFS=";" CONVFMT = "%2.2f" } { print $1 print $2 print ...
4
votes
2answers
292 views

Merge fields in a file

I have a file with 7 columns, a GFF file having chromosomal regions.I want to collapse the rows where REGION ="exon" to only one row in the file.The row has to be collapsed on the basis of regions ...
5
votes
4answers
5k views

Pass shell variable as a /pattern/ to awk

Having the following in one of my shell functions: function _process () { awk -v l="$line" ' BEGIN {p=0} /'"$1"'/ {p=1} END{ if(p) print l >> "outfile.txt" } ' } , so when called as ...
2
votes
5answers
327 views

Print text between (and including) two delimiters on a line

What's the best way to print the text between (and including) the first pattern ("SELECT" or "SSELECT") and a second pattern ("[") using a command-line took (e.g. cut, sed, awk)? For example, given ...
1
vote
3answers
181 views

How can I delete the first dot before the first number?

How to remove the first dot before the first number? For example I have the following file more file Version …………2.465.76.8.332 I want to get only the version number - 2.465768.332
7
votes
4answers
2k views

BEGIN and END with the awk command

According to the awk manual, BEGIN and END are not used to match input, but rather to provide start-up and clean-up information to the awk script. Here is the example given: ls -l | \ awk 'BEGIN { ...
2
votes
5answers
164 views

identifying words in file 1 and matching them to second file and printing them

I have two files as follows File 1 has list of Ids ram Tarah taikah jhdfklj File 2 has id=ram;*between*hkjhdswklf kjsdklhglshglkhgklsdgkhdfg id=taikah;jhdjfkhg**kjdfkjksfdjk ...
3
votes
5answers
182 views

Merging Two Files columns in order

I want to merge two files. Files A.txt 001;abc;def;ghi;jkl;pqr 002;abc;def;ghi;jkl;pqr 003;abc;def;ghi;jkl;pqr 004;abc;def;ghi;jkl;pqr 005;abc;def;ghi;jkl;pqr . Second File B.txt 001;mno 002;mno ...
0
votes
1answer
62 views

How can I use gsub with a pattern? And how to tell it which file to awk?

I've read here that I can replace the values of all columns containing a particular text using awk -F'\t' -vOFS='\t' '{ gsub("CC", "C", $1) ; gsub("AA", "A", $1) ; print }' But where do I tell awk ...
3
votes
3answers
1k views

awk remove the rows based on another column value

I have a file like below which I have sorted based on the username field. UserID score UserName 1234 200 Jack 5678 150 Jill 8543 200 Jill 5678 100 John I am trying to remove ...
1
vote
3answers
458 views

Print the last-but-one field

How to print the word before the last word in line (with ksh or awk or sed or perl one liner) Example 1: echo one two three will print "two" Example 2: echo 1 2 3 4 5 6 will print "5"
2
votes
1answer
69 views

awk to split the input file into 2 files(one for duplicated values and one for unique values)

I have an input file like below. file.txt 14 2.692022 author1 0 14 2.57826 author2 1 229 2 author3 2 323 2 author4 3 514 2.0625 author5 4 I am trying to split the file with the duplicated values ...
2
votes
6answers
92 views

Printing specific section of a line when a trigger value is present

I'm parsing data in the following format: prop1=value1:prop2=value2:prop3=value3+prop1=value4:prop2=value5 parts of the string are delimited by + properties can appear in any order the desired ...
0
votes
1answer
250 views

Confused with awk! Getting: 'cannot open “” for output'

Hi I have 2 files in my working directory. One is a simple file given by: $ cat file2 START Unix Linux START Solaris Aix SCO And I have a 6000 line xml file called dblp_1_76007.xml. When I ...
2
votes
4answers
255 views

How to multiply a data file with another index data file?

I want your help in multiplying my data file with another index data file. Both data files have the same number of rows. But the data file have more than 1000 columns while the index data has only one ...
1
vote
2answers
92 views

What's the difference between print and head?

I know head gives you the beginning section of a file. But in this case, I'm trying to go through my source control and find every file that has a shebang line. And I'm failing to get results when I ...
3
votes
4answers
3k views

Compare two text files and find matching lines

I have two files A and B. A looks like this (4 to 6 lines): GAGA CAGA GGGT TATT file B is a really big file with thousands of lines, here is a short example: AAATGTCAAGAGACAGAAATGTCAAGAGGGT ...
1
vote
1answer
439 views

syntax error in conditional expression while executing the shell script?

I am working on a project in which I need to make a url call to one of my server from the bash shell script.. http://hostname.domain.com:8080/beat After hitting the above url, I will be getting the ...
3
votes
2answers
383 views

Only print rows that have 2 or more fields?

I've got a text file and I'm trying to figure out how to filter out any lines with data in only 1 field, leaving lines with 2 fields. I came across the following code: awk '$NF > 1' but I can't ...
0
votes
2answers
87 views

search for first pattern and copy the string inside the ( ) into the 3rd field in lines where the second pattern found

Here are my spec: first search for "#firstpattern(" copy the content inside the ( ) to the 3rd field inside the ( ) of all lines where the "secondpattern" found. remove the entire line with the ...
6
votes
2answers
1k views

Counting number of times each IP address appears in log file

I have a file in the format as follows: $ cat file.txt 27.33.65.2 27.33.65.2 58.161.137.7 121.50.198.5 184.173.187.1 184.173.187.1 184.173.187.1 What's the best way to parse the file file.txt into ...
2
votes
1answer
284 views

solaris awk problems with date and if statements

I have daily files that come in via FTP with incorrect dates in the first column of the file. I have figured out how to deduct one day to derive the correct date and print this to a new file. The main ...
1
vote
1answer
309 views

awk date identification from input file

I currently have daily files that come in via FTP with incorrect dates in the first column of the file. I have figured out how to deduct one day to derive the correct date and print this to a new ...
1
vote
1answer
126 views

replace a pattern found with a number plus 1 on the same line

I have a file where each line contains a keyword (HELLO) and one or more numerical fields designated A1, A2, etc. The value in the numerical field is enclosed in brackets after the field name and is ...
6
votes
3answers
2k views

How to subtract rows (lines) with AWK

I'm trying to figure out how I can use AWK to subtract lines. For example, imagine the input file is: 30 20 The output would be: 10 Now, as a test I am trying to calculate the "Used" memory ...
-3
votes
2answers
76 views

How to get a device details only using awk commands

Friends, I have fdisk -l output in one file called test1.txt. Now I want to get the device name only using awk. How to get a device name? I want output like this: /dev/sda1 /dev/sda2 /dev/sda3 ...