awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

6
votes
4answers
142 views

How to insert lines with numbers from X to Y into another file after line Z?

I created two files: echo -e "1\n2\n3\n4\n5" > 123.txt echo -e "a\nb\nc\nd\ne" > abc.txt I want to get the file 123.txt with the following contents: 1 2 3 b c d 4 5 So in other words ...
5
votes
2answers
115 views

Is there a way to make this one-liner faster?

Context I have a directory of thousands of zip files that are dated in the form YYYYMMDD_hhmmss.zip and each about 300K. Within each zip file is about 400 xml files each about 3K. The problem I ...
5
votes
2answers
132 views

Sum in block with AWK (restart the sum when the pattern changes)

I have a file like this: A 100 A 200 A 300 #sum=600 B 400 B 500 #sum=900 A 600 A 700 A 800 #sum=2100 I would like the output to be: A 600 B 900 A 2100 C sum_of_C D sum_of_D I can do that with ...
2
votes
2answers
2k views

Diff, show modified line from right file only

I have file a and b and I would like to output lines of b that changed since it was cloned from a. Just the modified lines, no surrounding context, no diff offset marks. How can I do that using shell ...
0
votes
1answer
37 views

change date format and store in variable using awk

I have date in format YYYY-MM-DD in variable 'a'. I want to convert it in format YYYYMMDD and store in variable 'b'. The variable is in an awk variable. How can I do the same ?
1
vote
3answers
442 views

awk a specific column with unexpected line break

I can't seem to find a solution for this and it's driving me crazy. I know I can use awk to print a column(s). I'm having trouble printing a specific column though because of the way my file system is ...
6
votes
5answers
227 views

Strategy to extract movies's name from this ununiform dataset?

I am working on a movie database problem to improve regular expressions, this is the problem I'm running into. My dataset looks like this: Movie Name (variable space and tabs) year Movie1(can ...
2
votes
2answers
46 views

How can I compare two files based on the value of the first column?

I want to compare two files based on the value of their first column: file1 aaa 23 3 bbb 56 6 ccc 45 2 file2 bbb m a aaa a m ccc m m The output file should be: aaa a m bbb m a ccc m m ...
3
votes
1answer
57 views

I can't print blank lines when reading values from a text file using awk in a bash script

awk is behaving very strangely in my script. I am trying to read values from a text file (see file.txt below). awk is ignoring blank spaces in my script, but when I test the awk statement on its own ...
1
vote
1answer
38 views

Retrieving fasta sequences using bed file information from locally installed file

I have a .bed file containing around 30000 rows for which I have the sequences retrieved using fetch-sequences module of the rsat tool (http://rsat.ulb.ac.be/rsat/help.fetch-sequences.html#usage). ...
0
votes
1answer
27 views

Match first fields of two tab separated files and print matching values

I have two files. Match the 1st column of first file to 1st column of second file and print entire line of the second file. Second file is a <tab> separated file. I have tried many awk one ...
3
votes
2answers
78 views

Find and replace different columns using awk

I try to make a change in a file: Find a certain value and substitute it with another value in a different column using awk. Input (UiO-66Zr-EH.mof): unit ntype qqatom 1 'Zr1' 0.0d0 vibration ...
7
votes
4answers
172 views

Sort fields inline

I'm trying to sort within a line of input over an unknown number of fields: Input: ab bc bc ab cd ef bc bc cd ef cd bc ab ef ab bc cd gh Output: ab bc ab bc bc cd ef bc cd ef ab cb cd ab bc cd ...
3
votes
3answers
204 views

calculate sum of squares using shell script in perl/awk

I have 2 files as below. file1 0.34 0.27 0.32 file2 0.15 0.21 0.15 Now, I would like to calculate the sum of squares between each column. For example, [(0.34 - 0.15)^2 + (0.27 - 0.21)^2 + ...
3
votes
1answer
151 views

Awk print problem

I have a code that takes values from a log file and populates them into an excel spreadsheet: function ltrim(s) { sub(/^[ \t]+/, "", s); return s } function rtrim(s) { sub(/[ \t]+$/, "", s); return s ...
1
vote
3answers
29 views

How to truncate only given column length?

INPUT: a@notebook:~$ cat in.csv 'XYZ843141'^'ASDFSAFXYVFSHGDSDg sdGDS dsGDSgfa assfd faSDFAS saDFSAFD adFSA343fa sdfSADF'^'BAAAR'^'YYY'^'..... and so on, further columns ...
1
vote
1answer
25 views

awk getline not working as expected on first loop iteration

Consider this simple awk calculator program and its output $ awk '{while (getline) {n=0;for(i=NF;i>0;i--)n+=$i; print "sum: " n;}}' 1 1 2 2 sum: 4 3 3 sum: 6 Note how the first input line 1 1 ...
1
vote
2answers
121 views

Extract Values from simple html file via grep/awk

I have another grep/awk/sed issue within the attached HTML code below. Given is a simple HTML file with table inside. The HTML is generated by a smart meter (household energy meter). Inside the html ...
2
votes
3answers
55 views

replace a character from a string that is in between first and second search using sed or awk

I need to replace _ (underscore) with ? (question mark) in a file containing email addresses. The file looks like below: EFT_020034-E015133 20140624 /ACC/EMAIL+test_1@mysite.com SHR ...
2
votes
2answers
56 views

new pattern required from an input file using sed or awk

I have data as below: MGW: VMG110 836-16 836-18 836-19 336-20 836-23 MGW: VMG120 3802-1 3802-2 3802-3 3456-1 3456-4 Required output is: VMG110:836-16&-18&&-20&-23 ...
5
votes
2answers
62 views

Grepping for a block of text with parts that can be optional

I have multiple entries that describes an event in a very large log file, say A.log. I would like to do two things with the event entries in the log file: Count the number of occurrences of each ...
1
vote
1answer
24 views

How to echo from specific line of a file to another specific line [duplicate]

Assume I have a file with about 10000 lines. How can I print 100 lines, starting from line 1200 to line 1300?
2
votes
3answers
45 views

Get the last minute dated lines from list

If I have an input file with a list of users and their dates they were last signed in, how can I select with a sort | awk function the last minute lines? The dates are written in a +%H:%M:%S format. ...
4
votes
1answer
63 views

Is getline in awk command safe to use?

I got weird comments on posting an awk answer on SO which uses the getline function. Here is the link to that answer. After posting my answer, a user came up with the below comment,(I don't criticize ...
3
votes
2answers
51 views

awk: Blank line with date

I use the following code to write the online users' process numbers and the date they were checked: w | tail -n +3 | awk ' { "date +%H:%M:%S" | getline tim} {if ($1 != "") ...
0
votes
1answer
52 views

Search a string between two timestamps starting from bottom of file

I was trying to find a string Cannot proceed: the cube has no data in huge test.txt file only between timestamps yesterday 22:30 pm to today 00:30 am. Script: tac test.txt | awk -v today=$(date ...
3
votes
2answers
133 views

Search a string with last two characters as numeric

Suppose I have a file like this: adasfddfd09 dsassd90897 323sdsdsdsd sdddsdf56 dfdf45fdfdf I want to search only those strings which have last two characters numeric with sed, grep and awk.
4
votes
3answers
156 views

Extracting text using sed does not work as expected [duplicate]

I am just starting out using sed and I intend to use it to extract IP addresses from ping output. Here's what I am trying to achieve exactly: input text: ytmti (192.188.2.3) jjggy desired ...
1
vote
4answers
51 views

Printing a new line using awk

I have a load of files in which I need to take out specific rows and then put the data I've taken out into a spreadsheet. An example would be my file shows: Name: w Age: x Height: y Weight: z I ...
4
votes
4answers
211 views

Redirecting Output of print to multiples files in awk

I want to print same lines to 2 different files. What I am doing right now is: print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file1.txt"; print ID[i]" "Value[i]" "TS1[i]" "TS2[i] > "file2.txt"; ...
2
votes
4answers
120 views

How to initialize an array using awk and bash?

I am trying to store values of the first line of a text file into an array. Here is what I have so far: arr_values=() awk ' NR==1 { for (i=0; i<=NF; i++) ...
0
votes
1answer
41 views

using awk for subsetting fastq file based on length of sequence

I have a fastq file. I will explain what it is. It is something like this @SRR1024120.7 DBRHHJN1:259:D0PM7ACXX:1:1101:1386:1189 length=100 GATACAGGATGCCTGGGTCTAGGCTGTGTGACCTTGGGCCAGTTCCTCTC ...
4
votes
3answers
154 views

UNIX paste columns and insert zero's for all missing values

I would like to merge specific columns from two txt files containing varying number of rows, but same number of columns (as shown below): file1: xyz desc1 12 uvw desc2 55 pqr desc3 ...
3
votes
3answers
156 views

Replace multiple strings in a single pass

I'm looking for a way to replace placeholder strings in a template file with concrete values, with common Unix tools (bash, sed, awk, maybe perl). It is important that the replacement is done in a ...
8
votes
5answers
8k views

Remove comma between the quotes only in a comma delimited file

I have a input file delimited with commas (,). There are some fields enclosed in double quotes that are having a comma in them. Here is the sample row 123,"ABC, DEV 23",345,534.202,NAME I need to ...
1
vote
4answers
3k views

sed/awk/perl - delete spaces between two strings/words + solaris 10

please advice how to delete the spaces between "=" to the value/string ( after the "=" character) , should be implemented by awk or sed or perl command and command must to fit for solaris OS , ( ...
0
votes
1answer
43 views

I am having some technical issue with awk code [closed]

This is related to one of my previous question: Comparing text files using Bash and AWK file1.txt Name Col1 Col2 Col3 ----------------------- row1 1 4 7 row2 2 5 8 ...
1
vote
3answers
55 views

multiplication of every line

I would like to calculate a value on every line of a file such as this: 1 1441792,11534336 2 1048576,8388608 3 1441792,1153433 4 1966080,15728640 5 393216,3145728 6 2621440,20971520 7 ...
2
votes
3answers
253 views

Comparing text files using Bash and AWK

I have two text files, and I want to compare their corresponding values according to their rows and columns. By comparing, I mean to check if the values are equal and echo if the values are the same ...
7
votes
4answers
170 views

Groupwise uniq command?

I am searching for a command to get from a file in this format: hello 32 hello 67 hi 2 ho 1212 ho 1390 ho 3000 To this format (deduplicate by taking the last row of a "group"): hello ...
0
votes
2answers
44 views

awk to Read between two Particular Strings and discard the rest

I'm using awk to just read text between two strings "Check" and "Result". I've used many variations of it found on the internet and still can't get the favored result. I've tried: awk ...
2
votes
3answers
68 views

Use awk results as parameters to another command

I'm extracting rows from a set of text files with awk. The files look like this: 1000 1 75 1000 2 76 1001 1 76 1001 2 80 I'm searching several directories of these with this ...
0
votes
1answer
28 views

Creating a file from another file on the basis of a numeric value in each line

We will generate a csv file with below values. yp1234,577,1,3 yp5678,570,3,5 yp9012,132,8,9 I need to extract data and create files based on second column. If it's value greater than 577 then the ...
1
vote
1answer
44 views

Awk that I am stuck on

I have a project I am working on for work. The script I am building initially takes two arguments: one is an unique Identifier and the other is for a log file (both strings). Now I have the first part ...
3
votes
3answers
153 views

awk filtering live capture with file as pattern list

What I'm trying to do is quite simple. I'm generating output from tshark and redirecting it to awk with a pipe |. Since tshark gets live data, I want that awk search in every output for a pattern ...
6
votes
8answers
569 views

Grep rest of line…after match

I have a file containing only two lines, with the following structure: $ cat /tmp/pwpower.log 000D6F0000D34227, -114.10 000D6F0001A405C4, -130.09 The values are power values of my solar plant. ...
3
votes
3answers
156 views

AWK: Passing shell variables to awk

I am trying to pass a variable number of arguments from the shell script to a pattern recognition subset of a table. Here is my attempt so far: The file 'infile': ID,GROUP 1,GROUP2 ...
1
vote
3answers
144 views

awk for loop for each line in a file

I'm a newbie to bash scripting. I need help trying to figure out how to get loop through a text file and for every line take the fields of name , project #, and email, and replace them in a email ...
1
vote
1answer
30 views

Move data row(s) to single column while retaining row header(s)

I have reports that are generated in the following tab-delimited format: UNIT TC CC PC TCP FTX FRX HOUSE 55 65 75 85 95 105 CAR 100 200 300 400 500 600 H2 ...
3
votes
2answers
92 views

How can I derive an IPv6 address from a MAC address?

We know that we can use the MAC address to create an interface identifier, e.g. for a link-local IPv6 address which should be unique in the Network. The image shows the way to do this: My ...