awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

2
votes
1answer
27 views

Help with averaging specific columns awk

I need to process some data for my dissertation and I need to do it though Linux as I have > 2 million points. I've currently gotten my selected data it in a file that looks something like this using ...
-1
votes
2answers
27 views

putting output into sql table

I am trying to convert the output of ps awux into 11 fields with the headers eliminated and then importing that output into an existing table in SQL.
1
vote
1answer
48 views

Combine range of columns with the same key

Input file tab separated: A |B | C | KEY | D| E| F |G | H | I | J | k | ----|----|----|------|---|---|-----|------|---|---|-----|-----| word|word|word| one | 8| ...
2
votes
2answers
49 views

Replacing matching text after two matching lines

I have a YAML file which includes a stanza looks this: admin::common::passwords: alice: password: '$6$oTQhLvN/4VFJPscD$8LYwUMSFi' bob: password: '$6$JKOtLF0wHeZfIskt$W/M5.ugDS' If the ...
1
vote
3answers
252 views

How to extract XML from a text file

I have a text file as below. 1001=<?xml ns=dkfj><home><taga>dkjf</taga></home>1002=<?xml ns=dkfj><home1><taga>dkjf</taga></home1> I need ...
1
vote
3answers
49 views

Compare two files column values using awk

Here is the content of two files : Judi # cat File1 judi /export/home 76 judi /usr 83 judi # cat File2 judi /export/home 79 judi /usr 82 If column 3 of File2 is greater that column 3 ...
1
vote
1answer
41 views

Unable to run awk program

I have created a simple awk program file in /media/ravan/data/program The code: #! /usr/bin/awk -f BEGIN { print "Don’t Panic bro!" } While executing in terminal: ...
0
votes
1answer
92 views

print column value based on column name match in awk/sed [duplicate]

In my shell script after executing a few commands, I'm getting the output below: Name English Maths science Social studies ------ -------- -------- -------- --------------- James ...
3
votes
2answers
66 views

Remove headers and contents from a flat file if they are below a specific line count

I've a flat file containing about 10 million lines: query ID1 content1 content2 query ID2 content3 content4 ... content21 query ID3 content22 ...
0
votes
1answer
43 views

Split files using awk and generate the results in another directory

I am using a shell script where I split a file according to the values of a certain column (using awk). My script is in a folder X, though, and I need the resulting files to be created in a folder Y. ...
1
vote
1answer
30 views

display users that logged in since a date using last

I am trying to display the user name of the accounts that have logged in during the past month using the last command. At the moment I have this last | awk '{print $1, $4 ,$5 ,$6}' | grep -B 10 Jul ...
1
vote
2answers
89 views

Why is a2p (Awk to Perl translator) installed by default? Why would I want to convert Awk to Perl? [closed]

I am surprised to see the a2p utility installed by default in my Linux distribution. a2p is a command line utility that converts an Awk program from standard input to a perl program that it outputs ...
1
vote
1answer
61 views

Compare 2 Columns in 2 different files

I need to subtract second and third columns between 2 files based on the first column and do not show rows with diff_column_2(DIFF-IO)=diff_column_3(DIFF-SELECT)=0. Noting that the order can be random ...
1
vote
2answers
57 views

Reordering a file, based on a second file

I have a file that is out of order.I would like to reorder it by using a second file. File 1 1 ...
1
vote
2answers
123 views

What's the difference between print and printf in awk?

In awk what is the difference between print and printf? Is there any restrictions to use only print instead of printf and vice-versa
2
votes
3answers
127 views

How can I delete everything between two markers in a file?

I have a text in a text file, where I want everything that is between strings like \{{[} and {]}\} to be deleted - including these strings themselves. These two string can lie on different lines as ...
2
votes
1answer
30 views

need help with awk merging rows by sequence from multiple files into one file

have alot of files and need to merge strings between themself by groups of three files. each file ~100-500 rows and result file ~100-200mb seems like awk or sed can help me with paste or without it or ...
2
votes
4answers
568 views

Compare awk vs. grep

I have run below two commands for a very large files grep -E 'string1|string2' 151103*.log|grep 'string3' | grep string4 awk '/string1|string2/ && /string3/ && /string4/' 151103*.log ...
2
votes
1answer
47 views

I want to update a INI file using awk

I have an ini file which looks like this [backup] [persistence] log_backup_timeout_s = 900 log_mode = normal I want update this file to [backup] data_backup_parameter_file = ...
0
votes
1answer
23 views

using awk to list pids

I am trying to write a script that will list all of the processes running with a pid between 500 & 20000. This is what I have so far. ps awux |awk '{for (i=1; i<NF; i++) {if (i>=800 ...
1
vote
1answer
46 views

Merge records depending on common columns and specify the values which are uncommon in the last column as comma separated

I need to merge records depending on first three common columns and place the values which are uncommon in the last column as comma separated . I have a requirement to do the following- InputFile : ...
1
vote
1answer
57 views

sed, getting an error:sed: unterminated {

I'm using below sed command sed -n '\|<tag>|{:n; \|</tag>|!{N;bn}; y|\n| |;p }' test.xml OUTPUT: <tag> <name>jolly</name> <class>2nd</class> ...
-1
votes
2answers
49 views

First match if we are on the first line of input, if so print the whole line, else print the first word followed by a comma

for eg: CREATE TABLE MWWDATA.ACK997 ( AKTYPE CHAR(2) DEFAULT '' NOT NULL , AKNUM CHAR(9) DEFAULT '' NOT NULL ); CREATE TABLE MWWDATA.APREIDEXC ( EMPLID NUMBER(15, 0) DEFAULT NULL ); ...
1
vote
1answer
43 views

Matching content in two files and creating a new file [closed]

I have a script to read each line of a file called saitem.unl and store 3 variables in LINENO, ORDREF and PROD. I then use these fields to read prod.unl and if matched set GROUP to field $3. I want to ...
0
votes
3answers
32 views

i want to print the line which starts with a paricular word and in the rest records print only 1st field

for eg: CREATE TABLE MWWDATA."VTCat02" ( "ID" NUMBER(10) DEFAULT NULL , "Cat" VARCHAR2(255) DEFAULT NULL , "Style_Code" VARCHAR2(255) DEFAULT NULL , "Vendor_Style_#" VARCHAR2(255) ...
1
vote
1answer
47 views

Comapring columns in 2 files and printing the values that differ

I have a file (file-1) that looks like this, DIP-10097N|refseq:NP_416170|uniprotkb:P30015 DIP-10117N|refseq:NP_414973|uniprotkb:P08177 DIP-10168N|refseq:NP_418766|uniprotkb:P15005 ...
0
votes
1answer
104 views

Shell script to process service restart procedure

We're running a 3rd party TTY service on our servers, to support some thin clients manufactured by the 3rd party. From time to time, connection to a thin client can get locked up. Procedure is to ...
1
vote
2answers
45 views

gensub on multiple lines

I have a file which has many random lines like aaa bbb ccc ddd eee mark: 98 fff ggg ggg jjjj iii jjj kkkk I want to use awk AND only gensub to match the number "98" above. So far I have this code ...
0
votes
2answers
47 views

How to make this awk command work

I want to make the command below work but I can't. Please check this out: echo "random string random string mark: abcde random string random string" | awk '{gensub(/^.*mark: ...
1
vote
1answer
65 views

How to grep several files using part of filename as a pattern?

I have a folder which includes several files named order[ID].log where [ID] is a string e.g. orderID1.log orderID2.log orderID3.log orderID4.log ... I need to write a script in Linux to grep each ...
0
votes
2answers
63 views

watch not displaying output of script

I have a simple script which just uses awk to extract some information from output of ps. When I run this on the terminal, it works fine. But, when I run this script with watch: watch bench_run.sh ...
0
votes
1answer
46 views

AWK Help ending standalone script

I started to do this.. to swap fields in a csv file but I don't get to finish the script. Any help? What I want to do is to concatenate fields from 2 to 7 and group them by the first field. Fields 1, ...
6
votes
2answers
116 views

How to split a file according to a column (including header) and rename the generated files?

I have a .txt that can be exemplified like this: NAME | CODE name1 | 001 name2 | 001 name3 | 002 name4 | 003 name5 | 003 name6 | 003 I need to write a script to split this file according to the ...
0
votes
1answer
63 views

How to convert awk one-liner to standalone script? [closed]

How can i rewrite this awk one-liner as a standalone script? I want to improve readability to help me understand how it works. awk 'BEGIN{FS=OFS=";"} NR==FNR{for(i=2;i<=7;i++) if($i!="\"\"") ...
1
vote
0answers
24 views

exchange a column of many many csv files using awk [duplicate]

There is a similar post, but this one is slightly different. I have a csv file in which the second column contains some alphanumeric data. I need to delete this entire column and replace it with a new ...
2
votes
1answer
24 views

Compare two files and matched line send to new file

I have two files file1.txt a|1|cd a|2|cd a|4|cd file2.txt a|0001|hj|df a|0002|ed|nb a|0003|vf|za a|0004|er|ns a|0005|oi|lk I need a create new file with the lines matched by de second column of ...
1
vote
1answer
26 views

How can I access a column of a line from within a string?

I have a file I'm traversing in a for loop. I want to do something like... for line in file.out; do echo "Some text before $line($2) $line($3)"; done Where, for every line I insert the 2nd and 3rd ...
1
vote
4answers
124 views

Changing column of multiple csv files

I have a csv file called scenario1.csv in which the second column has column names like "0-4 years high risk", "65+ years first responder" etc. There are 20 such values. The 21st row, 2nd column has ...
2
votes
3answers
84 views

Replace a column and preserve spacing

This is a follow up to unix: replace one entire column in one file with a single value from another file I am trying to replace one column of a file (file1) with one specific value from another file ...
3
votes
3answers
216 views

Awk script to extract a portion of input file

How do I write a awk script that scans the input file for "start" and finds the line containing "next" and displays the following line? Something like this: [user]$ cat test.txt start ...
0
votes
2answers
25 views

Search for a non exact match and return column 1 of that file

I have a tab seperated file which looks like this: Adam Group1,Group2,Group8,Group9,Group10 James Group1,Group11,Group12,Group7 Kelly Group18 Alfred Group1,Group8,Group10 Manuel ...
1
vote
1answer
83 views

How to merge multiple files based on a timestamp

I have 2 files File 1: 01:12:00,001 Some text 01:14:00,003 Some text 02:12:01,394 Some text File 2: 01:12:00,001 Some text 01:12:01,029 Some text 01:13:21,123 Some text I need output as ...
0
votes
1answer
46 views

Quick shell (bash+awk) script to extract a column using the output of another command

Have to repeat below 100+ times. So first command takes paramater $1 and get me string A001C7F28AB20. Then second command is using that string. # Command1 ->snmpget -v1 -c elephantblue $1 ...
0
votes
2answers
43 views

Combine rows from different files, if pattern is matched

Input: two tab-delimited files, File1 and File2. File2 ($2) has to be parsed by patterns found in File1 ($1). Expected output: tab-delimited file, File3. File3 has to contain the same rows as File2, ...
2
votes
3answers
243 views

unix: replace one entire column in one file with a single value from another file

I want to replace a column in one file with a single value from another file using unix. File 1 is a pdb file, structured like this: HETATM 14 H4B FAD B 600 95.544 50.240 71.308 1.00 ...
2
votes
5answers
497 views

Replace characters in user part of email address

I have a list that looks like this: user-name@domain.example user.name@domain.example user_name@domain.example ... I would like to replace any occurence of a dot in the username part. So that: ...
2
votes
3answers
97 views

How to handle ^M in csv files by using sed & awk?

I have a file with delimiter ~, and was not able to load into database: Input file: 24~1~2~2~unsubscribed~wap~7~2015-10-23 20:03:00~2015-10-24 20:04:00~2015-10-25 02:53:26~Testing duplicate msisdn ...
2
votes
2answers
30 views

bash: how to add assigned value to file

I have bash script looping through several files in a directory. The file identifier has been saved as a variable $num. What I would like to do is add $num as another column in my data like in the ...
0
votes
1answer
46 views

Remove trailing spaces from first field only

I have a multi-lined, tab-delimited file like this: 99999bbbbbt999999bbt99t99t99t99 (where b=spaces, t=tabs) Trailing spaces in the 2nd field are ok, but the trailing spaces in the 1st field need ...
0
votes
2answers
41 views

Merge two lines of a file and adding number in the end of lines

I have a large text file abc.txt with the following format: BALT -6.110 130.600 4.06874414 -0.03254425 BALT -6.620 154.460 3.92286595 -0.17842244 BARM 32.740 140.680 3.94326190 -0.15802649 I need ...