awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

4
votes
5answers
464 views

Command line method to find repeat-word typos, with line numbers

Updated: Clarify line number requirement, some verbosity reductions From the command line, is there a way to: check a file of English text to find repeat-word typos, along with line numbers where ...
0
votes
1answer
44 views

How to select lines in a file that contain a number from a list of numbers

I have a text file, file.dat, with a number of columns separated by a space. There are some numbers in the second column. I have another text file, select.dat with a list of numbers. I'd like to ...
1
vote
1answer
48 views

grep content between two starting bracket

Below is file output that I want to access [1] RsyncCommand: 0 Number of files: 18 Number of files transferred: 0 Total file size: 24.01M bytes Total transferred file size: 0 bytes Literal data: 0 ...
0
votes
1answer
107 views

Identifying duplicate fields and REMOVE both with awk

Previously, I asked the question: Identifying duplicate fields and print both with awk. I have a file with multiple columns and wanted to identify lines where specific column values (cols 3-6) had ...
4
votes
5answers
253 views

How do I print all lines of a file with duplicate values in a certain column

Given a file like this: 1,768,12,46576457,7898 1,123,435,134,146 2,345,6756856,12312,1311 5,234,567465,12341,1341 1,3245,4356345,2442,13 9,423,2342,121,463 9,989,342,121,1212 I would like to list ...
1
vote
1answer
72 views

how concatenate two commands?

Please can someone help me to concatenate two commands on shell, I'd have the path result and concatenate with the number of column in the first field. EX: #this gives me the path of my path ...
1
vote
3answers
113 views

grep a awk string from a file and write to the other part of string file

My file, aaa,has 5 lines. 01_AAA-11-ID22 02_BBB-33-ID44 03_CCC-55-ID66 . . I tried to grep anything after underline from a file bigfile.txt for i in $(cat aaa) do A= $(awk -F '_' '{print $1}' $i) ...
5
votes
6answers
172 views

How to remove symbols from a column using awk

I have data like this: chr1 134901 139379 - "ENSG00000237683.5"; chr1 860260 879955 + "ENSG00000187634.6"; chr1 861264 866445 - "ENSG00000268179.1"; chr1 879584 894689 - "...
0
votes
1answer
24 views

How to remove a row if the difference between two columns is less than 2000

I have a dataset that looks as follows: chr1 HAVANA gene 69091 70008 . + . gene_id "ENSG00000186092.4"; transcript_id "ENSG00000186092.4"; gene_type "protein_coding"; gene_status "...
1
vote
1answer
103 views

split file into multiple pieces

Problem: given a file samplein, it can be split up into multiple pieces as follows: $ cat samplein START Unix Linux START Solaris Aix SCO $ awk '/START/{x="F"++i;}{print > x}' samplein $ ls F* ...
3
votes
2answers
247 views

Conditionally delete a line from file

In a file that looks like: 549.432086168 0.0000E+00 9.6988e-04 2.0580E-02 1.0000E+01 9.6988e-04 2.0580E-02 2.0000E+01 9.6988e-04 2.0580E-02 . . . 5.6000E+02 7.0997e-06 -3.7538E-04 Delete the last ...
6
votes
1answer
521 views

What is a *.sorted command and why doesn't it work in my bash?

In a Bash Guide for Beginners (sec. 3.2.1.2. Local variables) there is an example command which is supposed to list all of the local variables, that are not in enviromental variables, or so I think: ...
0
votes
1answer
54 views

Format the output of awk command

I have to format the output of the awk command which gets the input from a file Sample Input File 123 ABC 12 23 34 45 56 67 78 13 234 DEF 23 34 45 56 67 78 89 345 GHI 34 45 56 67 78 90 10 01 99 456 ...
0
votes
1answer
38 views

Get line after match using awk

I want MAths asad 2323 dfd to show line after asad I do awk '{if($1=="asad")next};1' test get results MAths 2323 dfd why is next not working? I only want to match the Maths not the rest of ...
3
votes
2answers
76 views

Set awk array on command line?

I know the -v switch can be used to awk on the command line to set the value for a variable. Is there any way to set values for awk associative array elements on the command line? Something like: ...
2
votes
2answers
79 views

How to sort data stream by 2 column names (column number may vary)?

I get a stream of data from an API with which look like this: redID blueID whiteID 1 22 2 44 15 41 2 15 15 31 2 14 What I need to do, is to sort ...
2
votes
2answers
34 views

How to alter a column in a row in a tab delimited text file if it's duplicated

Lets say I have these many rows chr1 66999638 66999638 SGIP1 1 + chr1 66999251 66999251 SGIP1 1 + chr1 33545778 33549778 AZIN2 1 + chr1 8376144 8380144 ...
0
votes
2answers
85 views

How to Pivot the data by awk in unix shell script?

I'm trying to pivot a file using awk. This is an example of my input File: P2P,P2P,MP,DL,AIR,UP,HR,UPE,GSM,S,PP,1,NA,0.2,0.2 P2P,P2P,MP,DL,AI,UP,HR,UPE,GSM,F,PP,4,NA,0.2,0.8 P2P,P2P,MP,DL,AIR,UP,HR,...
2
votes
2answers
64 views

Selective extraction of data

I have the following block of data in a txt file : Informatica(r) PMCMD, version [9.5.1 HotFix4], build [313.0217], SOLARIS 64-bit Copyright (c) Informatica Corporation 1994 - 2014 All Rights ...
0
votes
1answer
33 views

awk for loop with leading zeros in variable

I want to run a for loop in awk with leading zeros in the index variable. This isn't for printing a number with leading zeros, which I could easily handle with a printf statement. It's for checking ...
1
vote
2answers
65 views

Adding “:” time-formatting using awk

For an input file named Lab1: 034023 052030 034023 022130 044023 012030 034223 022030 034123 152030 024023 152030 AWK command awk 'gsub(/[0-9][0-9]/,"&:",$1) gsub(/[0-9][0-9]/,"&:",$2)...
0
votes
1answer
29 views

Removing “,” for calculation and adding it back using awk

I have a file as January 124,567.89 Feburary 234,567.78 March 349,123.77 April 112,248.00 May 107,345.22 June 180,670.23 July 122,457.03 August 345,789.99 September 234,...
0
votes
2answers
44 views

Use a shell variable with Awk

I have this: if [[ $1 = "-c" ]] && [[ $2 =~ ^[a-z A-Z]+$ ]] then awk -v var="$2" '{print $2}' something.txt fi What I want is with a comand like -c abc to show the lines in something.txt ...
1
vote
2answers
57 views

Print smallest integer from file using awk custom function?

awk function looks like this in a file name fun.awk: { print small() } function small() { a[NR]=$0 smal=0 for(i=1;i<=3;i++) { if( a[i]<a[i+1]) smal=a[i] else smal=a[i+1] } return smal } ...
1
vote
4answers
204 views

Move lines 2 3 after line 5 using awk

I have file content as: this line 1 no un1x this lines 22 0 butbutbut this 33 22 has unix but not 1 THIS is not butbutbut ffff second line awk 'NR==2 && NR==3 {f=$0;next} NR ==5 &&...
0
votes
1answer
43 views

Consecutively copy two lines and skip the third using awk

With fairly simply awk: awk '(NR%3)' awk.write for this file: this line 1 no un1x this lines 22 0 butbutbut this 33 22 has unix but not 1 THIS is not butbutbut ffff second line I have output as: ...
1
vote
3answers
109 views

Using bash variable substitution instead of cut/awk

Can I use bash variable substitution to extract a piece of a variable based on a delimeter? I'm trying to get the immediate directory name of a filename (in this case, foo). $ filename=./foo/bar/baz....
0
votes
1answer
24 views

Awk print next after matched condition?

using awk 'f&&!--f;{if(length($1)>8){f=1}}' awk.write I can print line after the matched line. I know use of flags but have no idea what &&!--f really do? this line 1 no un1x this ...
1
vote
4answers
289 views

Add line to configuration file from a bash script?

I am trying to figure out to locate a line in a configuration file, then to drop down two lines then to insert a line of code. I was attempting to do this in awk/sed but got stuck on the carriage ...
2
votes
1answer
103 views

Why POSIX left `$(expression)` in awk unspecified?

Answering this question, I found that in: echo 1 2 | awk '{print $(expression)}' the return value of expression is unspecified. It can lead to confusion, as if expression is an uninitialized ...
0
votes
2answers
86 views

Unable to understand printing array results for awk?

For a simple file which contains: 1 a 2 b 3 c 4 d 5 e 6 f 7 g 8 h 9 i 10 j I use that command: awk 'lines[NR]=$0 { print $lines[...
2
votes
2answers
397 views

Extracting subset from fasta file

I have a fasta file which looks like this: >chr1 ACGGTGTAGTCG >chr2 ACGTGTATAGCT >chrUn ACGTGGATATTT >chr21 ACGTTGATGAAA >chrX GTACGGGGGTGG >chrUn5 TGATAGCTGTTG I just want to ...
1
vote
3answers
45 views

Transpose paramter values from a file and Generate parameter files from it

There's a file containing a table with the parameter names as the first line, values in the following lines: How to perform the following using awk, sed or any shell scripting utility: generate a ...
2
votes
1answer
57 views

How can I refer to an awk field defined with an arbitrary expression?

In shell scripting we have the ${name} construct to explicitly specify a variable name and separate it from surrounding text. In awk, I can refer to the fifth field with $5 or the Nth field as ...
2
votes
2answers
709 views

How to keep the field separator when printing $0 with awk [duplicate]

I try to modify one column of my file, then print the result. awk -F"|" '{ if(NR!=1){$5 = $5+0.1} print $0}' myfile It does what I want, but when printing, only the first line keeps its field ...
5
votes
2answers
309 views

Command to remove the first N number of lines in input

Background I am running an SSH server and have this user that I want to delete. I cannot delete this user because he is currently running a few processes that I need to kill first. This is the ...
6
votes
1answer
188 views

How to extract first base64 code out of file?

On Linux I'm having text files and each contains 1 or 2 base64 blocks of code. Following is part of the file and 1 base64 block. Content-Transfer-Encoding: base64 ...
3
votes
2answers
145 views

Merge and Append columns for more than 2 files

The files are tab separated. I have "N" number of files whose contents are as shown in the screenshot above. I want to merge them and append the 5th column. The first 4 columns are the same. I tried ...
1
vote
2answers
44 views

Return line conditonally on next line

I am trying to return a list of IP Addresses for computers that have a windows OS (for my security course). The output is in this format Nmap scan report for 192.168.xx.xxx Host is up (0.066s ...
3
votes
5answers
396 views

split column after nth character

I'm trying to split my second column in the file below after every 4 characters. file.txt >1A THISISATEST >1B THATISATEST desired output: >1A THIS ISATEST >1B THAT ...
1
vote
1answer
66 views

Simulate sed read command with awk

I want to stimulate use of sed: sed '3r awk.scr' awk.script with awk.scr: a b c d e f and awk.script as: hello there is hello i'am there is hello sdfdf dfdfdf aads 23213 3 434 Using awk ...
1
vote
0answers
51 views

How to extract weather data from a html doc using linux string manipulation

I need to extract these info from this weather forecast site, but I don't know how much about string manipulation in linux: acm_RecentLocationsCarousel.push({name:"Irvine, CA", daypart:'day', ...
-1
votes
1answer
94 views

awk getline to display first three lines in file

I'm using awk 'BEGIN {while((getline gf < "awk.script") > 0) {print gf}}' to read me first 3 three lines of the pattern. I can do with head -3 . But I don't want to since, I have to switch ...
2
votes
2answers
71 views

How to duplicate a file a number of times while embedding an index in each file

I have a file named, for example, ascdrgi.txt, with the following contents: tiger lion cat I want to duplicate this file a (variable) number of times by changing the last character of the filename (...
3
votes
2answers
217 views

merge duplicate rows in columns

Given a file like this: x y y z x x x y z z y x x x z y y z z y x x x x x x x x I would like the output to be: x y+ z x x+ y z+ y x+ z y y z+ y x+ x+ Is that possible to do with awk or perl in a ...
2
votes
2answers
64 views

How do I list all lines after two subsequent patterns are matched up to when they don't

I have files that are used to install files with specific owner:group and permissions. I'll ignore permissions for clarity and the details of installing files. Once an owner & group are set to a ...
2
votes
6answers
243 views

Use sed to find and keep only the lines with 2 alpha characters in them

I have a file with many lines like these 33B87401 33B87402 33B87403 33B8EE44 33B87405 33B87406 33B87407 33B87408 33B87409 33B8740A 33B8740B 33B8740C 33B87D0D 33B8740E 33B8740F 33B87410 33B87411 ...
-2
votes
1answer
32 views

insert patterns into text file after the last line [closed]

I use below codes to insert the lines after specific line into text file w.r.t. the user's variable. echo -n "WHICH STATIONS?" read station echo ${station[${i}]} awk -v input="$station" ' BEGIN { ...
3
votes
4answers
72 views

Bulk data generation

I need to generate nearly 1 Billion records of unique integers. I tried with awk but it is not generating more than 5million records. Please help. Below is what I had tried so far - awk -v loop=...
3
votes
4answers
134 views

Extracting strings from between special characters

I have a large text file created from combining lots of html files. cat *.html > all_html_files.txt Inside the text file are specfic strings that I want to extract to another text file. For ...