awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

2
votes
3answers
122 views

Print Column if values is between two other values

I am fairly new to data analysis and I have a problem with filtering some of my values. I have data arranged in a file with four columns like this: A 1 10 5 B 10 100 120 C ...
1
vote
1answer
69 views

Is Perl ever a better tool than awk for text processing [closed]

This question was inspired by a discussion on stackoverflow - see in particular the comment thread under Ed Morton's answer. I figured this was the site where I could get a definitive answer: In ...
1
vote
1answer
77 views

How to get the JSON STRING from the given string

I have a string given below. I just need to get the first accountHeader (bolded) JSON string, I tried everything.. It didn't work. Can you help me to get the data? <START ...
0
votes
4answers
112 views

Removing certain fields from a CSV file

I have a large CSV file (~15GB) which a client has sent over as an export of a table. The format is a follows: field, ... , field, field which sometimes contains commas, another field which sometimes ...
5
votes
2answers
104 views

how to add command line calculator to bash that uses comma as decimal mark?

How can I add a command line calculator to my bash? I have found some, but all of them use the full stop as decimal mark, but I want to have it to use the comma as decimal mark as most of the world ...
2
votes
2answers
176 views

Do I need to encapsulate awk variables in quotes in order to sanitize them?

As per an answer on stackoverflow, it's my understanding that encapsulating bash variables in double-quotes is a fairly safe way of sanitizing user input. What about awk variables? For example, if ...
3
votes
2answers
104 views

Add thousands separator in a number

In python re.sub(r"(?<=.)(?=(?:...)+$)", ",", stroke ) To split a number by triplets, e.g.: echo 123456789 | python -c 'import sys;import re; print re.sub(r"(?<=.)(?=(?:...)+$)", ",", ...
2
votes
3answers
525 views

How to merge two files based on the matching of two columns?

I have file1 likes: 0 AFFX-SNP-000541 NA 0 AFFX-SNP-002255 NA 1 rs12103 0.6401 1 rs12103_1247494 0.696 1 rs12142199 0.7672 And a file2: 0 AFFX-SNP-000541 1 0 ...
1
vote
4answers
1k views

Finding out the common lines in two files using 4 fields using awk and UNIX

I have 2 files. If field 1, 2, 4 and 5 of both file1 and file2 matches, then I want to print the whole line of file1 and file2 one after another in my output file. File1: sc2/80 20 . ...
2
votes
1answer
43 views

Why does AWK not like dots in one of two substitutions?

I'm trying to use AWK in a bash script to look for a placeholder string in a template file and replace it with the contents of a variable that (can) contain various special characters. Example: ...
2
votes
2answers
338 views

Use awk to split line into array and use that array's values in calling shell

First of all, my apologies for the somewhat clumsily worded title. If you can think of a better one, please feel free to edit. [EDIT: done] What I am trying to do is use awk inside a bash script and ...
3
votes
1answer
44 views

Reorder Multiple Line Blocks with Sed

I am trying to reorder generated output as follows; given a tex fragment as follows, I want to move the /^Constant/ down below /Dummies/, with the line before and after /^Constant/. Further, I would ...
4
votes
2answers
50 views

awk pairwise differences

I run into this problem a lot with data, if I have some integers, I frequently want to look at differences between adjacent integers, I usually solve this in Ruby or Python, but I think such a thing ...
1
vote
1answer
35 views

Unable to unregister vms

I have the following command to extract the name of virtual machines currently available in the host system. vmslist=$(vboxmanage list vms | awk '{print $1}') I pass this list to a for loop in ...
0
votes
3answers
138 views

Awk/grep/sed get comma separated list of numbers from lines of text

I have a log file that has lines that look like this: blah blah blah Photo for (702049679 - blah blah blah Now I know I could get all the lines like that from the file by doing: grep "Photo for" ...
5
votes
2answers
240 views

Gawk: Passing arrays to functions

Stuck with GNU awk 3.1.6 and think I've worked around its array bugs but still have what looks like a scope problem in a 600-line awk program. Need to verify understanding of array scope in awk to ...
1
vote
2answers
96 views

what is awk '{print $1+0.45 “ ” $2 “ ” $3 }' file1> file2?

What does awk '{print $1+0.45 " " $2 " " $3 }' positionXYZ > positionX0.45YZ mean? Does it mean changing a row in the first file and put the new data into the second one?
0
votes
2answers
60 views

Using AWK to delete line and output result

I have a text file and manage to find the line through output comparison. With the book title and author name. awk -F ':' '/KungFu Feet|Chuck Norris/' test.txt Kungfu Feet:Chuck Norris:12:1:1 Is ...
1
vote
7answers
220 views

Sed / Awk - Find string exactly n characters long

Looking through the man pages for sed, awk and grep, I'm unable to find a way to search for a string that is exactly n characters long. Given the following text file, I want to extract just ...
0
votes
1answer
56 views

AWK command failing for large file

I have a .csv file of 6.5 GB. I need to sort the files with 1st, 2nd, 3rd, 4th, 5th and 13th column and need to get the unique records. Also I need to get the duplicate records and redirect it to some ...
3
votes
2answers
124 views

Bash Script To Find Debian Release Number from /etc/debian_version

The following is what I'm currently using for testing the version of Debian. Instead of doing it this way, can I make it more succinct? #!/bin/bash VERSION=NONE if [ $(cat /etc/debian_version) == ...
2
votes
3answers
42 views

To assign variable from a file separated by delimiter

I have a text file with datas as below separated by delimiter: 1000:Tom:user1 1002:Jerry:user2 1003:Victor:user3 1004:Jason:user4 I need to run a while loop (any loop) to fetch details from first ...
5
votes
3answers
94 views

The ERE regex to split() string between a delimiter and end-of-word

I'm using a long gawk 3.1.6 script to do a complex conversion of Zim markdown text into GtkDialog code and am stuck on the following problem... Sample ASCII input... [[link|label label]] [[link]] ...
2
votes
1answer
210 views

Processing shell script options with awk

I'm looking for way to process shell script arguments that is cleaner and more "self documenting" than getopt/getopts. It would need to provide... Full support of long options with or without a ...
3
votes
2answers
1k views

System calls, AWK, and Bringing in external inputs

awk '{ TEMPVAR="/usr/bin"; printf("%s", system("ls $TEMPVAR")); }' empty In this example I'm trying to bring in the variable TEMPVAR into the system call. How would I do this? What I'm aiming to ...
-1
votes
1answer
81 views

extracting ip address from a file [closed]

I'm trying to extract all the IP address from a file. FILE *>i85.95.2.160/29 172.16.24.4 0 200 0 64515 4445 15412 i *>i85.112.80.0/28 172.16.24.4 0 200 ...
1
vote
1answer
50 views

Processing FreeBSD's PF logs with tcpdump/sed/awk

I'm running FreeBSD and have PF firewall set up and this is command i use to display daily number of firewall hits (blocked traffic) sudo tcpdump -n -e -tttt -r /var/log/pflog | grep $(date ...
6
votes
5answers
3k views

Show sum of file sizes in directory listing

The Windows dir directory listing command has a line at the end showing the total amount of space taken up by the files listed. For example, dir *.exe shows all the .exe files in the current ...
1
vote
3answers
139 views

How to calculate the sum of the data that have the same ID in the first column?

I want to sum the values of the rows of my data that have the same ID in the first column. My data looks like data.txt Id a b c d e 1 1 2 3 4 5 1 2 3 4 5 ...
1
vote
4answers
180 views

How do I search for all the folders that begin with a dot ( . )?

My server had been infected and the attacker spread a bunch of .5randomletters folders around my file system, with malicious code inside of them. I want to search for all the folders that begin with a ...
1
vote
1answer
130 views

how to subtract a constant value from a data file?

I want to subtract a constant number from my data to get new data file. For example, I want to subtract 10 from my data. data.txt Id a b c d e 1 1 2 3 4 5 2 2 3 4 5 6 ...
0
votes
2answers
63 views

Merging two different files with conditions

I have two different files as shown below. Content of a.txt: HDR|1|||||||||| DTL|@||||||||||| TLR||||||||||||| HDR|1|||||||||||| ...
0
votes
2answers
72 views

How to Lookup the values of an index values from a big data file?

I have a data file (data_array.txt in the example) and index file, index.txt, at which I want to extract the data from the data file that have the same ID in the index file and store in to a new file, ...
1
vote
4answers
90 views

sed remove end of line for specific lines

I'm trying to find the sed command to remove the end of the line (meaning, get the next line to the current), but only if line starts with the string NOTOK. example input: NOTOK something ...
1
vote
5answers
254 views

Cut and transpose some lines of a file

I have 1344 rows of data column. I want to form 32 row by 42 columns from my input data file. I want to cut the first 42 row and paste to one raw of 42 column data and continue down likewise. My data ...
1
vote
2answers
192 views

Splitting giant files into a specific number of lines

I have a CSV that is 40 000 lines long. What is the simplest tool to read from line x to line y? I am sure there are more professional ways to do it but I would just like a bash script that works in ...
-1
votes
1answer
48 views

Using AWK to monitor changing values every 60 seconds [closed]

Does any one have an easy way of doing this using awk, maybe associative arrays? One other thing is there is a delay of 60 seconds between these two input streams and any time a value on the right ...
0
votes
1answer
25 views

Copying specific text from one file into a schema

I have two files. One being a schema, and the other a list of data. The schema looks like: line_name: shot_edit: channel edit: cable_edit and repeats regularly for the number of lines that I ...
0
votes
2answers
109 views

Substring only the first column in awk

I want to take first 500 characters of the first column and leave the rest of the columns as it is, using awk. This is what I am trying to do: awk '{print ${$1:1:500} $2,$NF;}' doc.txt But I get ...
0
votes
3answers
40 views

linux + awk + file manipulation

I have the following file more file.txt 1 2 3 4 4 2 23 4 5 23 2 3 END_OF_LINE 1 3 7 4 2 23 4 6 7 4 2 1 5 How to print only the fields 4 and 5 and 6 until the line END_OF_LINE , and from ...
5
votes
3answers
278 views

Extracting fixed width records with no delimiter from a single line

I need to extract strings of text from a single file containing one very long line of text with no delimiters. Using the sample line below, these are the following known facts: ...
0
votes
3answers
53 views

Print certain fields of each line until a marker is encountered, then print whole lines till the end of file

Hi have the following file more file.txt 1 2 3 4 4 2 23 4 5 23 2 3 END_OF_LINE 1 3 7 4 2 23 4 6 7 4 2 1 5 How to print only the fields 4 and 5 and 6 from each line until the line ...
1
vote
2answers
146 views

How can I make awk on Busybox treat the entire input with NULL character in between as a single record?

I wrote a base64 encode script for my Busybox system. #!/bin/sh base64encode() { /usr/bin/awk -- ' function asc(char,l_found) { l_found=0; for (i=0;i<=255;i++){ if ...
2
votes
2answers
96 views

Processing two files using awk

I read Comparing two files using Unix and Awk. It is really interesting. I read and tested it, but I can't understand it completely and use it in other cases. I have two files. file1 has one field ...
0
votes
1answer
33 views

rearranging from colum to row in a csv [duplicate]

I have a CSV file that presently has an output like this 1 2 3 4 5 what I need to do is reconfigure the output as follows 1 2 3 4 5 all the entry data on one line, any help would be greatly ...
1
vote
2answers
73 views

awk split parameter by char

file is: BASH.NIR SH.ABI I want the awk will show: User is NIR, SHELL is BASH User is ABI, SHELL is SH I dont know how to split a parameter by char. The Idea is: cat file.txt | awk '{print ...
2
votes
3answers
241 views

awk print ' char

I want to print: cat file.log | awk '{print nir's $1}' output should be: nir's aaa nir's bbb nir's abc nir's dbc The problem is with the ' in nir's. I also tried: cat file.log | awk '{print ...
1
vote
2answers
78 views

Grep and sort several unsorted things in several lines (in one document)

I have a file with several unsorted things in one line which I want to put into a new file This is an example of part of the file I have: X1314448: SaMi|SM_g2554.t1 SaMi|SM_g5072.t1 Des|Des_g3808.t1 ...
1
vote
3answers
63 views

How to add zero (0) in blanks in a big file (example file is below)

Here is a example file where I need to replace blanks with zero A 2 2 2 2 2 B 2 2 2 2 2 C 2 2 2 2 2 D A E B F D R
2
votes
1answer
97 views

Extract multiple lines with secondary keyword search

I have a text file with multiple log data, I would like to search for 'Keyword-1' in a line, and when it finds the line, to search within that line for another keyword such as 'engine-1'. Then ...