awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

4
votes
3answers
587 views

pipe line with non-deterministic output

I have happened upon a command that sometimes works and sometimes does not, even when executed multiple times in rapid succession in a bash shell (I have not tested the behavior in other shells). The ...
2
votes
4answers
456 views

Insert new line after non matching pattern

I have following output from script host1:101 host1:102 host2:101 host2:102 host2:103 host4:101 host5:101 host5:102 host5:103 host5:104 etc.. I wanted to insert a new line after each host group. ...
3
votes
4answers
306 views

Count how many elements in the nth column in a whole file

Input file: A 1,2,3,4 #length($2)=4 B 1,2 #length($2)=2 C 9,8,7,6,5,4 #length($2)=6 Expected output: 12 #4+2+6 A method like: awk -F '[\t,]' '{print length($2)}' but ...
1
vote
3answers
43 views

Concatenate Sequences under a Heading

I have a file like File1. With a heading for example ">Contig1" and I want to concatenate all the sequences under the respective heading. File1: >Contig1 ACCCCATACATCCAAA NNNNNNNNNNCATACA ...
2
votes
2answers
148 views

Merging two tables including multiple ocurrence of column identifiers

I would like to merge two tables: File 1: 1 today 2 tomorrow 3 red File 2: 1 a lot 1 sometimes 2 at work 2 at home 2 sometimes 3 new Desired output (file 3): 1 today ...
3
votes
3answers
203 views

How to view all the content in an awk array?

In my understanding, awk array is something like python dict. So I write down the code bellow to explore it: awk '{my_dict[$1] = $2} END { print my_dict}' zen And I got: awk: can't read value of ...
2
votes
4answers
114 views

Why isn't this awk command doing a full outer join?

Objective: Merge the contents of two files using common key present in the files file1.txt ========= key1 11 key2 12 key3 13 file2.txt ========= key2 22 key3 23 key4 24 key5 ...
2
votes
2answers
166 views

Extract group name from `id` and store it in a variable

$ id usera uid=830(usera) gid=799(groupa) groups=799(groupa) I need to extract the group name from the output of id and store it in a variable. In this case it's groupname=groupa
1
vote
2answers
64 views

Bash / Linux Tools for correcting xml

I am generating a set of 'xml' files from a 'csv' file. Some of the fields in csv file contain '&' and other invalid xml characters that need to be replaced with their appropriate xml counterpart ...
12
votes
5answers
27k views

How to parse JSON with shell scripting in Linux?

I have a JSON output from which I need to extract a few parameters in Linux. This is the JSON output: { "OwnerId": "121456789127", "ReservationId": "r-48465168", "Groups": ...
-1
votes
1answer
81 views

sort | uniq not working on different files [closed]

I have no idea why the command sort filename.txt | uniq is not working on this file: http://packetstormsecurity.com/files/32089/koran.gz.html I have also used: awk '!seen[$0]++' koran.txt perl ...
1
vote
1answer
91 views

Formatting grep output with awk. Simple case and background case

I'm struggling with this problem. I have this line: mplayer *.* 2>/dev/null | grep Playing which just launchs mplayer and only the line matching Playing filename is printed on screen. This works ...
0
votes
1answer
45 views

Search and replacing strings in a numerical data file

I have a data file I need to edit it is in the form: ...
0
votes
1answer
38 views

Multiple input files and output files in awk

I have a folder full of text files named 000.txt to 181.txt. How can I process all of them with the same awk script (program.awk) and send them to their respective output files (output000.txt - ...
0
votes
1answer
44 views

Add the number in the column if exists

I need to count Nodes_Util for each Queue. In the example below, queue1 has only one line, but it varies and it could be nothing sometimes and it applies to all queues. I want to dynamically add the ...
1
vote
2answers
53 views

Using sed to replace javascript code in multiple files

I need to replace an old google analytics code with the new universal tracking code in lots of static html files. The old code looks like this: _uacct = "UA-XXXXXX-X"; urchinTracker(); The new code ...
3
votes
3answers
1k views

reading entries from multiple files

I have few files like A.txt, B.txt and C.txt, each having a only single number in it i.e A.txt has 45, B.txt has 35 and C.txt has 100. How could I read these entries from each files and report with ...
0
votes
2answers
42 views

awk cut from specific word in string

I have those strings gcc-4.8.3-1.aix7.1.ppc.rpm gcc-c++-4.8.3-1.aix7.1.ppc.rpm gcc-cpp-4.8.3-1.aix7.1.ppc.rpm gcc-gfortran-4.8.3-1.aix7.1.ppc.rpm libgcc-4.8.3-1.aix7.1.ppc.rpm I want to cut ...
1
vote
2answers
38 views

Skip the first 6 lines/rows in a text file with awk

How can I skip the first 6 lines/rows in a text file (input.txt) and process the rest with awk? The format of my awk script (program.awk) is: BEGIN { } { process here } END { } My text file ...
0
votes
2answers
81 views

Co Record Verification

I have a task requiring some awk to verify a few records quickly. Let' say: A1,A2 B1,B2 C1,C2 C2,C1 A1,C1 A1,B1 B1,A1 Which would be the best way to check for reciprocity only between the A# the ...
1
vote
6answers
77 views

Count the number of 0 and 1 in a file [duplicate]

I have a file (see below), and I would like to count the number of 1's and of 0's: 1 1 1 0 0 1 1 0 0 0 ... I tried to use awk but it doesn't work, for example, to count the number of 1's: awk -F ...
1
vote
3answers
2k views

Using awk to process ls output with spaces in filenames/paths

All, I have a script in which I use find command to filter out all png files in a given folder and list them along with their size. I want the output in the following format: someFile.png => 1.2K ...
1
vote
2answers
65 views

How to strip time stamps from a file?

I have log which contains lines of data: Mon Apr 20 03:15:18 EDT 2015: my|data|data|data I am trying to write a script which would extract just the data from the log, stripping the leading time ...
1
vote
1answer
25 views

Unable to replace variable in shell script using awk

I have been struggling to replace every occurence of "<KEY>" in a file with ${encryptedkeyValue} I wrote this line but it is not doing ...
0
votes
4answers
46 views

Getting an error when running awk from a script

I have a file some.txt -rwxrw-r-- 1 ivamshky ivamshky 152 Apr 2 00:42 12.sh~ -rw-rw-r-- 1 ivamshky ivamshky 58 Apr 6 19:03 a.c -rw-rw-r-- 1 ivamshky ivamshky 98 Apr 1 20:27 all.sh ...
1
vote
2answers
62 views

Format Column Width with Printf

I have searched online for 10 hours now and tried various codes, but still need help. I am attempting to merge three files using 'paste' and 'awk'. However, the columns are not adjusting to the ...
3
votes
3answers
83 views

Match lines of a file with headers in other to obtain entire para

I want help with scripting to work with two files, wherein file 1 lists amino acids which are in specific order (one below the other and also might repeat) and the second file 2 constitutes the ...
0
votes
1answer
69 views

How to parse an awk variable

I'm trying to pull some information out of a log file, looking back over a specific period of time (e.g. the past hour). I've found the following awk construct that works, capturing the data as a ...
4
votes
4answers
1k views

How can awk cut certain fields and add to the end of each line?

I would like to convert a list containing bunch of svn URLs of issues: cat list.txt //svn.server.address/repos/project/module1/branches/issue-001-name1 ...
1
vote
0answers
24 views

How to use awk-it in Emacs?

I'm a pretty new Emacs user and I want to call awk from Emacs. After googling around I found awk-it at http://www.emacswiki.org/emacs/AWK_It. However I still can't understand how to use it after I ...
0
votes
3answers
65 views

Filtering multi-lines from a log

Should this question be moved to stackoverflow instead? I often need to read log files generated by java applications using log4j. Usually, a logged message (let's call it a log entry) spans over ...
2
votes
2answers
62 views

Need to obtain Awk output in specific sequence

Data in file: 1234567-5678907 3456789-1234563 3456789-1234567 . . . . n Now I want the output in the below format: 1234567-5678907,3456789-1234563,3456789-1234563....n I have used below command ...
1
vote
1answer
45 views

Can't use find -print in AWK

I'm trying to put together a command to find SUID System Executables, which are not the default ones in RHEL. To do this, I'm pulling a list of the local File System (df --local -P), piping it over ...
0
votes
2answers
50 views

Due to spaces between one Word Awk changed output

Due to spaces between one Word Awk changed output e.g : "Local Address" awk consider local and address Separately # netstat -antulp | awk 'NR>1' Proto Recv-Q Send-Q Local Address ...
3
votes
1answer
834 views

Walking Multidimensional arrays in Mawk

I'm able to do this properly in gawk, but when I tried to post my code to the machine where it will run, I realized it was using mawk... $ cat multidim.gawk # test of multidimensional arrays // { ...
8
votes
4answers
438 views

How do you check if a file exists within awk? [-d 'filename'] failing

I'm trying to generate a list of users who have a home directory set which does not exist. It seems I should be able to do this with awk, but something is wrong with my syntax. It keeps telling me ...
-5
votes
2answers
63 views

How to average a column in a text file without using ' character? [closed]

I need to do it without using the ' character. I could do it like this: awk '{sum+=$5} END { print "Average = ",sum/NR}' except this contains a '.
4
votes
2answers
82 views

How to cut from the 1st character to -7 character in awk?

I have a file with multiple lines. Which the second column is the user_id, but the last seven number are random numbers. It need to be trimmed off to get the authentic ID. So what I need is to get ...
0
votes
3answers
48 views

modify file using awk or sed

I have a text file contains ldap data in the give below format, our default structure says every user must have its own respective "home directory" but in given below i found many users have the same ...
2
votes
1answer
38 views

Remove Quotes (double or single) with Awk

I want to remove quotes(double quotes) appearing in the awk output, how it can be achieved # systool -c fc_host -v | awk '/Class Device =/{host=$4}/port_state/{print host,$3}' (This is my awk ...
2
votes
3answers
254 views

How to find a specific 'string' and print out the whole line?

I have a file named myfile.csv containing the following: abc:123:myname:1231 def:423324:arbitrary:value:string StackExchange:Unix:Linux From the terminal I run ./myscript.sh def The contents of ...
1
vote
4answers
144 views

matching and merging lines with awk & printing with Solaris

I want to merge the matching pattern with awk on a Solaris system. $ luxadm probe No Network Array enclosures found in /dev/es Found Fibre Channel device(s): Node WWN:2000000c50f1c8da Device ...
1
vote
2answers
57 views

Print line have special format and amount of comma

This is my sample : 6751176513,ahggasda,3453453,hgdahgsd,hgfhgfhg,, ,, ajhgdha | sajhsgdahj|sdahgdhjaghjsd 6751176513,ahggasda,gdfgdf,2015-10-20 05:15:25,sdfsd,, I need to print the lines that ...
2
votes
3answers
179 views

AWK & Bash: Checking that a User's Home is Owned by the User

I'm trying to cycle through /etc/passwd with a single-line command in order to find those users who have a home directory which is not owned by the user, but I'm VERY new to bash programming, and I ...
1
vote
2answers
49 views

Insert line after matching line in sshd inside pam.d

I want to insert a line after matching line in sshd file which is present inside pam.d directory. For e.g. it should search for this line account required pam_nologin.so and it should ...
0
votes
2answers
41 views

How to scrape multiple IP's and add in a script as a variable?

I am trying to add in my script an awk statement or a: if; else; then if eth0:4 is a match then put in eth0:4 IP How would you get the IP in your script as a variable if you have multiple IP's ...
0
votes
1answer
50 views

How to accumulate data for the previous x number of days

Q: For the argument of the question, say today is 26/03/15. How can I 'accumulate' data for the last 5 days. But these dates are just an example. Ideally I would like to be able to accumulate data ...
1
vote
1answer
41 views

Displaying “No Results” When AWK Command has no output

I have the following code, thank-you to the people who helped me here awk -F: '{if(system( "[ -d " $6 " ]") == 1) {print "The directory " $6 " does not exist for user " $1 }}' /etc/passwd However, ...
4
votes
3answers
133 views

Remove duplicate lines while keeping the order of the lines

[root@server]# awk '!seen[$0]++' out.txt > cleaned awk: (FILENAME=out.txt FNR=8547098) fatal error: internal error Aborted [root@server]# The ""server"" has: 8 GByte RAM + 16 GByte SWAP, x>300 ...
1
vote
0answers
32 views

Unix : Merge the contents of two files using common key present in the files [duplicate]

Objective : Merge the contents of two files using common key present in the files file1.txt ========= key1 11 key2 12 key3 13 file2.txt ========= key2 22 key3 23 key4 24 key5 25 ...