awk - pattern-directed scanning and processing language

learn more… | top users | synonyms

3
votes
5answers
109 views

split column after nth character

I'm trying to split my second column in the file below after every 4 characters. file.txt >1A THISISATEST >1B THATISATEST desired output: >1A THIS ISATEST >1B THAT ...
6
votes
4answers
42k views

How to use regex with AWK for string replacement?

Suppose there is some text from a file: (bookmarks ("Chapter 1 Introduction 1" "#1" ("1.1 Problem Statement and Basic Definitions 23" "#2") ("Exercises 31" "#30") ("Notes and References 42" "#34")) ) ...
1
vote
0answers
33 views

How to extract weather data from a html doc using linux string manipulation

I need to extract these info from this weather forecast site, but I don't know how much about string manipulation in linux: acm_RecentLocationsCarousel.push({name:"Irvine, CA", daypart:'day', ...
2
votes
2answers
59 views

How to duplicate a file a number of times while embedding an index in each file

I have a file named, for example, ascdrgi.txt, with the following contents: tiger lion cat I want to duplicate this file a (variable) number of times by changing the last character of the filename ...
7
votes
4answers
4k views

remove blank lines in awk display

I was trying to figure out a solution for this question. I wanted to use awk for the solution. My input file is something like below. -bash-3.2$ cat file ramesh ramesh_venkat ramesh3_venkat3 ...
-1
votes
1answer
38 views

awk getline to display first three lines in file

I'm using awk 'BEGIN {while((getline gf < "awk.script") > 0) {print gf}}' to read me first 3 three lines of the pattern. I can do with head -3 . But I don't want to since, I have to switch ...
3
votes
2answers
180 views

merge duplicate rows in columns

Given a file like this: x y y z x x x y z z y x x x z y y z z y x x x x x x x x I would like the output to be: x y+ z x x+ y z+ y x+ z y y z+ y x+ x+ Is that possible to do with awk or perl in a ...
2
votes
6answers
160 views

Use sed to find and keep only the lines with 2 alpha characters in them

I have a file with many lines like these 33B87401 33B87402 33B87403 33B8EE44 33B87405 33B87406 33B87407 33B87408 33B87409 33B8740A 33B8740B 33B8740C 33B87D0D 33B8740E 33B8740F 33B87410 33B87411 ...
4
votes
4answers
111 views

Extracting strings from between special characters

I have a large text file created from combining lots of html files. cat *.html > all_html_files.txt Inside the text file are specfic strings that I want to extract to another text file. For ...
3
votes
4answers
63 views

Bulk data generation

I need to generate nearly 1 Billion records of unique integers. I tried with awk but it is not generating more than 5million records. Please help. Below is what I had tried so far - awk -v ...
1
vote
1answer
70 views

Want to print NULL if value is not present as awk output

I am trying to print a tshark command output using awk below is my command: tshark -r "test.pcap" -odiameter.tcp.ports:"1234" -R 'diameter.cmd.code == 272 and diameter.flags.request==0 and ...
-2
votes
1answer
24 views

insert patterns into text file after the last line [closed]

I use below codes to insert the lines after specific line into text file w.r.t. the user's variable. echo -n "WHICH STATIONS?" read station echo ${station[${i}]} awk -v input="$station" ' BEGIN { ...
1
vote
1answer
42 views

Awk split and conditional printing

I have an ugly data file that I need to clean up - Date User Name Item Id Title Quantity Price ...
1
vote
3answers
34 views

awk to replace columns and print if condition statisfy

I have a file with below output Sending showtrans string ... Oldest redo log files necessary to restart Extract are: Redo Thread 1, Redo Log Sequence Number 29334, SCN 3364.4078507030 ...
5
votes
5answers
425 views

Remove lines from file up to a pattern, unless the pattern doesn't exist

I am attempting to write a filter using something like sed or awk to do the following: If a given pattern does not exist in the input, copy the entire input to the output If the pattern exists in ...
10
votes
5answers
15k views

Delete range of lines above pattern with sed (or awk)

I have the following code that will remove lines with the pattern banana and 2 lines after it: sed '/banana/I,+2 d' file So far, so good! But I need it to remove 2 lines before banana, but I can't ...
1
vote
1answer
40 views

Unix awk decimal export in pipe

I'm struggling to adapt a script, which loops through a large number of individual files (subj_id), and prints out the value of $4 in a specified line. Text file contains lines # Measure Brain, ICV, ...
1
vote
3answers
88 views

Compare two files with awk

I'm trying to compare two files and print the output if they match on some fields. File1.txt: bart:29482164591748 apu:29482164591748 smithers:68468468468464 lisa:68468468468464 ...
0
votes
2answers
63 views

Substituting \ for \\ in column

I exported a column of thousands of salts from mysql for hashcat. Every instance of the character \ was replaced with a double character \\ in the exported column. Is there a way to either sed or awk ...
3
votes
3answers
281 views

awk manipulation of file

I have a big file in this format, whose first few lines I am showing you including the header. gene c1 c2 c3 c4 c5 c6 c7 c8 G1*1 0 0 0 0 0 0 0 0 G2*2 0 0 0 0 1 ...
3
votes
2answers
42 views

using printf function for repeated patterns

I have codes which append the patterns into text file w.r.t. the user's input as follows; echo -n "WHICH STATIONS?" read station awk -v input="$station" ' BEGIN { n = split(tolower(input), ...
0
votes
2answers
42 views

Remove both lines if column A values are repeated in next row

My input is like this: pak_c35079_g1_i1|m.14890 Pfam pak_c20797_g1_i1|m.3458 ProDom pak_c20797_g1_i1|m.3458 Pfam pak_c28265_g1_i3|m.9595 TIGRFAM pak_c28265_g1_i3|m.9595 ...
2
votes
2answers
58 views

Exchanging One Column From File1 with files from File2

I have two files. The first file has 11 columns, for example: 1 2 3 4 5 6 7 8 9 10 11 The second has 10 columns and might look like this: 11 22 33 44 55 66 77 88 99 100 What I want to do is look ...
1
vote
1answer
41 views

Unable to run awk program

I have created a simple awk program file in /media/ravan/data/program The code: #! /usr/bin/awk -f BEGIN { print "Don’t Panic bro!" } While executing in terminal: ...
3
votes
2answers
81 views

different results from awk and grep

This gives result 124: awk 'BEGIN {FS = ","; count = 0}; { if ($7 ~ /Nature Life/) { count++ }} END {print count}' file.csv This gives result 123: grep -cE '^([^,]*,){6}[^,]*Nature Life' ...
5
votes
5answers
1k views

Why doesn't awk make the sum zero but instead a very small number?

I have this file and I want to sum all the number in first column. Easy: awk '{s+=$1;print $1,s}' file 0.1048 -1.2705 0.4196 -0.8509 0.4196 -0.4313 0.2719 -0.1594 0.0797 -0.0797 0.0797 -5.55112e-17 ...
5
votes
4answers
354 views

How can I blank the nth to mth field using the awk command?

Can someone please give me a solution for the below using awk command? if any other command would help, that is also much appreciated. Below is the input U,N,UNIX,000,A,5 N,P,SHELL,111,B,6 ...
-1
votes
1answer
34 views

Trying to sum a decimal column from a csv file for each different vendor

I am new to Unix/Linux and am trying to sum a column for each vendor in a loop. I apologize for the vagueness. I will try to explain. It is suppose to read from a file and based on the vendor name ...
1
vote
1answer
45 views

Remove leading 0 from specific field

I have a file with multiple fields but only want to remove the leading 0s from the 2nd column (col B) rather than all of the columns. I have tried: sed 's/^0//g' filename > new_filename but this ...
1
vote
1answer
27 views

Matching 2 main columns between files; and paste other columns into the output file when those main columns match. Keep row size of 1st file intact

This is a follow-up question to my previous question asked about 24 hours ago. Link to previous question: Previous Question Link G-Man solved that problem with a useful code, but I have a follow-up ...
0
votes
3answers
36 views

Matching lines in two files [duplicate]

Let's say I have two files, file1 and file2. file1: passd:xxx hopla:alli gnar:gungg araf:utap file2: alli utap How can i check what lines/words from file2 match file? Indeed I could probably do ...
3
votes
2answers
917 views

How do I insert new columns into a .csv file?

Example Data: "Monday","12345","Friday" In the above file, I want to add a new column in between 12345 and Friday. How can I do that ?
2
votes
1answer
69 views

Matching two main columns at the same time between files, and paste supplementary columns into the output file when those main columns match

I have 3 files, each with a unique number of columns, all tab-separated, but some columns are shared between the 3 files. It's the shared columns between the 3 files that I want to use to create some ...
3
votes
3answers
76 views

Print all matching lines and the previous unindented line

For example, lets say I want to find all lines with 'matching' and the previous unindented lines in this text. Container 1 some text some text matching text some text Container 2 some ...
51
votes
15answers
65k views

Is there a way to get the min, max, median, and average of a list of numbers in a single command?

I have a list of numbers in a file, one per line. How can I get the minimum, maximum, median and average values? I want to use the results in a bash script. Although my immediate situation is for ...
14
votes
5answers
32k views

Remove line containing certain string and the following line

I use this cat foo.txt | sed '/bar/d' to remove lines containing the string bar in the file. I would like however to remove those lines and the line directly after it. Preferably in sed, awk or ...
4
votes
1answer
4k views

awk high precision arithmetic

I am looking for a way to tell awk to do high-precision arithmetic in a substitution operation. This involves, reading a field from a file and substituting it with a 1% increment on that value. ...
0
votes
1answer
39 views

awk script to extract the argument of an option

Can some one explain what the awk '{for(i=1;i<=NF;i++) if($i~/-f/) print $(i+1)}') part does in the following snippet line='/wwws/apache/apache2.4.16w-r01/instroot/bin/httpd -f ...
18
votes
11answers
65k views

Show sum of file sizes in directory listing

The Windows dir directory listing command has a line at the end showing the total amount of space taken up by the files listed. For example, dir *.exe shows all the .exe files in the current ...
4
votes
1answer
55 views

How to use Awk to format numbers with a thousands separator

Can anyone tell me how to use the awk command for this statement "the number will contain commas after every three digits. For example, 100000000 will become 100,000,000." I know how to use sed ...
4
votes
4answers
121 views

Use awk interactively through a pipe

Looking at this question, I've noticed that awk can't read user input if the file is being piped into standard input, but it does behave as expected if reading input from a file given as a command ...
7
votes
5answers
607 views

How to abbreviate /path/to/file to /p/t/file

I'm looking for an elegant one-liner (eg, awk) that will shorten a string of a Unix path using the first character of each parent/intermediate level, but the full basename. Easier to show by ...
-4
votes
1answer
30 views

I'm using the following snippet for Log rotation

Can some one explain how this worksps -ef | grep mongod | awk '$0!~/grep/ && $2~/[0-9]/{print "kill -SIGUSR1 "$2}' | bash; find /tmp/logs/* -type f -mtime +7 -exec rm {} \;
0
votes
4answers
40 views

How to remove a first character of that particular sentences from a list of output [duplicate]

grep /var/log /etc/rsyslog.conf | awk '{print $2}' Output: /var/log/messages /var/log/secure -/var/log/maillog /var/log/cron -/var/log/spooler /var/log/boot.log I want the result to be: ...
1
vote
2answers
52 views

awk print to file

I have a big c file that gives a lot of output that I don't care about right now. I wrote this awk script to do mostly what I want. Not sure how to put the finishing touches on it. I would also like ...
1
vote
0answers
36 views

Unix: C-shell:Disk space checking in Hp-UX

i have 10 devices that using hp-ux and i want to check the disk space in each devices. my requirement is if the space more than 90%, the info of device ans space will be save to a log. this is list of ...
0
votes
1answer
29 views

writing of user's input with loop into text file

echo -n "##### STATIONS? #######" read station awk -v input="$station" ' BEGIN { n = split(tolower(input), user) fmt = "%-4s\n" } {print} /stab_site clear/ { for (i=1; ...
1
vote
1answer
914 views

How to select rows from a CSV file based on different column values?

I have a CSV file with values like the following: col1,col2,col3,col4,col5,col6,col7 1,0,0,0,0,BTS,Active 4,5,3,1,1,LocalMode,Offlne 1,2,5,6,3,PermFault,Offline 1,2,6,6,2,BSC,Active ...
2
votes
4answers
3k views

Error: ambiguous redirect while transferring output to command

Output to redirect in script: 511@ubuntu:~/Unix/test$ ls -ltr |awk '{print $9}' default.txt dfah.txt fruit.txt fruit_prices.txt dfh.txt header.txt testfile.txt topoutput.txt Script written on shell ...
0
votes
1answer
30 views

Make new line above last line and add something there

Like a reverse of awk 'NR==2 {print ""} 1 which adds a new line under the first and prints whatever there. I can't replace the number after NR== with a fixed value because I want to be able to do this ...