Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [awk]

A pattern-directed scanning and processing language.

3
votes
1answer
122 views

Merge 2 huge files matching multiple columns and preserving the order (print matching and non-matching values) - scale up from awk

I'm having trouble with merging data from 2 files. It is genetic data with chromosome, position, reference and alternate alleles and I need to merge the files by matching all 4 of those columns - with ...
2
votes
1answer
60 views

uniq -c Equivalent for Groups of Lines of Arbitrary Count

I've got a file of ~1-2 million lines that I'm trying to reduce down by counting duplicate groups of lines, preserving order. uniq -c works okay : $ perl -E 'say for (("foo") x 4, ("bar") x 4, "baz"...
2
votes
1answer
165 views

Formatting and fill missed data based on column

I have the data in this format and I want to sort it based on the label in each column. Would want to replace the missing field by a comma (,) like shown below 2018-03-13 02:09:12,AB=123,BC=6,CAB=Jax,...
2
votes
1answer
88 views

replacing values in one with the values in another file

I have a csv file named eche in following format: INCON,--,INITIAL,CONDITIONS,FOR*****,ELEMENTS,AT,TIME ,0.315570E+13 VC76,0.10000000E+00,0.2837726135782E+08,0.6756896308414E+02 1K02,0.10000000E+00,0....
2
votes
1answer
60 views

Averaging over many rows of data

I have a file of data for 1024 time steps that looks like this: 1 0.240112 0.0610352 1.00659 2 0.240112 0.0610352 1.00659 ... 1024 0.237366 0.0576172 1.00189 I want to do two things -- ...
2
votes
1answer
118 views

awk in solaris 5.8 / get value from two fields/lines

I'm new to UNIX-Solaris, and I got to make a script and I'm having an issue with a value I require. Basically I need to detect if two process are running. I find out I can get that info using the ...
2
votes
1answer
763 views

how to capture the latest kernel version from grub.conf

How can I capture the latest kernel version from grub.conf ( in case default=0)? For example the latest kernel from the file is 2.6.18-409.el5 # all kernel and initrd paths are relative to ...
1
vote
1answer
30 views

Element comparison between subsequent lines, assigning a value for each null output

I have an output which looks like this: ASCII file I want to pick up the last element from the rows that contain the string "Fe\s2b". But the thing is each occurrence of "H\s\s1\s\s\s\s\s\s4861.33A" ...
1
vote
1answer
29 views

awk for loop on files in directory

How could I loop through each file in a directory using this awk command? awk -F: 'FNR==NR{a[$1]=$2;next} $2 in a{print $1 FS a[$2]}' email.phone.txt username.email.txt Basically this partially ...
1
vote
1answer
310 views

Skipping the first line of file

How to skip the first line to print columns in some format and after some arithmetic operation below? Is it possible with next? Input #filename 4e+06 5e+06 6e+06 5e+06 5e+06 6e+06 Code: BEGIN { ...
1
vote
1answer
31 views

Output lines that contain specific strings

I have a file that looks like this: Chr Start End Ref Alt Func.refGene Gene.refGene ExonicFunc.refGene AAChange.refGene Func.knownGene Gene.knownGene ...
1
vote
1answer
46 views

Output log lines between startTime and endTime

Given I have log file app.log with contents: 2018-08-29 07:43:17.123-0500 CDT [osb_02] [[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'] INFO - [OSB Tracing] Outbound ...
1
vote
1answer
43 views

Merge two tables based on value in first column

I have 1 table like this (csv file): p1 p10 p16 p19 p25 p3 p5 p6 p8 p9 con1 567 0 3 0 18 17 8 4 6 7 con3 490 7 6 2 23 26 20 14 12 29 con4 737 1 4 1 ...
1
vote
1answer
574 views

How to remove newline character within a column in a tsv file?

I need "This contains newline" in one row, newline from the column2 and column4 need to remove. nb: For your understanding the data I upload an image if you want I will upload the text content "...
1
vote
1answer
46 views

filter list of partial duplicates by condition(s)

I have a list of partial duplicate records. Each unique record is identified by its first 5 fields, however each record has more than one "feature" associated with it, defined by the contents of the ...
1
vote
1answer
418 views

Multi-line String Pattern Matching, Insertion and Deletion with sed or awk

I'm trying to write a script to automatically update my .ssh/config. Basically it's an ssh management tool that will add an .ssh/config entry for an input computer, user, etc. Question: Are there ...
1
vote
1answer
758 views

Extract intermediate certificate from openssl s_client output

I'm looking for some easy way to get intermediate certificate details from openssl s_client. I can just pipe output to openssl x509 but it takes leaf cert first. I came up with this script, it works ...
1
vote
1answer
415 views

how to use awk to process iostat output, with flags

here are my variables: HEADER='Device bps tps bread bwrtn' HEADERIZE="BEGIN {print \"$HEADER\"}" PRINTF='{printf "%-10s %11s %11s %12s %12s %13s %13s %13s\n", device, bps, tps, ...
1
vote
1answer
182 views

awk function not getting called if I have a begin statement in the awk file

I have an awk file that loads other awk files. Rather than calling the loading code everytime I run the main function of the file I'm trying to load everything in a BEGIN statement first, but if I do ...
9
votes
0answers
246 views

Is this documented behavior for BSD Awk, or a bug?

Apparently, a bracket expression in BSD Awk which contains a character class will ignore any further characters after the character class: MacOS $ cat file.txt _ - . a B 8 : ; @ ~ , MacOS $ awk '/[@~....
2
votes
0answers
181 views

use gff2fasta instead of a bash script to get parts of DNA sequences out of a full genome

EDIT and a solution Because my original question was badly phrased and I was trying to re-invent the wheel I am answering my own question now (maybe it helps someone else): gff2fasta is a tool which ...
2
votes
0answers
755 views

awk to process very large file

I have a file here where I want to tag each line with FILEB and FILEC as references. If column 1 of fileb or filec is in column 2 of filea, get the column 2 of fileb or filec (if column 1 file c ...
2
votes
0answers
113 views

Paging with follow less +F and colorize with awk

I would like to paginate a file with follow, i.e. $less +F and also colorize the output on the screen. I want to use less because I can leave the follow mode, browse the file and then re-enter the ...
2
votes
0answers
332 views

How to use awk-it in Emacs?

I'm a pretty new Emacs user and I want to call awk from Emacs. After googling around I found awk-it at http://www.emacswiki.org/emacs/AWK_It. However I still can't understand how to use it after I ...
1
vote
0answers
38 views

awk print tab delimited columns with find execdir

I am trying to print out columns that are separated by tabs. First, I use find command to identify a list of files to process. Second, I parse the file path to generate the filename to be used for ...
1
vote
0answers
31 views

Compute median over windows and groups

I have a file that looks like this: chr pos value A 1 23 A 2 32 ... A 200 32 ... B 1 41 B 2 28 ... The file is sorted by chr and pos. The ... is just to show the lots of rows in ...
1
vote
0answers
27 views

Merge 2 files based on the 2nd column of each and overwrite common lines in first file

I have a couple of CSVs'. file1 has a large number of URLs and file2 also has a URL field which have to be matched with file1. Now file1 has a couple of columns I need to add to file2. I have managed ...
1
vote
0answers
280 views

strings concatenation on variable

how to collect add do existing "b" values next matching? input: 1:2:3:4:5 6:7:8:9:0 1:3:2:4:5 $ echo -e "1:2:3:4:5\n6:7:8:9:0\n1:3:2:4:5"|awk -F: '$1=="1"{b="line "NR" "$2;print b}' line 1 2 line 3 ...
1
vote
0answers
584 views

How to speed up awk script that uses several large gzip files?

I have two data files: file_1.in, containing over 2k lines like "12 AB0001": 10 AB0001 11 AC0002 12 AD0003 ... A list of *.gz gzipped files (about 1 to 3 million lines) I should extract and parse ...
1
vote
0answers
170 views

How do split and patsplit work?

From Gawk's manual patsplit(string, array, fieldpat, seps) Divide string into pieces defined by fieldpat and store the pieces in array and the separator strings in the seps array. Is it ...
1
vote
0answers
1k views

How to redirect linux command outputs to specific row and column in csv?

I'm working on Unix Build post documentation process, I have a CSV file to be filled in and uploaded for every new build, to ensure the servers are build as per the original request. The CSV file ...
1
vote
0answers
159 views

Unix: C-shell:Disk space checking in Hp-UX

i have 10 devices that using hp-ux and i want to check the disk space in each devices. my requirement is if the space more than 90%, the info of device ans space will be save to a log. this is list of ...
0
votes
0answers
37 views

How to rearrange a data?

I have Following type of data: each block starting with "Atom An X Y Z", I have 4 rows and 5 columns for each number point. I can arrange somehow for 3 columns and 2 rows with following code. awk '!(...
0
votes
0answers
24 views

Inserting column headings in awk command

I have a raw file (gx.txt) having below sample contents shown below. TIME;PCS/Main:GxCCAInitialSuccES;PCS/Main:GxCCAInitialSuccSILP;PCS/Main:GxCCATermSuccES;PCS/Main:GxCCATermSuccSILP;PCS/Main:...
0
votes
0answers
27 views

URL match in a file using shell script

I need to write a unix shell script Lets say i have a URL maps.google.com and file1.txt as below: google.com,Id=0 m*.google.com,Id=1 m*r.google.com,Id=2 map*.google.com,Id=3 *p*.google.com,Id=4 *aps....
0
votes
0answers
38 views

Printing one column from one file - awk

I have this program that restrict the area in input file where is a table and print the number from it. Some of them after an arithmeric operation. I use this program for several files. BEGIN { ...
0
votes
0answers
24 views

convert my csv file output to html table with heading bold and and send email using post fix & also want to change the font to calibri

for i in $(cat HPE_FRAMES_IP); do ssh Reports@$i showsys | awk '{print $2","$5}' ; ssh Reports@$i showpd -degraded -failed | awk '{print $1","$2","$3","$5}' ; done | grep -v "MiB" | grep -v "total" | ...
0
votes
0answers
54 views

How can I parse through a log file, extract certain values and construct a tab or comma delimited text file?

I am trying to convert a pre-formatted section within a log file into a comma or tab delimited tabular view. Here's a sample section contained in the log file: Appliance-01> home Appliance-01>...
0
votes
0answers
40 views

Bash Script with awscli Query Flag Failure

I know the syntax is very ugly but my script below is working fine except "--query" flag of awscli. It's deactivating the old keys and generating the new one for each user. I want the script to write ...
0
votes
0answers
286 views

Send Mail HTML Table body & Attachment Text file

I am try to send a mail with HTML table by reading data from a text file. The table is getting formatted properly and can be seen in mail body but i also want the same text file to be send in same ...
0
votes
0answers
68 views

sed/awk/grep/cut: multiply a number in an xml-attribute by factor of ten

I would like to multiply the attribute in viewBox of svg-files by a factor of ten. I want to change <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100" version="1.1"> to <svg xmlns=...
0
votes
0answers
182 views

invalid char '@' in expression awk output

I have an awk command on ksh script, the command extract fields from a log file to then struct them in new one. ultimate=$(grep -oP 'Time:\K.*' $log2 | column -t | awk '{print $3,"/","'"$month"'","/",...
0
votes
0answers
43 views

Process multiple files, pull out multiple matches and print (save) to a file

I’m trying to parse multiple files to create a summary report. Each file contains multiple lines of interest which I want to combine into a single line. Each file contains the following data set that ...
0
votes
0answers
125 views

What is the fastest command to trim the header of a csv file?

I need to trim the header in a csv file. I have used tail -n +2 ... which works fine but it is really slow (I have lots of 100M files), and I don't understand why since no memory is needed from tail ...
0
votes
0answers
137 views

How to rename by a sequence in seq/nl/…?

Assume you have a list of files with random base names, some of of them have a order such as the following input example. I really would like to do this by nl/seq/..., not by some loop approaches as ...
0
votes
0answers
122 views

How to restore sbatch array job output file names to their original names

In order to submit array jobs using Slurm, I've renamed hundreds of files with incremental numbers using the following command: COUNT=1 for file in ./*; do mv "$file" "${COUNT}" let COUNT++ ...
-1
votes
0answers
59 views

Compare CSV file with while read loop in bash

I have two csv file like this : File 1 : Monday,XXXXX,XXXXX,XXXXX XXXXX,XXXXX,XXXXX,Friday XXXXX,XXXXX,XXXXX,Thursday XXXXX,Wednesday,XXXXX,XXXXX File 2 : Monday,XXXXX,XXXXX,XXXXX XXXXX,XXXXX,...
-1
votes
0answers
45 views

awk text processing for multiple pattern pairs

This is a repost of my previous query which was well answered by Gerard, search pattern pair using awk Now I have another addition to this query, So here is my query. Below is input string: Policy ...
-1
votes
0answers
53 views

How to get department wise total from text file

I have a text file as below: sales|100 accounts|500 marketing|200 sales|50 sales|70 accounts|300 marketing|40 accounts|150 marketing|200 sales|80 with the following script, I have generated ...
-4
votes
0answers
24 views

Linux script to move item from one folder to another

I want to write a script to read a file which contains the below information.The script needs to process each line. Folder A goes to Folder AX condition=name starts with RED Folder B goes to Folder ...