Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [awk]

A pattern-directed scanning and processing language.

3
votes
1answer
126 views

Merge 2 huge files matching multiple columns and preserving the order (print matching and non-matching values) - scale up from awk

I'm having trouble with merging data from 2 files. It is genetic data with chromosome, position, reference and alternate alleles and I need to merge the files by matching all 4 of those columns - with ...
2
votes
1answer
73 views

Remove deduplicates in a very large file()

In this given input file, there are 4 columns. We have to remove the duplicates but there is a catch. There is a preference order C2>C3>C4. so in the output there is only one row with a, one row with ...
2
votes
1answer
63 views

uniq -c Equivalent for Groups of Lines of Arbitrary Count

I've got a file of ~1-2 million lines that I'm trying to reduce down by counting duplicate groups of lines, preserving order. uniq -c works okay : $ perl -E 'say for (("foo") x 4, ("bar") x 4, "baz"...
2
votes
1answer
166 views

Formatting and fill missed data based on column

I have the data in this format and I want to sort it based on the label in each column. Would want to replace the missing field by a comma (,) like shown below 2018-03-13 02:09:12,AB=123,BC=6,CAB=Jax,...
2
votes
1answer
89 views

replacing values in one with the values in another file

I have a csv file named eche in following format: INCON,--,INITIAL,CONDITIONS,FOR*****,ELEMENTS,AT,TIME ,0.315570E+13 VC76,0.10000000E+00,0.2837726135782E+08,0.6756896308414E+02 1K02,0.10000000E+00,0....
2
votes
1answer
64 views

Averaging over many rows of data

I have a file of data for 1024 time steps that looks like this: 1 0.240112 0.0610352 1.00659 2 0.240112 0.0610352 1.00659 ... 1024 0.237366 0.0576172 1.00189 I want to do two things -- ...
2
votes
1answer
127 views

awk in solaris 5.8 / get value from two fields/lines

I'm new to UNIX-Solaris, and I got to make a script and I'm having an issue with a value I require. Basically I need to detect if two process are running. I find out I can get that info using the ...
2
votes
1answer
785 views

how to capture the latest kernel version from grub.conf

How can I capture the latest kernel version from grub.conf ( in case default=0)? For example the latest kernel from the file is 2.6.18-409.el5 # all kernel and initrd paths are relative to ...
1
vote
1answer
66 views

Improve performance when using “system” call (shell escape) processing large files in awk

I have an awk script that processes very large files that look something like this: K1353 SF3987.7PD833391.4 KARE K1353 SF3987.2KD832231.4 MEAKE K1332 IF4987.7RP832231.2 LEAOS K1329 SF2787.7KD362619....
1
vote
1answer
31 views

Element comparison between subsequent lines, assigning a value for each null output

I have an output which looks like this: ASCII file I want to pick up the last element from the rows that contain the string "Fe\s2b". But the thing is each occurrence of "H\s\s1\s\s\s\s\s\s4861.33A" ...
1
vote
1answer
62 views

awk for loop on files in directory

How could I loop through each file in a directory using this awk command? awk -F: 'FNR==NR{a[$1]=$2;next} $2 in a{print $1 FS a[$2]}' email.phone.txt username.email.txt Basically this partially ...
1
vote
1answer
544 views

Skipping the first line of file

How to skip the first line to print columns in some format and after some arithmetic operation below? Is it possible with next? Input #filename 4e+06 5e+06 6e+06 5e+06 5e+06 6e+06 Code: BEGIN { ...
1
vote
1answer
31 views

Output lines that contain specific strings

I have a file that looks like this: Chr Start End Ref Alt Func.refGene Gene.refGene ExonicFunc.refGene AAChange.refGene Func.knownGene Gene.knownGene ...
1
vote
1answer
53 views

Output log lines between startTime and endTime

Given I have log file app.log with contents: 2018-08-29 07:43:17.123-0500 CDT [osb_02] [[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'] INFO - [OSB Tracing] Outbound ...
1
vote
1answer
48 views

Merge two tables based on value in first column

I have 1 table like this (csv file): p1 p10 p16 p19 p25 p3 p5 p6 p8 p9 con1 567 0 3 0 18 17 8 4 6 7 con3 490 7 6 2 23 26 20 14 12 29 con4 737 1 4 1 ...
1
vote
1answer
600 views

How to remove newline character within a column in a tsv file?

I need "This contains newline" in one row, newline from the column2 and column4 need to remove. nb: For your understanding the data I upload an image if you want I will upload the text content "...
10
votes
0answers
263 views

Is this documented behavior for BSD Awk, or a bug?

Apparently, a bracket expression in BSD Awk which contains a character class will ignore any further characters after the character class: MacOS $ cat file.txt _ - . a B 8 : ; @ ~ , MacOS $ awk '/[@~....
2
votes
0answers
184 views

use gff2fasta instead of a bash script to get parts of DNA sequences out of a full genome

EDIT and a solution Because my original question was badly phrased and I was trying to re-invent the wheel I am answering my own question now (maybe it helps someone else): gff2fasta is a tool which ...
2
votes
0answers
798 views

awk to process very large file

I have a file here where I want to tag each line with FILEB and FILEC as references. If column 1 of fileb or filec is in column 2 of filea, get the column 2 of fileb or filec (if column 1 file c ...
2
votes
0answers
116 views

Paging with follow less +F and colorize with awk

I would like to paginate a file with follow, i.e. $less +F and also colorize the output on the screen. I want to use less because I can leave the follow mode, browse the file and then re-enter the ...
2
votes
0answers
337 views

How to use awk-it in Emacs?

I'm a pretty new Emacs user and I want to call awk from Emacs. After googling around I found awk-it at http://www.emacswiki.org/emacs/AWK_It. However I still can't understand how to use it after I ...
1
vote
0answers
53 views

awk print tab delimited columns with find execdir

I am trying to print out columns that are separated by tabs. First, I use find command to identify a list of files to process. Second, I parse the file path to generate the filename to be used for ...
1
vote
0answers
38 views

Compute median over windows and groups

I have a file that looks like this: chr pos value A 1 23 A 2 32 ... A 200 32 ... B 1 41 B 2 28 ... The file is sorted by chr and pos. The ... is just to show the lots of rows in ...
1
vote
0answers
27 views

Merge 2 files based on the 2nd column of each and overwrite common lines in first file

I have a couple of CSVs'. file1 has a large number of URLs and file2 also has a URL field which have to be matched with file1. Now file1 has a couple of columns I need to add to file2. I have managed ...
1
vote
0answers
365 views

strings concatenation on variable

how to collect add do existing "b" values next matching? input: 1:2:3:4:5 6:7:8:9:0 1:3:2:4:5 $ echo -e "1:2:3:4:5\n6:7:8:9:0\n1:3:2:4:5"|awk -F: '$1=="1"{b="line "NR" "$2;print b}' line 1 2 line 3 ...
1
vote
0answers
595 views

How to speed up awk script that uses several large gzip files?

I have two data files: file_1.in, containing over 2k lines like "12 AB0001": 10 AB0001 11 AC0002 12 AD0003 ... A list of *.gz gzipped files (about 1 to 3 million lines) I should extract and parse ...
1
vote
0answers
178 views

How do split and patsplit work?

From Gawk's manual patsplit(string, array, fieldpat, seps) Divide string into pieces defined by fieldpat and store the pieces in array and the separator strings in the seps array. Is it ...
1
vote
0answers
1k views

How to redirect linux command outputs to specific row and column in csv?

I'm working on Unix Build post documentation process, I have a CSV file to be filled in and uploaded for every new build, to ensure the servers are build as per the original request. The CSV file ...
1
vote
0answers
165 views

Unix: C-shell:Disk space checking in Hp-UX

i have 10 devices that using hp-ux and i want to check the disk space in each devices. my requirement is if the space more than 90%, the info of device ans space will be save to a log. this is list of ...
0
votes
0answers
13 views

How to replace substrings with values taken from a dictionary indexed by the substrings

I need to parse a big file replacing possibly occurring substrings, that are part of possibly occurring strings that match a regex, with a value taken from an array whose indexes are the substrings in ...
0
votes
0answers
19 views

How to average the values of different files and save them in a new file

I have about 140 files with data which I would like to process with a script. The files have two types of names: sys-time-4-16-80-15-1-1.txt known-ratio-4-16-80-15-1-1.txt where the two last ...
0
votes
0answers
54 views

What is a great way to list files found in a directory that add up to a certain file size

This list will then be used to create tar balls for easy movement. This is what I came up with so far. Add up files that are just greater than 10GB list them to file. find . -type f -exec ls -al {} ...
0
votes
0answers
25 views

unix shell scripting :compare 2 days values and on current day the value should not be zero

I have two different files in two different location, like one file is generated on today and another one is generated on previous day.Now, i want to compare the data of column B for both the files ...
0
votes
0answers
50 views

Find duplicated columns in file

I can search a file for unique columns and it works with one file. But how do I search multiple files? awk -F"|&" 'x[$1]++ { print $1 " is duplicated"}' /home/pi/tmpdata/f1.txt This howeever ...
0
votes
0answers
18 views

I want to parse a Terraform vars file and store variables from the TF vars file as vars in a Makefile

I have two files: a Terraform variables file (vars.tf), and a Makefile. What I want to be able to do, is to have the Makefile 'read' some of the variables in the Terraform vars.tf file. The vars.tf ...
0
votes
0answers
29 views

Comparing all combinations of lines within two columns using awk + if statement

I have a file with 11 Columns of data deliminated by a space between each. For each line in column 2, I need to find the closest value in column 9. So below it would take the first entry in column 2 (...
0
votes
0answers
20 views

macOS High Sierra “log show ” ouput

The log show command output from a macOS High Sierra is detailed below: **Command 1** log show --info --predicate 'eventMessage contains "system boot:"' log: warning: ./system_logs.logarchive ...
0
votes
0answers
25 views

Inserting column headings in awk command

I have a raw file (gx.txt) having below sample contents shown below. TIME;PCS/Main:GxCCAInitialSuccES;PCS/Main:GxCCAInitialSuccSILP;PCS/Main:GxCCATermSuccES;PCS/Main:GxCCATermSuccSILP;PCS/Main:...
0
votes
0answers
28 views

URL match in a file using shell script

I need to write a unix shell script Lets say i have a URL maps.google.com and file1.txt as below: google.com,Id=0 m*.google.com,Id=1 m*r.google.com,Id=2 map*.google.com,Id=3 *p*.google.com,Id=4 *aps....
0
votes
0answers
39 views

Printing one column from one file - awk

I have this program that restrict the area in input file where is a table and print the number from it. Some of them after an arithmeric operation. I use this program for several files. BEGIN { ...
0
votes
0answers
40 views

convert my csv file output to html table with heading bold and and send email using post fix & also want to change the font to calibri

for i in $(cat HPE_FRAMES_IP); do ssh Reports@$i showsys | awk '{print $2","$5}' ; ssh Reports@$i showpd -degraded -failed | awk '{print $1","$2","$3","$5}' ; done | grep -v "MiB" | grep -v "total" | ...
0
votes
0answers
56 views

How can I parse through a log file, extract certain values and construct a tab or comma delimited text file?

I am trying to convert a pre-formatted section within a log file into a comma or tab delimited tabular view. Here's a sample section contained in the log file: Appliance-01> home Appliance-01>...
0
votes
0answers
53 views

Bash Script with awscli Query Flag Failure

I know the syntax is very ugly but my script below is working fine except "--query" flag of awscli. It's deactivating the old keys and generating the new one for each user. I want the script to write ...
0
votes
0answers
425 views

Send Mail HTML Table body & Attachment Text file

I am try to send a mail with HTML table by reading data from a text file. The table is getting formatted properly and can be seen in mail body but i also want the same text file to be send in same ...
0
votes
0answers
72 views

sed/awk/grep/cut: multiply a number in an xml-attribute by factor of ten

I would like to multiply the attribute in viewBox of svg-files by a factor of ten. I want to change <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 100" version="1.1"> to <svg xmlns=...
0
votes
0answers
201 views

invalid char '@' in expression awk output

I have an awk command on ksh script, the command extract fields from a log file to then struct them in new one. ultimate=$(grep -oP 'Time:\K.*' $log2 | column -t | awk '{print $3,"/","'"$month"'","/",...
0
votes
0answers
43 views

Process multiple files, pull out multiple matches and print (save) to a file

I’m trying to parse multiple files to create a summary report. Each file contains multiple lines of interest which I want to combine into a single line. Each file contains the following data set that ...
0
votes
0answers
128 views

What is the fastest command to trim the header of a csv file?

I need to trim the header in a csv file. I have used tail -n +2 ... which works fine but it is really slow (I have lots of 100M files), and I don't understand why since no memory is needed from tail ...
0
votes
0answers
141 views

How to rename by a sequence in seq/nl/…?

Assume you have a list of files with random base names, some of of them have a order such as the following input example. I really would like to do this by nl/seq/..., not by some loop approaches as ...
0
votes
0answers
130 views

How to restore sbatch array job output file names to their original names

In order to submit array jobs using Slurm, I've renamed hundreds of files with incremental numbers using the following command: COUNT=1 for file in ./*; do mv "$file" "${COUNT}" let COUNT++ ...