Questions tagged [awk]

A pattern-directed scanning and processing language.

Filter by
Sorted by
Tagged with
3 votes
4 answers
564 views

how do you use awk to count lines that are not starting with # in multiple files and list them out with filenames?

I have been using grep "^[^#]" file.txt | wc -l to count the number of lines in a single file. How do I use awk to count the number of lines not starting with # and print out with file names ...
user avatar
1 vote
2 answers
73 views

how to create a csh script that can identify the correct USB disk where I have saved my virtual machines

I'm running Freebsd 13R and as shell I'm using csh. I would like to create a script,that looks the presence of the word "r1w1e2" or "NM13N4CZ" or "3e4d31334e34435a" when ...
user avatar
  • 371
1 vote
3 answers
222 views

Split .txt file into 3 files based on text in line using shell script

I have a requirement where I'm getting a .txt file in output from some tool and I want to split that into three. Example text from file: First line Second line 23456 45677 45678 Third line 90909 ...
user avatar
  • 31
1 vote
1 answer
68 views

Extract linux (admin) users report for remote hosts [closed]

I am trying to find the best way to extract a report of Linux users and also which one of them are admin. The targets are Debian (8,9) and Ubuntu (18,20). The final report would be very long, so I ...
user avatar
2 votes
4 answers
364 views

Convert mm to hh:mm

I am using awk to get all of the 4th comma-delimited fields, which is a duration in minutes. The required output is in the format of hh:mm, so I need to convert that from minutes. "jone",&...
user avatar
  • 71
0 votes
4 answers
120 views

Calculate avg salary of department from input string

If user inputs the employees details as string and we want the output to display the average salary department wise and the output should be in same order as they appear in the input string. For ...
user avatar
  • 23
0 votes
1 answer
24 views

Summary Report of Running JVMs

Would like to output a summary of the currently running java processes. The fields I am interested in are PID, start time, min heap, max help and name of the app. The PID and start time are always at ...
user avatar
  • 3
0 votes
1 answer
70 views

Sorting lines based on certain digit pattern

Assume I have a text file with the following lines: 192.168.1.1.55555 192.168.1.1.55555 192.168.1.1.5555 192.168.1.1.555 192.168.1.1.55 192.168.1.1.55 192.168.1.1.5 8.8.8.8.4433 8.8.8.8.443 8.8.8.8....
user avatar
3 votes
3 answers
305 views

How to trim multiline string to n lines max

I have a very long input string that's multiline, in a local variable $INPUT, for example: a b c d e f How do I trim it to n lines max, let's say n=3, and add a message at the end: a b c ... message ...
user avatar
0 votes
1 answer
174 views

Using AWK command I need to replace the column values for particular columns which is having 0000000X

I need to replace a column value in a fixed width file which is having unknown value in one of the columns and need to replace the column value with some column value in the same line. A B ...
user avatar
0 votes
1 answer
62 views

compare multiple files rows by two columns, match the first compare the second

I have two input files like these: file1 ABC001;text;text;5.00;text;text;;20/06/2020;http://www.domain.com/img/foobar4325.jpg ABC002;text;text;15.20;text;text;;4/12/2021;http://www.domain.com/img/...
user avatar
  • 17
-1 votes
3 answers
70 views

Trim column X leaving a specific number of chatacters if column Y and column Z are not equal to NULL

I am trying to find a way to use awk/sed to filter a csv to truncate the value in a specific column such as column 1 to retain the last 8 characters and replace the beginning with a wildcard, only if ...
user avatar
-1 votes
2 answers
126 views

awk/sed filter example for column manipulation based on data in other columns

I am trying to find an example of how I may effeciently use sed or awk to manipulate csv data in a specific column (col 1 for instance) if the data in col 2 AND col 5 are not equal to "null"....
user avatar
0 votes
1 answer
101 views

parse command output for use in next command

I would like to take each line of the output of one gcloud command for use in another gcloud command. gcloud container clusters list |grep jesse (alias gccl) output: jesse-gke1 us-eastx-...
user avatar
  • 3
0 votes
7 answers
130 views

Finding a substring and extracting those lines from a text file

I have a tab seperated text file which as a following format Name city info Stef NY=123 ID=1;CITY=NY John SE=981;819 ID=110;CITY=SE Stef SE=01 ID=20;...
user avatar
  • 115
3 votes
5 answers
663 views

Recursive grep matching only specific line number

How can I can I find files that contain a specific pattern on a specific line number ? Let's assume I have a directory with a bunch of text files containing 3 lines, such as: Title A Category X ...
user avatar
  • 2,067
-1 votes
2 answers
51 views

averaging a column of a file

I have a CSV file (with time-stamped data), and I have a script that will break each month of the year into a separate file.  The script will take that month and average out the data to a single value....
user avatar
0 votes
1 answer
47 views

Using expect to write a bash script

I am trying to write a bash script using expect. This bash script will reside on a remote server, to be executed at a later date/time. I am using expect to read in a large data file, perform some ...
user avatar
  • 1
-1 votes
4 answers
611 views

How can I remove text between two characters with awk

I would like to know how can I remove everything between the "<" and the ">" of this expression: i`< 1| | 1> -1.0000000000000 The result could be like ...
user avatar
0 votes
2 answers
84 views

Complete file line by line reading dates

I have a text file containing 2 columns. The first one has DATES (DD/MM/YYYY) and the second one, numbers. It looks like this: 15/01/1945 105.0 16/01/1945 4.2 17/01/1945 3.0 31/01/1945 12.0 01/02/...
user avatar
  • 1
7 votes
3 answers
1k views

What are the benefits of this command line (/usr/bin/awk '{$1=$1};1') instead of pure cat

I am reading the AWS eic_harvest_hostkeys script and I don't understand this line: key=$(/usr/bin/awk '{$1=$1};1' < "${file}") What is the of benefit awk? Isn't key=$(/bin/cat "${...
user avatar
  • 564
-1 votes
3 answers
58 views

Trim pathname in CSV file

I am struggling over a filter where I am trying to trim data in a specific column of a CSV after the 3rd or nth occurrence of the character \. My data looks something like this: data,data,c:\path1\...
user avatar
-2 votes
1 answer
134 views

Select first n words of each line of a file

Using sed or awk, I want to select the first n words of each line of a file. I have a file with three lines, say one two three four one two three four one two three four and I want the output file to ...
user avatar
  • 1
1 vote
6 answers
307 views

count duplicate entries within each column

My file contains a hundred of columns. I need to: first counting duplicates entries within each column then display output columns order by the number of rows (max to min). Input file: a b c ...
user avatar
  • 35
-1 votes
4 answers
99 views

How can I filter lines via a numeric range?

This file just has a single column: fs://derps123/20210103/ fs://derps123/20210104/ fs://derps123/20210105/ fs://derps123/20210106/ fs://derps123/20210107/ fs://derps123/20210108/ fs://derps123/...
user avatar
0 votes
2 answers
63 views

remove row with condition and duplicate value on multi column

i need to remove line with condition, column 2 only "eating" and combination values on column 3 and column 4 already occurred in a previous line my sample data csv like this: a,eating,apple,...
user avatar
-1 votes
2 answers
59 views

Replace empty spaces with TAB except in last column

I have a file that looks like this. How can I replace each empty space with a TAB except in the last column that starts with ID? chr1 13513 14763 Medtr1g004950 . + JCVI gene . ID=Medtr1g004950;Note=...
user avatar
0 votes
4 answers
181 views

awk loop over range of records write to separate files

I have a file with several thousand lines describing the output from an elastic wave mode code. The size of the file is variable depending on the frequency, the thickness of the model, and the number ...
user avatar
  • 1
2 votes
4 answers
179 views

Unable to exclude lines containing ^M from grep

I have the following file from which I want to extract only Removed '2022-01-30_01-00-05', at the end. Removing '2022-01-30_01-00-05'... 0.46% complete (00:03:45 remaining)^M 5.49% complete (00:00:...
user avatar
  • 263
4 votes
7 answers
343 views

awk: deal with newline separation in wrong place

I have some data that looks like this: abc 123 456 789 def 111 222 333 ghi 999 888 777 666 i.e. the records are separated by multiple newlines but in the wrong place. What I want is to get it like ...
user avatar
  • 145
-3 votes
3 answers
173 views

Any idea how to calculate median value of each part of column

I have a data like this: 111 5 111 6 111 1 222 8 222 9 222 1 222 3 555 9 555 7 555 6 and for each value of $1, I want to get the median value of all the values of $2 for that $1 if possible using AWK ...
user avatar
0 votes
1 answer
55 views

print byte from number in awk

I can print a byte from a string literal like: awk 'BEGIN {print "\001"}' | cat -v But I need to print a byte of the result of a bitwise OR. So how can I print a byte from a number? Gawk is ...
user avatar
  • 5
0 votes
0 answers
32 views

How to break a long line into max length by word and use that programatically in a git commit [duplicate]

I have a scenario where I need to convert a long string of text and break it by words not exceeding a set line length (70 characters). I need to replace the space with a '\n' where the line break ...
user avatar
0 votes
2 answers
389 views

Compare one column in one file, against another column and if they match, append column

I would like to compare one column from one file against another column in another file. If the columns match, I would like to append the first column from the first file, as the first column of the ...
user avatar
0 votes
1 answer
103 views

Reading multiple zipped files in awk script

I have an awk script named text_processing.awk . I need to pass multiple zipped files named 01JAN21.txt.gz , 02JAN21.txt.gz etc. to this script for further processing. The output file takes the date ...
user avatar
2 votes
1 answer
72 views

Merge two files using first column

I have two different files and I'd like to do a merge of their information using the first column. File1.txt A,info1,info2 234,info3,info4 CD,info5,info6 File2.txt 234,ccc,bb CD,aaa,dd Expected ...
user avatar
  • 161
2 votes
2 answers
123 views

Matching the first occurrence only of a string then adding N lines awk

Hi would like extract N number of lines after matching the first incidence of a string printing the N lines following using awk. The string is repeated a number of times in the file I am ...
user avatar
  • 25
1 vote
0 answers
15 views

How to import environment variable in awk? [duplicate]

I have the following code: export my_var=3 cat /etc/passwd | awk {'print $1'} Now I need to use my my_var in my awk pseudo code. How can I apply this?
user avatar
  • 21
1 vote
2 answers
89 views

AWK: Insert in random selected lines target terms after source terms from a dictionary

Note: I have already asked a similar question in AWK: Quick way to insert target words after a source term and I am at the beginner level of AWK. This question considers the insertion of multiple ...
user avatar
2 votes
1 answer
190 views

After AWK the number of lines has been increased

After running my AWK script awk -i inplace '(NR==FNR){a[$1];next} (FNR in a) && gsub(/\<Source Term\>/,"& Target Term") 1 ' <(shuf -n 198058 -i 1-$(wc -l &...
user avatar
0 votes
0 answers
28 views

I need to replace placeholders in a yaml file with the lines from another file [duplicate]

I have a text file looking like this Name: somename username:some alphanumeric string password:some alphanumeric string And a yaml file looking like this apiVersion: v1 kind: secret metadata: name: ...
user avatar
0 votes
1 answer
69 views

AWK: Quick way to insert target words after a source term

I'm not familiar with awk. For inserting a single target term after a source term in 198058 random lines I have this code here awk -i inplace '(NR==FNR){a[$1];next} (FNR in a) && gsub(/\&...
user avatar
0 votes
2 answers
93 views

Average of lines with different conditions at header and tail of files

Edit: edited it for clarity and made the example files minimal and more reproducible for easier help. Thanks! I have files that gave 1000+lines. Each file is formatted with the same amount of lines. ...
user avatar
0 votes
1 answer
98 views

How to group docker stats by container name prefix?

How to group docker stats --no-stream by container prefix, and show the sum of RAM and CPU and network and IO? If we consider the first two columns of the docker stats --no-stream command to be like ...
user avatar
1 vote
1 answer
54 views

AWK: Add missing lines in csv file and set specific column to "0"

(Note: this question is heavily shortened because the detailed question was blocked with "This looks like spam.") I'm having a hard time adding missing lines to a series of data in a ...
user avatar
1 vote
2 answers
283 views

Gunzip inside awk script

I am creating an output file inside an awk script which I want to compress then. Input file - Marks.txt Student1:AP:Maths:30:Science:43 Student2:AP:Maths:23:Science:35 Student3:Non_AP:Maths:17:Science:...
user avatar
1 vote
2 answers
213 views

How to programmatically maintain umask in /etc/profile and /etc/bashrc

I am looking for a way to programmatically maintain a consistent umask in the files /etc/profile and /etc/bashrc. They have an entry such as follows: if [ $UID -gt 199 ] && [ "`/usr/bin/...
user avatar
  • 11
4 votes
5 answers
295 views

Manipulating text using shell script: How can I fill in "missing" lines?

I have a list of data, like a CSV but some lines are missing a value. I'd like to generate a value for the missing line based on the lines before and after using linux shell script. Take this table ...
user avatar
0 votes
0 answers
28 views

How to join two files into one in multiple folders using specific columns and add file names where they originated from

I would like to ask, how can I join multiple files into one using a specific column and name it by the originating folder. Let me explain: I have the following files, having multiple columns and rows: ...
user avatar
  • 5
7 votes
1 answer
304 views

Is "warning: range of the form `[o-b]' is locale dependent" a bug in gawk 4.1.0?

While writing some awk code I was using the --lint option for gawk 4.1.0. To my surprise I get a warning warning: range of the form `[o-b]' is locale dependent but my code has only a foo-bar outside ...
user avatar
  • 804

1
3 4
5
6 7
156