Questions tagged [awk]

A pattern-directed scanning and processing language.

Filter by
Sorted by
Tagged with
1 vote
5 answers
82 views

How do I replace all the spaces in just a substring of certain lines in a file?

I have 3000 Markdown files named "journal/*.md", some of which contains file links like this: blah blah text [Label here](file:///path/to file/may contain/spaces) I want to change all such ...
user avatar
  • 111
-1 votes
1 answer
40 views

write bash code loop a file line by line that have values, first word as a row and create csv [closed]

i have this sample data dn; uid=456456456,cn=users,dc=efsfb,dc=org,dc=co; objectclass; top; objectclass; person; objectclass; organizationalPerson; objectclass; inetOrgPerson; uid; 456456456; ...
user avatar
  • 101
3 votes
1 answer
96 views

How do I remove anagram duplicates from a list?

I have a tab-delineated .csv with strings and descriptions. A bit like this: strap tight elba Island foo bar able Cain parts buy able Cain First, I'm sorting the file by length of ...
user avatar
  • 31
3 votes
1 answer
47 views

How to get the output file format(with defined spaces) as same as input file format when passing through loops [closed]

I need to get the output file same as the input file format after passing through the while loop. eg: cat samp.csv a b c d cat samp.csv | while IFS= read -r line do echo $...
user avatar
0 votes
2 answers
65 views

Reformatting a list of elements using bash/sed

I have a CSV file with a list like this: URL,Domain,anchor https://example1.com,Example1,Category1 I need to reformat this into HTML, like this: <li><a href="https://example1.com" ...
user avatar
-1 votes
2 answers
53 views

Extremely large text file manipulation (adding a column; changing contents of column)

I have a set of very large (80GB) files (genomic variants dosage) which are ".txt.gz" at the moment and have the following layout (columns/variables): Chr RSID POS REF ALT Ind1 Ind2...Ind(n)...
user avatar
-2 votes
4 answers
104 views

Add two new columns in tab delimited file

I have to add two new columns at the end of the tab-delimited file with specific entries. My file looks like this chr2 88861474 88861574 IGKJ2 chr2 88861170 88861270 IGKJ3 chr2 ...
user avatar
2 votes
4 answers
94 views

How to append string to uppercase letters in bash

Here's some sample code: {"key_code":"a"}, {"key_code":"B"}, {"key_code":"c"}, {"key_code":"D"} Here's what I'm trying ...
user avatar
  • 19
1 vote
6 answers
107 views

How to remove the same motif and affect all the other line in awk

I have a file containing two DNA sequence (with the same number of base) : >seq1 NNNNNAGAATGGGTGANNATTTCCCNN >seq2 NNAGGGTCCCAATCCNNAACCTTNNNN And I want to remove the position that contain a N ...
user avatar
  • 499
-3 votes
2 answers
53 views

merge lines startimg from pattern to next pattern excluding second patten

Input: @@string1 string2, word @@string3 string4, word @@string5, word Desired output: @@string1string2,word @@string3,string4,word @@string5, word Tried sed -n '/^@@/{:a;N;/^@@/!ba};/^@@/s/\n//g;p' ...
user avatar
0 votes
2 answers
83 views

Save results from awk to a variable

I have a variable (that contains multiple lines of text) that I'm modifying using awk to replace the whitespace between fields with a comma and I'm trying to save it to another variable. This is what ...
user avatar
  • 3
0 votes
4 answers
101 views

Select relevant data from a file (per columns and rows)

I have file1 which looks like this: 0 0 0 1 0 8 ST1 2 3 5 2 2 2 ST3 4 3 4 2 5 5 ST5 1 9 1 5 7 8 Desired output file: ST1 6 5 2 ST3 12 4 5 ST5 9 1 7 How to solve this? I would like to use awk. I don'...
user avatar
  • 69
0 votes
2 answers
80 views

Need help to convert csv file format using Unix commands

Input File : "Run Date/Time: 2022-02-09 12:47",,,GOOD_MORNING_WORLD "File Processed: AB-FILE2.20220209.110516",,,GOOD_MORNING_WORLD AB1234,5," PQR2",...
user avatar
2 votes
2 answers
265 views

How to format this IP list?

I have a comma-separated list in this rather simple format: IPrangestart,IPrangeend,int number,date (delimited by slashes),Name (is dirty, contains spaces, dots, dashes, quotemarks etc) The first ...
user avatar
0 votes
5 answers
460 views

Remove all characters till the first letter ([a-zA-Z]) in a string

I want to remove all characters till the first letter [a-zA-Z] in a string. For example: #121Abc --> Abc %Ab#c --> Ab#c Ab#c --> Ab#c Abc --> Abc 1Abc --> Abc 1 21Abc --> Abc 1^1 ...
user avatar
  • 1,362
2 votes
10 answers
327 views

Separate awk delimiters for the same line

I want to separate 3 columns, the first with a single digit 4, 5 or 6. The second with an ID and the third with its description. Sample input: %ASA-4-105505: (Primary|Secondary) Failed to connect to ...
user avatar
0 votes
1 answer
74 views

Filtering a file based on values from another file

I have two files file1: U 20 100 1_A 1_A U 14 200 1_B 1_B U 14 300 1_C 1_C file2: D 12 90 1_A 1_A D 15 97 1_A 1_A D ...
user avatar
  • 3
0 votes
0 answers
26 views

Refill one column in one file based on values from another file

I have two files: file1.csv: fruit_count|fruit_id 10|1234-5 20|1560-0 40|4500-0 100|6700-0 120|6770-8 110|5555-1 80|6666-9 15|7777-4 13|3333-2 file2.csv: fruit_id|fruit_name|n1|n2| 1234-5|...
user avatar
0 votes
3 answers
53 views

Concatenating some strings in some specific lines of multiple text files into a single file with Bash

I have ~100 text files named *.txt in the following form: ========================================================= rho_center e_center Mass Mass_0 R_e 1.0000e-03 1.1236e-03 1.5379e+...
user avatar
  • 1
1 vote
4 answers
403 views

AWK to help solve Wordle puzzles

I have seen several articles describing how basic Linux commands (grep and regex) can solve Wordle puzzles. If you are not familiar with the game check the notes at the end of this question. I decided ...
user avatar
-1 votes
1 answer
63 views

AWK - How to subtitute one line with a pattern?

I've this input file, I'd like to modify one line and keep the rest : Input : Config.Ini|GENERAL|Release|| Config.Ini|GENERAL|Foo1|0| Config.Ini|GENERAL|Foo2|65#36| Config.Ini|RESULT|Value|AAA#3#0#...
user avatar
-2 votes
2 answers
73 views

how to merge and amend columns of two files

I need to merge two files based on the 1st and 2nd columns. I need to add the difference between 3rd and 4th columns and add TOTAL rows per type of 2nd column. Here are the two input files: File 1: ...
user avatar
  • 239
5 votes
2 answers
252 views

add one blank line in awk on a condition

Morning everybody, I would like to divide groups of matching lines of a file with a blank line. Being new to awk I tinkered around a bit and came up with this: awk '!($0 in a) {print "\n"; a[...
user avatar
0 votes
5 answers
100 views

I want to add awk command result to end of each rows in a csv file

I have files as below: InputFile.csv: "Input_File:",AB-ABCDE.20220128.02064600,,,, "Date_Time Received:","1-28-22 3:47",,,, 1, abc, xyz,pq,11 2, def, xyz,rs,22 3, ghi, ...
user avatar
3 votes
4 answers
462 views

how do you use awk to count lines that are not starting with # in multiple files and list them out with filenames?

I have been using grep "^[^#]" file.txt | wc -l to count the number of lines in a single file. How do I use awk to count the number of lines not starting with # and print out with file names ...
user avatar
1 vote
2 answers
67 views

how to create a csh script that can identify the correct USB disk where I have saved my virtual machines

I'm running Freebsd 13R and as shell I'm using csh. I would like to create a script,that looks the presence of the word "r1w1e2" or "NM13N4CZ" or "3e4d31334e34435a" when ...
user avatar
  • 371
1 vote
3 answers
156 views

Split .txt file into 3 files based on text in line using shell script

I have a requirement where I'm getting a .txt file in output from some tool and I want to split that into three. Example text from file: First line Second line 23456 45677 45678 Third line 90909 ...
user avatar
  • 31
1 vote
1 answer
63 views

Extract linux (admin) users report for remote hosts [closed]

I am trying to find the best way to extract a report of Linux users and also which one of them are admin. The targets are Debian (8,9) and Ubuntu (18,20). The final report would be very long, so I ...
user avatar
2 votes
4 answers
361 views

Convert mm to hh:mm

I am using awk to get all of the 4th comma-delimited fields, which is a duration in minutes. The required output is in the format of hh:mm, so I need to convert that from minutes. "jone",&...
user avatar
  • 35
0 votes
4 answers
112 views

Calculate avg salary of department from input string

If user inputs the employees details as string and we want the output to display the average salary department wise and the output should be in same order as they appear in the input string. For ...
user avatar
  • 23
0 votes
1 answer
24 views

Summary Report of Running JVMs

Would like to output a summary of the currently running java processes. The fields I am interested in are PID, start time, min heap, max help and name of the app. The PID and start time are always at ...
user avatar
  • 3
0 votes
1 answer
65 views

Sorting lines based on certain digit pattern

Assume I have a text file with the following lines: 192.168.1.1.55555 192.168.1.1.55555 192.168.1.1.5555 192.168.1.1.555 192.168.1.1.55 192.168.1.1.55 192.168.1.1.5 8.8.8.8.4433 8.8.8.8.443 8.8.8.8....
user avatar
3 votes
3 answers
287 views

How to trim multiline string to n lines max

I have a very long input string that's multiline, in a local variable $INPUT, for example: a b c d e f How do I trim it to n lines max, let's say n=3, and add a message at the end: a b c ... message ...
user avatar
0 votes
1 answer
89 views

Using AWK command I need to replace the column values for particular columns which is having 0000000X

I need to replace a column value in a fixed width file which is having unknown value in one of the columns and need to replace the column value with some column value in the same line. A B ...
user avatar
0 votes
1 answer
44 views

compare multiple files rows by two columns, match the first compare the second

I have two input files like these: file1 ABC001;text;text;5.00;text;text;;20/06/2020;http://www.domain.com/img/foobar4325.jpg ABC002;text;text;15.20;text;text;;4/12/2021;http://www.domain.com/img/...
user avatar
  • 17
-1 votes
3 answers
69 views

Trim column X leaving a specific number of chatacters if column Y and column Z are not equal to NULL

I am trying to find a way to use awk/sed to filter a csv to truncate the value in a specific column such as column 1 to retain the last 8 characters and replace the beginning with a wildcard, only if ...
user avatar
-1 votes
2 answers
90 views

awk/sed filter example for column manipulation based on data in other columns

I am trying to find an example of how I may effeciently use sed or awk to manipulate csv data in a specific column (col 1 for instance) if the data in col 2 AND col 5 are not equal to "null"....
user avatar
0 votes
1 answer
84 views

parse command output for use in next command

I would like to take each line of the output of one gcloud command for use in another gcloud command. gcloud container clusters list |grep jesse (alias gccl) output: jesse-gke1 us-eastx-...
user avatar
  • 3
0 votes
7 answers
118 views

Finding a substring and extracting those lines from a text file

I have a tab seperated text file which as a following format Name city info Stef NY=123 ID=1;CITY=NY John SE=981;819 ID=110;CITY=SE Stef SE=01 ID=20;...
user avatar
  • 115
3 votes
5 answers
580 views

Recursive grep matching only specific line number

How can I can I find files that contain a specific pattern on a specific line number ? Let's assume I have a directory with a bunch of text files containing 3 lines, such as: Title A Category X ...
user avatar
  • 2,067
-1 votes
2 answers
51 views

averaging a column of a file

I have a CSV file (with time-stamped data), and I have a script that will break each month of the year into a separate file.  The script will take that month and average out the data to a single value....
user avatar
0 votes
1 answer
38 views

Using expect to write a bash script

I am trying to write a bash script using expect. This bash script will reside on a remote server, to be executed at a later date/time. I am using expect to read in a large data file, perform some ...
user avatar
  • 1
-1 votes
4 answers
358 views

How can I remove text between two characters with awk

I would like to know how can I remove everything between the "<" and the ">" of this expression: i`< 1| | 1> -1.0000000000000 The result could be like ...
user avatar
0 votes
2 answers
75 views

Complete file line by line reading dates

I have a text file containing 2 columns. The first one has DATES (DD/MM/YYYY) and the second one, numbers. It looks like this: 15/01/1945 105.0 16/01/1945 4.2 17/01/1945 3.0 31/01/1945 12.0 01/02/...
user avatar
  • 1
7 votes
3 answers
1k views

What are the benefits of this command line (/usr/bin/awk '{$1=$1};1') instead of pure cat

I am reading the AWS eic_harvest_hostkeys script and I don't understand this line: key=$(/usr/bin/awk '{$1=$1};1' < "${file}") What is the of benefit awk? Isn't key=$(/bin/cat "${...
user avatar
  • 544
-1 votes
3 answers
53 views

Trim pathname in CSV file

I am struggling over a filter where I am trying to trim data in a specific column of a CSV after the 3rd or nth occurrence of the character \. My data looks something like this: data,data,c:\path1\...
user avatar
-2 votes
1 answer
99 views

Select first n words of each line of a file

Using sed or awk, I want to select the first n words of each line of a file. I have a file with three lines, say one two three four one two three four one two three four and I want the output file to ...
user avatar
  • 1
1 vote
6 answers
252 views

count duplicate entries within each column

My file contains a hundred of columns. I need to: first counting duplicates entries within each column then display output columns order by the number of rows (max to min). Input file: a b c ...
user avatar
  • 35
-1 votes
4 answers
79 views

How can I filter lines via a numeric range?

This file just has a single column: fs://derps123/20210103/ fs://derps123/20210104/ fs://derps123/20210105/ fs://derps123/20210106/ fs://derps123/20210107/ fs://derps123/20210108/ fs://derps123/...
user avatar
0 votes
2 answers
59 views

remove row with condition and duplicate value on multi column

i need to remove line with condition, column 2 only "eating" and combination values on column 3 and column 4 already occurred in a previous line my sample data csv like this: a,eating,apple,...
user avatar