Questions tagged [awk]
A pattern-directed scanning and processing language.
7,688
questions
1
vote
5
answers
82
views
How do I replace all the spaces in just a substring of certain lines in a file?
I have 3000 Markdown files named "journal/*.md", some of which contains file links like this:
blah blah text [Label here](file:///path/to file/may contain/spaces)
I want to change all such ...
- 111
-1
votes
1
answer
40
views
write bash code loop a file line by line that have values, first word as a row and create csv [closed]
i have this sample data
dn; uid=456456456,cn=users,dc=efsfb,dc=org,dc=co;
objectclass; top;
objectclass; person;
objectclass; organizationalPerson;
objectclass; inetOrgPerson;
uid; 456456456;
...
- 101
3
votes
1
answer
96
views
How do I remove anagram duplicates from a list?
I have a tab-delineated .csv with strings and descriptions. A bit like this:
strap tight
elba Island
foo bar
able Cain
parts buy
able Cain
First, I'm sorting the file by length of ...
- 31
3
votes
1
answer
47
views
How to get the output file format(with defined spaces) as same as input file format when passing through loops [closed]
I need to get the output file same as the input file format after passing through the while loop.
eg:
cat samp.csv
a b c d
cat samp.csv | while IFS= read -r line
do
echo $...
0
votes
2
answers
65
views
Reformatting a list of elements using bash/sed
I have a CSV file with a list like this:
URL,Domain,anchor
https://example1.com,Example1,Category1
I need to reformat this into HTML, like this:
<li><a href="https://example1.com" ...
-1
votes
2
answers
53
views
Extremely large text file manipulation (adding a column; changing contents of column)
I have a set of very large (80GB) files (genomic variants dosage) which are ".txt.gz" at the moment and have the following layout (columns/variables):
Chr RSID POS REF ALT Ind1 Ind2...Ind(n)...
- 3
-2
votes
4
answers
104
views
Add two new columns in tab delimited file
I have to add two new columns at the end of the tab-delimited file with specific entries. My file looks like this
chr2 88861474 88861574 IGKJ2
chr2 88861170 88861270 IGKJ3
chr2 ...
- 89
2
votes
4
answers
94
views
How to append string to uppercase letters in bash
Here's some sample code:
{"key_code":"a"},
{"key_code":"B"},
{"key_code":"c"},
{"key_code":"D"}
Here's what I'm trying ...
- 19
1
vote
6
answers
107
views
How to remove the same motif and affect all the other line in awk
I have a file containing two DNA sequence (with the same number of base) :
>seq1
NNNNNAGAATGGGTGANNATTTCCCNN
>seq2
NNAGGGTCCCAATCCNNAACCTTNNNN
And I want to remove the position that contain a N ...
- 499
-3
votes
2
answers
53
views
merge lines startimg from pattern to next pattern excluding second patten
Input:
@@string1
string2,
word
@@string3
string4,
word
@@string5, word
Desired output:
@@string1string2,word
@@string3,string4,word
@@string5, word
Tried
sed -n '/^@@/{:a;N;/^@@/!ba};/^@@/s/\n//g;p'
...
- 65
0
votes
2
answers
83
views
Save results from awk to a variable
I have a variable (that contains multiple lines of text) that I'm modifying using awk to replace the whitespace between fields with a comma and I'm trying to save it to another variable. This is what ...
- 3
0
votes
4
answers
101
views
Select relevant data from a file (per columns and rows)
I have file1 which looks like this:
0 0
0 1
0 8
ST1 2 3
5 2
2 2
ST3 4 3
4 2
5 5
ST5 1 9
1 5
7 8
Desired output file:
ST1 6 5 2
ST3 12 4 5
ST5 9 1 7
How to solve this? I would like to use awk. I don'...
- 69
0
votes
2
answers
80
views
Need help to convert csv file format using Unix commands
Input File :
"Run Date/Time: 2022-02-09 12:47",,,GOOD_MORNING_WORLD
"File Processed: AB-FILE2.20220209.110516",,,GOOD_MORNING_WORLD
AB1234,5," PQR2",...
2
votes
2
answers
265
views
How to format this IP list?
I have a comma-separated list in this rather simple format:
IPrangestart,IPrangeend,int number,date (delimited by slashes),Name (is dirty, contains spaces, dots, dashes, quotemarks etc)
The first ...
- 637
0
votes
5
answers
460
views
Remove all characters till the first letter ([a-zA-Z]) in a string
I want to remove all characters till the first letter [a-zA-Z] in a string. For example:
#121Abc --> Abc
%Ab#c --> Ab#c
Ab#c --> Ab#c
Abc --> Abc
1Abc --> Abc
1 21Abc --> Abc
1^1 ...
- 1,362
2
votes
10
answers
327
views
Separate awk delimiters for the same line
I want to separate 3 columns, the first with a single digit 4, 5 or 6. The second with an ID and the third with its description.
Sample input:
%ASA-4-105505: (Primary|Secondary) Failed to connect to ...
0
votes
1
answer
74
views
Filtering a file based on values from another file
I have two files
file1:
U 20 100 1_A 1_A
U 14 200 1_B 1_B
U 14 300 1_C 1_C
file2:
D 12 90 1_A 1_A
D 15 97 1_A 1_A
D ...
- 3
0
votes
0
answers
26
views
Refill one column in one file based on values from another file
I have two files:
file1.csv:
fruit_count|fruit_id
10|1234-5
20|1560-0
40|4500-0
100|6700-0
120|6770-8
110|5555-1
80|6666-9
15|7777-4
13|3333-2
file2.csv:
fruit_id|fruit_name|n1|n2|
1234-5|...
- 1
0
votes
3
answers
53
views
Concatenating some strings in some specific lines of multiple text files into a single file with Bash
I have ~100 text files named *.txt in the following form:
=========================================================
rho_center e_center Mass Mass_0 R_e
1.0000e-03 1.1236e-03 1.5379e+...
- 1
1
vote
4
answers
403
views
AWK to help solve Wordle puzzles
I have seen several articles describing how basic Linux commands (grep and regex) can solve Wordle puzzles. If you are not familiar with the game check the notes at the end of this question. I decided ...
user492570
-1
votes
1
answer
63
views
AWK - How to subtitute one line with a pattern?
I've this input file, I'd like to modify one line and keep the rest :
Input :
Config.Ini|GENERAL|Release||
Config.Ini|GENERAL|Foo1|0|
Config.Ini|GENERAL|Foo2|65#36|
Config.Ini|RESULT|Value|AAA#3#0#...
- 43
-2
votes
2
answers
73
views
how to merge and amend columns of two files
I need to merge two files based on the 1st and 2nd columns. I need to add the difference between 3rd and 4th columns and add TOTAL rows per type of 2nd column. Here are the two input files:
File 1:
...
- 239
5
votes
2
answers
252
views
add one blank line in awk on a condition
Morning everybody,
I would like to divide groups of matching lines of a file with a blank line. Being new to awk I tinkered around a bit and came up with this:
awk '!($0 in a) {print "\n"; a[...
- 53
0
votes
5
answers
100
views
I want to add awk command result to end of each rows in a csv file
I have files as below:
InputFile.csv:
"Input_File:",AB-ABCDE.20220128.02064600,,,,
"Date_Time Received:","1-28-22 3:47",,,,
1, abc, xyz,pq,11
2, def, xyz,rs,22
3, ghi, ...
3
votes
4
answers
462
views
how do you use awk to count lines that are not starting with # in multiple files and list them out with filenames?
I have been using grep "^[^#]" file.txt | wc -l to count the number of lines in a single file.
How do I use awk to count the number of lines not starting with # and print out with file names ...
- 33
1
vote
2
answers
67
views
how to create a csh script that can identify the correct USB disk where I have saved my virtual machines
I'm running Freebsd 13R and as shell I'm using csh. I would like to create a script,that looks the presence of the word "r1w1e2" or "NM13N4CZ" or "3e4d31334e34435a" when ...
- 371
1
vote
3
answers
156
views
Split .txt file into 3 files based on text in line using shell script
I have a requirement where I'm getting a .txt file in output from some tool and I want to split that into three.
Example text from file:
First line
Second line
23456
45677
45678
Third line
90909
...
- 31
1
vote
1
answer
63
views
Extract linux (admin) users report for remote hosts [closed]
I am trying to find the best way to extract a report of Linux users and also which one of them are admin. The targets are Debian (8,9) and Ubuntu (18,20). The final report would be very long, so I ...
- 19
2
votes
4
answers
361
views
Convert mm to hh:mm
I am using awk to get all of the 4th comma-delimited fields, which is a duration in minutes. The required output is in the format of hh:mm, so I need to convert that from minutes.
"jone",&...
- 35
0
votes
4
answers
112
views
Calculate avg salary of department from input string
If user inputs the employees details as string and we want the output to display the average salary department wise and the output should be in same order as they appear in the input string.
For ...
- 23
0
votes
1
answer
24
views
Summary Report of Running JVMs
Would like to output a summary of the currently running java processes. The fields I am interested in are PID, start time, min heap, max help and name of the app. The PID and start time are always at ...
- 3
0
votes
1
answer
65
views
Sorting lines based on certain digit pattern
Assume I have a text file with the following lines:
192.168.1.1.55555
192.168.1.1.55555
192.168.1.1.5555
192.168.1.1.555
192.168.1.1.55
192.168.1.1.55
192.168.1.1.5
8.8.8.8.4433
8.8.8.8.443
8.8.8.8....
- 89
3
votes
3
answers
287
views
How to trim multiline string to n lines max
I have a very long input string that's multiline, in a local variable $INPUT, for example:
a
b
c
d
e
f
How do I trim it to n lines max, let's say n=3, and add a message at the end:
a
b
c
... message ...
0
votes
1
answer
89
views
Using AWK command I need to replace the column values for particular columns which is having 0000000X
I need to replace a column value in a fixed width file which is having unknown value in one of the columns and need to replace the column value with some column value in the same line.
A B ...
- 11
0
votes
1
answer
44
views
compare multiple files rows by two columns, match the first compare the second
I have two input files like these:
file1
ABC001;text;text;5.00;text;text;;20/06/2020;http://www.domain.com/img/foobar4325.jpg
ABC002;text;text;15.20;text;text;;4/12/2021;http://www.domain.com/img/...
- 17
-1
votes
3
answers
69
views
Trim column X leaving a specific number of chatacters if column Y and column Z are not equal to NULL
I am trying to find a way to use awk/sed to filter a csv to truncate the value in a specific column such as column 1 to retain the last 8 characters and replace the beginning with a wildcard, only if ...
- 11
-1
votes
2
answers
90
views
awk/sed filter example for column manipulation based on data in other columns
I am trying to find an example of how I may effeciently use sed or awk to manipulate csv data in a specific column (col 1 for instance) if the data in col 2 AND col 5 are not equal to "null"....
- 11
0
votes
1
answer
84
views
parse command output for use in next command
I would like to take each line of the output of one gcloud command for use in another gcloud command.
gcloud container clusters list |grep jesse (alias gccl)
output:
jesse-gke1 us-eastx-...
- 3
0
votes
7
answers
118
views
Finding a substring and extracting those lines from a text file
I have a tab seperated text file which as a following format
Name city info
Stef NY=123 ID=1;CITY=NY
John SE=981;819 ID=110;CITY=SE
Stef SE=01 ID=20;...
- 115
3
votes
5
answers
580
views
Recursive grep matching only specific line number
How can I can I find files that contain a specific pattern on a specific line number ?
Let's assume I have a directory with a bunch of text files containing 3 lines, such as:
Title A
Category X
...
- 2,067
-1
votes
2
answers
51
views
averaging a column of a file
I have a CSV file (with time-stamped data), and I have a script
that will break each month of the year into a separate file.
The script will take that month and average out the data to a single value....
- 65
0
votes
1
answer
38
views
Using expect to write a bash script
I am trying to write a bash script using expect. This bash script will reside on a remote server, to be executed at a later date/time. I am using expect to read in a large data file, perform some ...
- 1
-1
votes
4
answers
358
views
How can I remove text between two characters with awk
I would like to know how can I remove everything between the "<" and the ">" of this expression:
i`< 1| | 1> -1.0000000000000
The result could be like ...
0
votes
2
answers
75
views
Complete file line by line reading dates
I have a text file containing 2 columns. The first one has DATES (DD/MM/YYYY) and the second one, numbers. It looks like this:
15/01/1945 105.0
16/01/1945 4.2
17/01/1945 3.0
31/01/1945 12.0
01/02/...
- 1
7
votes
3
answers
1k
views
What are the benefits of this command line (/usr/bin/awk '{$1=$1};1') instead of pure cat
I am reading the AWS eic_harvest_hostkeys script and I don't understand this line:
key=$(/usr/bin/awk '{$1=$1};1' < "${file}")
What is the of benefit awk?
Isn't key=$(/bin/cat "${...
- 544
-1
votes
3
answers
53
views
Trim pathname in CSV file
I am struggling over a filter where I am trying to trim data in a specific column of a CSV after the 3rd or nth occurrence of the character \.
My data looks something like this:
data,data,c:\path1\...
- 11
-2
votes
1
answer
99
views
Select first n words of each line of a file
Using sed or awk, I want to select the first n words of each line of a file.
I have a file with three lines, say
one two three four
one two three four
one two three four
and I want the output file to ...
- 1
1
vote
6
answers
252
views
count duplicate entries within each column
My file contains a hundred of columns.
I need to:
first counting duplicates entries within each column
then display output columns order by the number of rows (max to min).
Input file:
a b c ...
- 35
-1
votes
4
answers
79
views
How can I filter lines via a numeric range?
This file just has a single column:
fs://derps123/20210103/
fs://derps123/20210104/
fs://derps123/20210105/
fs://derps123/20210106/
fs://derps123/20210107/
fs://derps123/20210108/
fs://derps123/...
- 954
0
votes
2
answers
59
views
remove row with condition and duplicate value on multi column
i need to remove line with condition, column 2 only "eating" and combination values on column 3 and column 4 already occurred in a previous line
my sample data csv like this:
a,eating,apple,...
- 27