Questions tagged [awk]
A pattern-directed scanning and processing language.
7,970
questions
3
votes
3
answers
235
views
Replace single quote at beginning of columns
I receive input files that are outside my control that contain a leading single quote in some columns, such as
'foo|'012|that's nice|bar
I want to remove all leading single quotes from each field, to ...
- 632
2
votes
3
answers
170
views
awk appends column in .gz file as seen with cat -A, but changes column name in regular output
Using awk, to this table I want to add a column where first row is "INFO" and rest of the rows are all "1".
$ gunzip -c foo.gz | head
SNPID CHR BP Allele1 Allele2 Freq1 Effect ...
- 227
0
votes
3
answers
61
views
Copy matching lines to the next line and modify?
Given a file like this:
a foo 1
b bar 2
c foo 3
d baz 4
e bar 5
I want to modify that file as follows: copy every line containing bar to the following line and change bar to bar2, so that the result ...
- 1,613
0
votes
1
answer
31
views
Returning lines from a JSON log file which contain IP and timestamps read from a CSV file?
so I'm new to Shell Scripting and could use some help. So in my CSV file, I want to see if IP and timestamp values exists in some way in the line entries of the JSON log file, and if so, return that ...
1
vote
3
answers
247
views
awk print between lines when "/" is part of the name
I need to print lines between those that contain a "/" in the name.
I tried with:
awk '/+SOLUTION/ESTIMATES/,/-SOLUTION/ESTIMATES/' $F > fil$F
and
awk '/+SOLUTION"/"ESTIMATES/,/...
- 23
0
votes
1
answer
75
views
shell one-liners for bulk-renaming multiple files
I am looking for two renaming one-liners to manage my music library.
First concerns filenames. I would like to use as simple as possible names,
that is : %track %title.ext
for instance 01 Song name....
- 81
-3
votes
3
answers
162
views
AWK match exact value from comma separated column in brackets
So I need to find an exact number match from one column in a file. The input files is similar to the following:
33.xx 1111 0 0 0 some+text 12:12:12.9999 9999:090909 88 [11,22,33] 2020-12-12 12:12:...
1
vote
2
answers
87
views
identify content inside base64 encoded content and replace the whole line
I have a script that outputs some information on my shell, like this:
field1: value1
field2: value2
...
fieldn: valuen
one such field/value can be:
CmdLine: C:\Windows\System32\WindowsPowershell\...
- 111
0
votes
6
answers
237
views
Find all lines between two patterns, EXCLUSIVE of the second pattern?
Consider that I have the file listed below. I need to select all lines from every instance of the regex pattern Word A to before the regex pattern Word D.
Word A
Word B
Word C
Word D
Word E
Word F
...
- 1,016
0
votes
4
answers
203
views
merge some line start with special character
I have some log like this:
2023-11-15T08:59:28.000000+00:00 database-1 # Time: 231115 8:59:28
# User@Host: rdsadmin[rdsadmin] @ localhost []
# Thread_id: 3 Schema: QC_hit: No
# Query_time: 0....
- 5
-1
votes
2
answers
95
views
Grep a log file for SQL queries and their execution time
I have an application log file that also captures the execution time for each executed SQL statement, like below:-
Sample logs:
I 2023-11-15 07:18:00.743Z 1760463 37 ZVRwqItu6aw-b8eejMLI_gAAAAU 1@...
- 109
-2
votes
3
answers
113
views
bash + how to verify folders under specific path are ended with number/s
I want to check the folder/s under /var/kafka that all them are ended with number otherwhile I will exit with error
ls -ltr /var/kafka
drwxr-xr-x 399 kafka kafka 28672 Nov 9 13:10 data6
drwxr-xr-x ...
- 12.8k
2
votes
1
answer
184
views
Can awk be told to count the character string length rather than byte string length for '%10s' printf formats?
Try this for an output of |Ü| X|:
echo 'Ü X' | awk '{printf("|% 2s|% 2s|\n", $1, $2)}'
Obviously awk counts the byte length, not the character length of the Ü, so the count is 2 and no left ...
- 878
0
votes
1
answer
42
views
Why doesn't gawk -d create awkvars.out in my current working directory?
gawk -d does not create awkvars.out in my current working directory. Why?
Instead I get the following stdout:
Usage: gawk [POSIX or GNU style options] -f progfile [--] file ...
Usage: gawk [POSIX or ...
- 621
1
vote
4
answers
315
views
Retrieve the 1st and 5th column of a tab-separated file, convert the spaces in the 5th to tabs
I have a tsv file with tab-separated columns. I want to obtain the 5th column, which has space-separated values. Convert the space-separation to tab-separation and save as a new file.
Attempt:
cut -d&...
- 133
2
votes
4
answers
172
views
Parse text file add semicolons before and after numbers; join all other words that are before/ between numbers with hyphens
I have a text file with accounts like:
Input Sample
Paid 100 15/02/2022 3000
recd 50 15/02/2022 nelur trip 3050
PAID 80 25/03/2022 Adjusted towards trip 3130
14 PAID 50 26/03/2022 Given to Nate Cash (...
- 249
0
votes
2
answers
91
views
awk statement to print char variable number of times not working
I have the following to print a dashed line based on the longest line of a csv file:
awk -F ',' '
BEGIN {
longest_line=0
for (i = 1; i <= NF; i++) {
longest[i] = "...
- 235
2
votes
4
answers
296
views
linux + delete by bash script the unused/incorrect UUID number/s from fstab file
our goal is to create bash script that delete the unused / unnecessary UUID number/s from /etc/fstab file ,
brief background - in our labs , we have more then 500 RHEL servers , and we want to fix the ...
- 12.8k
0
votes
1
answer
84
views
bash script to make awk command edit file in place not working
I have the following if-else statements that checks to see if the i flag was added to the command and if so, write to the file from stdin:
29 if [ $inplace -eq 1 ]; then
30 tmp_file=$(mktemp)
31 ...
- 235
2
votes
3
answers
119
views
sed cut and paste matched text to start of file
I’ve a large number of text files in a directory and want to cut and paste the first comment section to be at the start of the file (the length and start point of the comment text varies, in some ...
- 23
0
votes
3
answers
152
views
linux csv column with multiple comma separators - sed regex match and replace with quotes
It's bit of tough, trying to hack the solution.
One of the column from exported sql table as csv has bit of strings separated by commas, data kind of below. Some rows have only strings without any ...
- 27
-1
votes
1
answer
80
views
running awk on stdin not working
I'm trying to run an awk command based on whether or not a file was given as an argument and, if not, just run it on stdin like so:
38 for index in "${line_numbers[@]}"; do
39 if [ -n &...
- 235
-1
votes
3
answers
102
views
How to execute a function inside system function in shell script?
I have a text file named meta.txt with the following information:
And a folder named data
output_folder = "data"
Want to execute a function named conv inside the system function like below,...
- 351
0
votes
2
answers
88
views
Parsing and formatting syslog data with awk
I am getting syslog data from my generator, and I would like to extract the date and time at the beginning of the line and the action that is at the end of the file.
what I have so far is simply this:
...
- 11
1
vote
1
answer
83
views
how to refactor simple script to be pipeable (take in input from stdout)
I have the following script:
#!/bin/bash
file=$1
echo "Phone:"
read phone
echo
echo "Column:"
read column
echo
echo "New Value:"
read value
echo
index=$(cat $file | ...
- 235
3
votes
8
answers
680
views
How to enclose each line in single/double quote and end with comma except for the last line should not have a comma?
I want to enclose these in single/double quote and ending with a comma. But I also need it so that the last line does not include the comma as it is the last record.
I've found several post of the ...
- 31
0
votes
3
answers
113
views
How to filter a table using awk
I was wondering how to filter a table with several columns based on a specific value in each of the columns of interest.
I have this example here:
Chr1 16644 0 0 1 1
...
- 197
1
vote
1
answer
72
views
bash using var inside awk cmd not working
I have the following script to filter an ascii table if one of the fields equals the user input:
ascii () {
arg1=$1
cmd=$(man ascii | awk '/000/,/077/ {print}' | sed '/^$/d')
...
- 235
2
votes
3
answers
472
views
using awk to print two columns one after another
I'm trying to simply print column $4 followed by $NF with the following data:
000 0 00 NUL '\0' (null character) 100 64 40 @
001 1 01 SOH (start of heading) ...
- 235
-1
votes
3
answers
128
views
Replace new lines at end for those lines whose 2nd field in colon separated file span over to multiple line
I have got multiple files with different set of contents but the pattern is same across mulitple lines in different files.
sample input file : cat -n test.txt
1 adjfkhjdhfkjd: ...
1
vote
3
answers
556
views
AWK if condition
Given a file input.txt with contents:
xyMnZnMs76XHgsWJ
size: YN3hxFqFeUxXUMrQ
USPqYEnbquwFhUW7
vhJJduHZu25qHzTw
size: JaTDhQYQtLRWBLjj
uE5HGDAZN6DHU73Q
QxVPfQQaKTgR9LXQ
grnsm8bwSz5fJbQS size:
...
- 420
0
votes
0
answers
49
views
Help with creating a bash code to compare two different files
I'm currently encountering some challenges while working on a bash script in the Linux terminal to perform the following tasks:
Compare the values in the third column of two different files line by ...
- 1
0
votes
4
answers
97
views
awk command to map ASCII char to hex code not working
I'm experimenting with awk and I thought it would be a cool idea to write a small program that maps every ASCII char with its hex code. this is what ive done so far:
The string is:
...
- 235
0
votes
0
answers
38
views
awk cmd to find n! not working
I'm not sure why the following awk cmd isn't working:
seq 10 | awk 'BEGIN {print "---Numbers---"} {total*=$1} 1; END {print "---Total-----\n" total} '
the result is:
---Numbers---
...
- 235
0
votes
1
answer
60
views
compare 2 csv files using specific column and ignore specific string
My OS: RHEL7.X
I've a csv file with 3 columns, like so -
#######Category: Fruit
Berries are fruit,04415bef-8c82-4672-848a-9ac80035040a,"Blah/Blah And/More/Blah"
Apple is a fruit,073c6bf3-...
- 549
-3
votes
4
answers
111
views
Print between two patterns on a single line
I have a file with the following contents:
<alarm="yes">
<moduleId>aaa</moduleId>
<documentation>
<description>bbb</description>
<Object>ccc</...
- 1
-1
votes
2
answers
239
views
Filtering out values in a column that are zero
I have the following dataset:
071,36,035,08422,46217,00000,1,A,Broadalbin village (pt.),New York,18,46,46,46,45,45,44,46,44,46,43,43
071,36,045,29322,56209,00000,1,A,Glen Park village (pt.),New York,...
- 235
0
votes
1
answer
69
views
Using awk -v with Shell variable works great and prints to terminal, but when redirect to a file only the 1st record prints to the file
I am reading a file with a matching pattern then getting the first five lines after the matching pattern. The patterns are not in proper numeric order, so I am using a numeric variable in the pattern ...
0
votes
4
answers
83
views
Using variables in gawk for the repeat interval value of an interval expression
OS Distribution: Ubuntu 22.04.3 LTS
gawk version: GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)
I have a text file that every once in a while a line will have a significant amount of white ...
- 3
1
vote
5
answers
184
views
Duplicate part of a line to another part
I would like to copy the first part (IxoscaEVm****t1_, without the '.p[number]') of the line starting with ">" and paste a before the last ":" of the same line.
Input:
>...
- 139
1
vote
2
answers
119
views
Recursive pattern search - output formating: for each match file print out the specific file name "\n", line number, sentence matched "\n" in colors
I want a file content search command, under Linux so that:
it will search on specified files, e.g: md, txt, htm.
recursively from the folder and its subfolders, e.g: .
the content search can be ...
- 21
0
votes
3
answers
100
views
How to subtract column values by the first row with awk?
The input file is lines of "id number1 number2 count", both number1 and number2 are ascendant.
Barcode000000 48521 1855 0
Barcode000001 48621 1855 0
Barcode000002 48721 1855 1
Barcode000003 ...
- 237
-4
votes
1
answer
81
views
I need to get value of a certain Row and Column in AWK
I need only to get the value of WOX which is just down it and it is in this Example equal 4
IOT CND FO WZX END
0 0 0 0
WOX SR POW LIO LIO2
4 00 0
FDR OTY M
0
I tried ...
- 1
2
votes
5
answers
855
views
Need to Replace multiple spaces between PIPE symbols with no space
Given a pipe-delimited file:
1|123|A |Normal Behaviour Exhibit
2|345|B|Embedded|delimiter
3|678|D|dimension 1"
4| | | | nvalue
5| | | |Missing cvalue
I need to use Unix tools like sed or ...
- 131
1
vote
4
answers
155
views
Read nth column iteratively in two large files, cut and paste side by side to create new nth file
I have two files with around 100,000 space-separated columns in each. I want to extract each with column of the two files, write to a separate file and do command 2.
Example for a file with two ...
- 37
1
vote
5
answers
694
views
awk on date conditions
I am trying delete rows from a person.csv (below) for the condition that person who are not born in last 1 year:
Dataset1:
"Index","User Id","First Name","Last Name&...
- 21
1
vote
3
answers
218
views
How to replace strings in matching column?
suppose here is the file
abc,def,ghi
1,a,zeta
2,b,beta
3,c,ceta
4,d,xaq
5,gh,lpa
Above is just an example but in real there are many columns and I need to replace column strings (e.g having heading ...
- 19
0
votes
1
answer
103
views
Need Help with Date expression - AWK in a csv file
Situation: csv file has date column with different date formats, I want to convert them into a unambiguous date format (like +"%m-%d-%Y").
Example File data : File name = Date_Test_new.csv
3/...
4
votes
5
answers
638
views
Replace a block of spaces with a comma
I have a file in txt format that I want to convert to csv. Between each field there are blocks of spaces. The amount of spaces is not the same between every field.
Is there a command with sed or awk ...
- 185
1
vote
4
answers
86
views
Remove duplicates by adding numerical suffix
How do I append a numerical suffix to lines to remove duplicates?
Pseudo code:
if currLine.startsWith("tag:")
x = numFutureLinesMatching(currLine)
if (x > 0)
currLine = currLine + ...
- 738