Questions tagged [awk]

A pattern-directed scanning and processing language.

Filter by
Sorted by
Tagged with
3 votes
3 answers
235 views

Replace single quote at beginning of columns

I receive input files that are outside my control that contain a leading single quote in some columns, such as 'foo|'012|that's nice|bar I want to remove all leading single quotes from each field, to ...
enharmonic's user avatar
2 votes
3 answers
170 views

awk appends column in .gz file as seen with cat -A, but changes column name in regular output

Using awk, to this table I want to add a column where first row is "INFO" and rest of the rows are all "1". $ gunzip -c foo.gz | head SNPID CHR BP Allele1 Allele2 Freq1 Effect ...
jay.sf's user avatar
  • 227
0 votes
3 answers
61 views

Copy matching lines to the next line and modify?

Given a file like this: a foo 1 b bar 2 c foo 3 d baz 4 e bar 5 I want to modify that file as follows: copy every line containing bar to the following line and change bar to bar2, so that the result ...
Nathan Long's user avatar
  • 1,613
0 votes
1 answer
31 views

Returning lines from a JSON log file which contain IP and timestamps read from a CSV file?

so I'm new to Shell Scripting and could use some help. So in my CSV file, I want to see if IP and timestamp values exists in some way in the line entries of the JSON log file, and if so, return that ...
Linuxnoob's user avatar
1 vote
3 answers
247 views

awk print between lines when "/" is part of the name

I need to print lines between those that contain a "/" in the name. I tried with: awk '/+SOLUTION/ESTIMATES/,/-SOLUTION/ESTIMATES/' $F > fil$F and awk '/+SOLUTION"/"ESTIMATES/,/...
euge1780's user avatar
0 votes
1 answer
75 views

shell one-liners for bulk-renaming multiple files

I am looking for two renaming one-liners to manage my music library. First concerns filenames. I would like to use as simple as possible names, that is : %track %title.ext for instance 01 Song name....
sylvainsab's user avatar
-3 votes
3 answers
162 views

AWK match exact value from comma separated column in brackets

So I need to find an exact number match from one column in a file. The input files is similar to the following: 33.xx 1111 0 0 0 some+text 12:12:12.9999 9999:090909 88 [11,22,33] 2020-12-12 12:12:...
Scott Nipp's user avatar
1 vote
2 answers
87 views

identify content inside base64 encoded content and replace the whole line

I have a script that outputs some information on my shell, like this: field1: value1 field2: value2 ... fieldn: valuen one such field/value can be: CmdLine: C:\Windows\System32\WindowsPowershell\...
João Ciocca's user avatar
0 votes
6 answers
237 views

Find all lines between two patterns, EXCLUSIVE of the second pattern?

Consider that I have the file listed below. I need to select all lines from every instance of the regex pattern Word A to before the regex pattern Word D. Word A Word B Word C Word D Word E Word F ...
RonJohn's user avatar
  • 1,016
0 votes
4 answers
203 views

merge some line start with special character

I have some log like this: 2023-11-15T08:59:28.000000+00:00 database-1 # Time: 231115 8:59:28 # User@Host: rdsadmin[rdsadmin] @ localhost [] # Thread_id: 3 Schema: QC_hit: No # Query_time: 0....
dvthanh's user avatar
-1 votes
2 answers
95 views

Grep a log file for SQL queries and their execution time

I have an application log file that also captures the execution time for each executed SQL statement, like below:- Sample logs: I 2023-11-15 07:18:00.743Z 1760463 37 ZVRwqItu6aw-b8eejMLI_gAAAAU 1@...
Arpit Jain's user avatar
-2 votes
3 answers
113 views

bash + how to verify folders under specific path are ended with number/s

I want to check the folder/s under /var/kafka that all them are ended with number otherwhile I will exit with error ls -ltr /var/kafka drwxr-xr-x 399 kafka kafka 28672 Nov 9 13:10 data6 drwxr-xr-x ...
yael's user avatar
  • 12.8k
2 votes
1 answer
184 views

Can awk be told to count the character string length rather than byte string length for '%10s' printf formats?

Try this for an output of |Ü| X|: echo 'Ü X' | awk '{printf("|% 2s|% 2s|\n", $1, $2)}' Obviously awk counts the byte length, not the character length of the Ü, so the count is 2 and no left ...
Harald's user avatar
  • 878
0 votes
1 answer
42 views

Why doesn't gawk -d create awkvars.out in my current working directory?

gawk -d does not create awkvars.out in my current working directory. Why? Instead I get the following stdout: Usage: gawk [POSIX or GNU style options] -f progfile [--] file ... Usage: gawk [POSIX or ...
John Smith's user avatar
1 vote
4 answers
315 views

Retrieve the 1st and 5th column of a tab-separated file, convert the spaces in the 5th to tabs

I have a tsv file with tab-separated columns. I want to obtain the 5th column, which has space-separated values. Convert the space-separation to tab-separation and save as a new file. Attempt: cut -d&...
Anon's user avatar
  • 133
2 votes
4 answers
172 views

Parse text file add semicolons before and after numbers; join all other words that are before/ between numbers with hyphens

I have a text file with accounts like: Input Sample Paid 100 15/02/2022 3000 recd 50 15/02/2022 nelur trip 3050 PAID 80 25/03/2022 Adjusted towards trip 3130 14 PAID 50 26/03/2022 Given to Nate Cash (...
tgkprog's user avatar
  • 249
0 votes
2 answers
91 views

awk statement to print char variable number of times not working

I have the following to print a dashed line based on the longest line of a csv file: awk -F ',' ' BEGIN { longest_line=0 for (i = 1; i <= NF; i++) { longest[i] = "...
Mathew's user avatar
  • 235
2 votes
4 answers
296 views

linux + delete by bash script the unused/incorrect UUID number/s from fstab file

our goal is to create bash script that delete the unused / unnecessary UUID number/s from /etc/fstab file , brief background - in our labs , we have more then 500 RHEL servers , and we want to fix the ...
yael's user avatar
  • 12.8k
0 votes
1 answer
84 views

bash script to make awk command edit file in place not working

I have the following if-else statements that checks to see if the i flag was added to the command and if so, write to the file from stdin: 29 if [ $inplace -eq 1 ]; then 30 tmp_file=$(mktemp) 31 ...
Mathew's user avatar
  • 235
2 votes
3 answers
119 views

sed cut and paste matched text to start of file

I’ve a large number of text files in a directory and want to cut and paste the first comment section to be at the start of the file (the length and start point of the comment text varies, in some ...
Guy's user avatar
  • 23
0 votes
3 answers
152 views

linux csv column with multiple comma separators - sed regex match and replace with quotes

It's bit of tough, trying to hack the solution. One of the column from exported sql table as csv has bit of strings separated by commas, data kind of below. Some rows have only strings without any ...
Indranil's user avatar
-1 votes
1 answer
80 views

running awk on stdin not working

I'm trying to run an awk command based on whether or not a file was given as an argument and, if not, just run it on stdin like so: 38 for index in "${line_numbers[@]}"; do 39 if [ -n &...
Mathew's user avatar
  • 235
-1 votes
3 answers
102 views

How to execute a function inside system function in shell script?

I have a text file named meta.txt with the following information: And a folder named data output_folder = "data" Want to execute a function named conv inside the system function like below,...
stack_learner's user avatar
0 votes
2 answers
88 views

Parsing and formatting syslog data with awk

I am getting syslog data from my generator, and I would like to extract the date and time at the beginning of the line and the action that is at the end of the file. what I have so far is simply this: ...
ml41782's user avatar
  • 11
1 vote
1 answer
83 views

how to refactor simple script to be pipeable (take in input from stdout)

I have the following script: #!/bin/bash file=$1 echo "Phone:" read phone echo echo "Column:" read column echo echo "New Value:" read value echo index=$(cat $file | ...
Mathew's user avatar
  • 235
3 votes
8 answers
680 views

How to enclose each line in single/double quote and end with comma except for the last line should not have a comma?

I want to enclose these in single/double quote and ending with a comma. But I also need it so that the last line does not include the comma as it is the last record. I've found several post of the ...
benbart's user avatar
  • 31
0 votes
3 answers
113 views

How to filter a table using awk

I was wondering how to filter a table with several columns based on a specific value in each of the columns of interest. I have this example here: Chr1 16644 0 0 1 1 ...
Assa Yeroslaviz's user avatar
1 vote
1 answer
72 views

bash using var inside awk cmd not working

I have the following script to filter an ascii table if one of the fields equals the user input: ascii () { arg1=$1 cmd=$(man ascii | awk '/000/,/077/ {print}' | sed '/^$/d') ...
Mathew's user avatar
  • 235
2 votes
3 answers
472 views

using awk to print two columns one after another

I'm trying to simply print column $4 followed by $NF with the following data: 000 0 00 NUL '\0' (null character) 100 64 40 @ 001 1 01 SOH (start of heading) ...
Mathew's user avatar
  • 235
-1 votes
3 answers
128 views

Replace new lines at end for those lines whose 2nd field in colon separated file span over to multiple line

I have got multiple files with different set of contents but the pattern is same across mulitple lines in different files. sample input file : cat -n test.txt 1 adjfkhjdhfkjd: ...
Ashis Sahoo's user avatar
1 vote
3 answers
556 views

AWK if condition

Given a file input.txt with contents: xyMnZnMs76XHgsWJ size: YN3hxFqFeUxXUMrQ USPqYEnbquwFhUW7 vhJJduHZu25qHzTw size: JaTDhQYQtLRWBLjj uE5HGDAZN6DHU73Q QxVPfQQaKTgR9LXQ grnsm8bwSz5fJbQS size: ...
wjwrpoyob's user avatar
  • 420
0 votes
0 answers
49 views

Help with creating a bash code to compare two different files

I'm currently encountering some challenges while working on a bash script in the Linux terminal to perform the following tasks: Compare the values in the third column of two different files line by ...
duduzba's user avatar
0 votes
4 answers
97 views

awk command to map ASCII char to hex code not working

I'm experimenting with awk and I thought it would be a cool idea to write a small program that maps every ASCII char with its hex code. this is what ive done so far: The string is: ...
Mathew's user avatar
  • 235
0 votes
0 answers
38 views

awk cmd to find n! not working

I'm not sure why the following awk cmd isn't working: seq 10 | awk 'BEGIN {print "---Numbers---"} {total*=$1} 1; END {print "---Total-----\n" total} ' the result is: ---Numbers--- ...
Mathew's user avatar
  • 235
0 votes
1 answer
60 views

compare 2 csv files using specific column and ignore specific string

My OS: RHEL7.X I've a csv file with 3 columns, like so - #######Category: Fruit Berries are fruit,04415bef-8c82-4672-848a-9ac80035040a,"Blah/Blah And/More/Blah" Apple is a fruit,073c6bf3-...
Raman Kathpalia's user avatar
-3 votes
4 answers
111 views

Print between two patterns on a single line

I have a file with the following contents: <alarm="yes"> <moduleId>aaa</moduleId> <documentation> <description>bbb</description> <Object>ccc</...
ashutosh's user avatar
-1 votes
2 answers
239 views

Filtering out values in a column that are zero

I have the following dataset: 071,36,035,08422,46217,00000,1,A,Broadalbin village (pt.),New York,18,46,46,46,45,45,44,46,44,46,43,43 071,36,045,29322,56209,00000,1,A,Glen Park village (pt.),New York,...
Mathew's user avatar
  • 235
0 votes
1 answer
69 views

Using awk -v with Shell variable works great and prints to terminal, but when redirect to a file only the 1st record prints to the file

I am reading a file with a matching pattern then getting the first five lines after the matching pattern. The patterns are not in proper numeric order, so I am using a numeric variable in the pattern ...
Joey Simon's user avatar
0 votes
4 answers
83 views

Using variables in gawk for the repeat interval value of an interval expression

OS Distribution: Ubuntu 22.04.3 LTS gawk version: GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1) I have a text file that every once in a while a line will have a significant amount of white ...
NeilB's user avatar
  • 3
1 vote
5 answers
184 views

Duplicate part of a line to another part

I would like to copy the first part (IxoscaEVm****t1_, without the '.p[number]') of the line starting with ">" and paste a before the last ":" of the same line. Input: >...
alex kiarie's user avatar
1 vote
2 answers
119 views

Recursive pattern search - output formating: for each match file print out the specific file name "\n", line number, sentence matched "\n" in colors

I want a file content search command, under Linux so that: it will search on specified files, e.g: md, txt, htm. recursively from the folder and its subfolders, e.g: . the content search can be ...
user2718593's user avatar
0 votes
3 answers
100 views

How to subtract column values by the first row with awk?

The input file is lines of "id number1 number2 count", both number1 and number2 are ascendant. Barcode000000 48521 1855 0 Barcode000001 48621 1855 0 Barcode000002 48721 1855 1 Barcode000003 ...
Galaxy's user avatar
  • 237
-4 votes
1 answer
81 views

I need to get value of a certain Row and Column in AWK

I need only to get the value of WOX which is just down it and it is in this Example equal 4 IOT CND FO WZX END 0 0 0 0 WOX SR POW LIO LIO2 4 00 0 FDR OTY M 0 I tried ...
hesham's user avatar
  • 1
2 votes
5 answers
855 views

Need to Replace multiple spaces between PIPE symbols with no space

Given a pipe-delimited file: 1|123|A |Normal Behaviour Exhibit 2|345|B|Embedded|delimiter 3|678|D|dimension 1" 4| | | | nvalue 5| | | |Missing cvalue I need to use Unix tools like sed or ...
pchegoor's user avatar
  • 131
1 vote
4 answers
155 views

Read nth column iteratively in two large files, cut and paste side by side to create new nth file

I have two files with around 100,000 space-separated columns in each. I want to extract each with column of the two files, write to a separate file and do command 2. Example for a file with two ...
AMS's user avatar
  • 37
1 vote
5 answers
694 views

awk on date conditions

I am trying delete rows from a person.csv (below) for the condition that person who are not born in last 1 year: Dataset1: "Index","User Id","First Name","Last Name&...
zoomer's user avatar
  • 21
1 vote
3 answers
218 views

How to replace strings in matching column?

suppose here is the file abc,def,ghi 1,a,zeta 2,b,beta 3,c,ceta 4,d,xaq 5,gh,lpa Above is just an example but in real there are many columns and I need to replace column strings (e.g having heading ...
Nirmal Singh's user avatar
0 votes
1 answer
103 views

Need Help with Date expression - AWK in a csv file

Situation: csv file has date column with different date formats, I want to convert them into a unambiguous date format (like +"%m-%d-%Y"). Example File data : File name = Date_Test_new.csv 3/...
ArielUnix2023's user avatar
4 votes
5 answers
638 views

Replace a block of spaces with a comma

I have a file in txt format that I want to convert to csv. Between each field there are blocks of spaces. The amount of spaces is not the same between every field. Is there a command with sed or awk ...
pwrsheller's user avatar
1 vote
4 answers
86 views

Remove duplicates by adding numerical suffix

How do I append a numerical suffix to lines to remove duplicates? Pseudo code: if currLine.startsWith("tag:") x = numFutureLinesMatching(currLine) if (x > 0) currLine = currLine + ...
Steven's user avatar
  • 738

1
2 3 4 5
160