Questions tagged [awk]

A pattern-directed scanning and processing language.

Filter by
Sorted by
Tagged with
0 votes
0 answers
46 views

Text processing grep and awk together on live script not working

I am using ssh to login to a remote server (Ubuntu), and then issue a command which outputs text/value each sec (live script). In the live output, I want to filter out the lines containing '...
SAM's user avatar
  • 1
1 vote
2 answers
88 views

awk - combine 2 steps into one

I need to read the output table of transmission-remote to determine if a download is taking place. This is currently achieved with trans="transmission-remote 0.0.0.0:9092 -n $user:$pass -l" [...
Tony Puryer's user avatar
2 votes
7 answers
546 views

How to remove embedded newlines from CSV fields

I am struggling with processing a file. Normally I should have a line for each host definition. But sometimes, someone split some fields on different lines. Here is an example: "host1","...
Nicolas Greder's user avatar
-3 votes
4 answers
103 views

How to catch all lines of a repeating pattern and do some actions with the subresults

I am looking for a possibility to catch in a repeating text pattern all variable amount of lines between them and then do an action with it in bash. Example text: Total: text1 text2 Total: text3 Total:...
André Letterer's user avatar
1 vote
2 answers
127 views

syntax to delete lines

I use this syntax to delete lines from 2.txt to 1.txt: awk 'NR==FNR{a[$0]=1;next}!a[$0]' 2.txt 1.txt > lines.txt My lines are in this format: [email protected]:something If this line is identical ...
Abracadabra's user avatar
1 vote
6 answers
116 views

manipulate ls text output to add path to filenames

I get sometimes files with following ls output format: /etc/cron.d: -rw-r--r-- 1 root root 128 May 15 2020 0hourly -rw------- 1 root root 235 Dec 17 2020 sysstat /etc/cron.daily: -rw------- 1 root ...
André Letterer's user avatar
-1 votes
1 answer
68 views

Script to replace matching pattern between file 1 (rev.fasta) and file 2 (necessary_files/index_rev.fasta) IN FILE 1 with information from file 2

I'm a molecular biologist who just recently started working with bash and is trying to write a couple of scripts to automate some things that I tipically have to do manually. I have two files (rev....
Joana's user avatar
  • 29
1 vote
3 answers
97 views

sed to replace in a file, with both old and new strings in files

I want to automatically comment out a code block in PHP file, as below: The original block: // Enable all errors ini_set('display_startup_errors', 1); ini_set('display_errors', 1); ...
alancc's user avatar
  • 213
0 votes
3 answers
106 views

Remove all lines between two similar patterns if a specific pattern in between is not matched

The title should be: Remove all lines between two similar patterns, including the line of the starting pattern. if another specific pattern in between is not matched. I made it shorter because its ...
pwrsheller's user avatar
2 votes
2 answers
60 views

Listing a certain part of filenames and their contents

I have a Linux machine with BusyBox, which has a directory called /data/var/lib/connman.  That directory holds some directories that I'm not interested in.  But it contains many .config files that ...
Norbert's user avatar
  • 121
0 votes
5 answers
152 views

How to add content to a file before the last } character using Busybox utilities?

There is a file with contents { "first_name": "John", "last_name": "Smith", "is_alive": true, "age": 27, "address": { ...
Maxim's user avatar
  • 119
-2 votes
1 answer
64 views

Problem with variable in awk

I want to replace the text sd in the following command with a variable: lsblk -r --output NAME,MOUNTPOINT | awk -F \/ '/sd/ { dsk=substr($1,1,3);dsks[dsk]+=1 } END { for ( i in dsks ) { if (dsks[i]==...
Peter Talen's user avatar
5 votes
5 answers
658 views

Remove lines from a csv file based on column value

I have a csv file with 12 million lines in the following format: mcu_i,INIT,200,iFlash, 11593925, 88347,,0x00092684,r,0x4606b570, ok,, 32,single,op-c,0,, 0, 0, 0,...
raphui's user avatar
  • 84
0 votes
0 answers
37 views

Is it safe to redirect output of the awk command to the same file used for input? [duplicate]

In this comment, community member DannyDannyDanny asked if it is safe to redirect output of the awk command to the same file being used for input. No one answered that comment, so I would like to ...
End Anti-Semitic Hate's user avatar
2 votes
5 answers
902 views

Replace new lines with spaces using awk

I have a text file that I generated of all files in a directory. I'd like to use this file as input into a script that I have, but I need the text file to be formatted in a particular way to be parsed ...
lovelyrubbish's user avatar
4 votes
2 answers
417 views

awk (and gawk) : how to avoid fatal error when 1 of the N input files is unreadable

test case: (with a non-root user, as root would disregard the 000 permission...) #in a clean directory: [ -f file_1 ] && chmod 600 file_? # for repeat tests... for i in file_1 file_2 file_3; ...
Olivier Dulac's user avatar
2 votes
3 answers
88 views

Extract a string from a searched column index on the row below?

I'm trying to make a simple disk usage shell script (in dash), but unsure how to parse this string because I'm not familiar with AWK's syntax. I use this command: df -P -BG /, and, for example, it ...
mrjpaxton's user avatar
-1 votes
4 answers
133 views

awk command to extract columns from a csv file whose header contains a certain substring

I have a huge csv file that is of the format aaa1, "aaa2, aa214", aa21, "aa, a14", aa211, aa44, aaa445 data, data, data, data, data, data, data, ......................................
User's user avatar
  • 254
6 votes
3 answers
515 views

awk: sort by first column then second; output unique 1st column once but all 2nd column

I have a two-column file that you can create as follows cat > twocol << EOF 007 03 001 03 003 01 137 12 001 11 002 01 002 02 002 03 001 02 002 04 137 94 010 21 001 01 EOF The resultant file, ...
bballdave025's user avatar
-2 votes
4 answers
140 views

I need to delete lines when pattern match

I have file on a Linux system, aa.text, which contains the lines below: $ cat aa.text I am a user ORA-1653: unable to extend table SYS.AUD$ by 128 in tablespace SYSTEM [PCRMR2S] Fri Feb 09 23:57:51 ...
ANW's user avatar
  • 7
-1 votes
3 answers
58 views

How to split a file based on new line and get just the second field in a file after splitting

Sample text file below: #This is a sample file ## version 3.0 - change 1 - change 2 ## version 2.5 - change 1 - change 2 - change 3 ## version 2.4 - change 1 - change 2 I'm only after the latest ...
Stacker's user avatar
1 vote
3 answers
121 views

Fetch data from a large file based on time and date recursively

Fetch data based on date(column 1) and time (column 2). Each date has time in column 2. Based on each date in column1 two files are to be generated which will have all the fields based on time 6:00 to ...
Gavin's user avatar
  • 91
2 votes
2 answers
73 views

Problems with using AWK with XARGS

I'm having trouble with the following command --> find . -type f -name 'out*' | xargs awk 'BEGIN{print "Filename, Energy"}/TOTAL ENERGY/{print FILENAME, "," $4}' >> ...
dtx1's user avatar
  • 21
0 votes
2 answers
80 views

I need to print value from 3rd column when 4th column condition is matched

cat Data.log REPLICAT RUNNING RPGUJ3 02:00:12 00:00:06 REPLICAT RUNNING RPGUJ4 00:00:10 00:00:06 REPLICAT RUNNING RPGUJ3_3 01:00:11 00:00:00 REPLICAT RUNNING RPGUJ4_3 00:00:12 00:00:06 I want ...
ANW's user avatar
  • 7
0 votes
2 answers
102 views

How to delete rows that contain 0 in the "last column" in a linux terminal?

I'm wondering how can I delete rows when the value in the last column is "0" in multiple .tsv files where the last column number might not be the same for all files. And total number of ...
Debajyoti Kabiraj's user avatar
1 vote
3 answers
114 views

print a text block / lines between pattern using grep or awk

My file file.txt looks like this: [NamesA] Andreas Alex [NamesB] Bernd Bruno [NamesC] Casper [NamesD] Doris I would like accomplish the following 3 different outputs, using grep or awk to use it ...
AKSNET's user avatar
  • 13
-3 votes
1 answer
74 views

How to use awk to get a special field from a file?

This is the output of ifconfig, I want to get only the number 78 (4th octet of the ip address) using awk ns33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.78 ...
Mahmoud Youssef's user avatar
-2 votes
2 answers
55 views

Iis possible to rewrite those string put the part of string starting with "# " above the string itself?

Simple and fast question. I have this awk -f file # awk use file as input awk '/word/' file # awk show me word awk '/32$/' text # all words with 32.. I want this # awk use file ...
elbarna's user avatar
  • 12.7k
0 votes
2 answers
90 views

Trying to figure out how this AWK script works

In my shop, we have a piece of AWK code that someone wrote sometime ago but did not document, so we have no idea how to interpret the code. awk -F'~"' ' {gsub(/~/, " ", $3); print $1&...
PandaGuy1961's user avatar
2 votes
5 answers
422 views

How to split a given column's string values in a text file

I have a text file on a Linux machine with two columns: Column 1 = id_no (most are 5, with some 6 digits long); Column 2 = genetic_markers (all are 50674 digits long); 12345 0102010205 54322 ...
Michiel Van Niekerk's user avatar
-1 votes
4 answers
104 views

Summarise comma delimited text file by merging based on column 2 and 3, and calculating the average of column 4

I have a file that reports every incident of something and its size. I'm trying to summarise that information by merging consecutive incidences which can be identified by column 3 of a line above ...
Giles's user avatar
  • 897
1 vote
2 answers
94 views

conditions within brace blocks

To make multiple passes on the same input file, this simple case works fine: $ awk 'p==1{print $1}; p==2{print $2}; p==3{print $3}' p=1 infile p=2 infile p=3 infile But fail with this change? $ awk '...
user9101329's user avatar
  • 1,103
1 vote
4 answers
110 views

adding first 5 characters of a file name to each line in the file

I have many ".txt" files in a folder. For each .txt file, I need to get the first 5 characters in the name of the file and add them to the beginning of each line in file. I also need to add ...
kaur's user avatar
  • 23
3 votes
7 answers
428 views

How can I print the longest sequence of lines featuring numbers smaller than a threshold?

I am learning Perl, but I don't know how to solve this problem. I have a .txt file of the following form: 1 16.3346384 2 11.43483 3 1.19819 4 1.1113829 5 1.0953443 6 1.9458343 7 1.345645 8 1....
69 420 1970's user avatar
0 votes
0 answers
104 views

Get right Pipewire object with object id

I need to know Pipewire object.serial that corresponds to a specific object. Parsing output of pw-cli list-objects is what I need, but I'm very poor with multiline regex. pw-cli list-objects gives ...
Basher's user avatar
  • 1
-3 votes
1 answer
158 views

Template for formatting output as a dynamic table in bash

I have a created a simple script which is using the ps aux command to give information for a process entered by a user, and presents the output in a table like format (something like the mysql shell ...
October's user avatar
0 votes
1 answer
69 views

How to only zcat files between two time durations?

Scenario: I've a file called abcd.log.gz. I want to zcat only the files from time durations 17:30:00 to 18:30:00. Currently, I zcat the entire file. Then awk the logs between two time durations which ...
barnyard9's user avatar
  • 157
0 votes
5 answers
360 views

print characters between a specific pattern containing <>

i am asking for your help please. I reviewed a lot of sources and did some testing with awk and sed, but i can't get it to work. Below is a snippet of a config file which i could get an output via ...
AKSNET's user avatar
  • 13
0 votes
1 answer
91 views

Bash script with input ipaddress file & csv file

I have a bash script with an ipaddress.txt input file, that I want to use against a csv file. I want find the IP and then display the values of Column A (for now). I may want to display two or three ...
burownidl's user avatar
4 votes
4 answers
353 views

Awk- Compare Numbers from Two Files and write Differences in New File

I have two lists with item numbers and want to mark the difference between these lists by writing the numbers wich aren't in both files in a new file. Both Files have the item number in column 2 and ...
user avatar
-1 votes
5 answers
262 views

Delete all lines after n lines below the line containing the first match

If a file contains the following: Line 1 FooLine 5 Line 2 Line 55 Line 3 Line 4 Line 5 Line 6 Line 7 Line 5 Line 8 Line 9 How to delete the lines below two lines after "Line 5", so that the ...
Sadi's user avatar
  • 495
5 votes
4 answers
511 views

find and replace specific string in multiple csv files

I have multiple csv files and the contents are in the following format: "TIMESTAMP",col2,col3,col4 "yyyy-mm-dd HH:mm",20,19,17 I want to replace - with / so that in the end I have:...
dante_t's user avatar
  • 151
5 votes
7 answers
1k views

AWK - SED - add a character to the beginning of every column if it starts with a number

How can I add a character to the beginning of every column that starts with a number: Example adding a c to: 4 r4 8 8 4 So I get this output: c4 r4 c8 c8 c4 It should work for a variable number of ...
nath's user avatar
  • 5,694
1 vote
4 answers
197 views

How to split with awk escaping all special characters

I'm trying to create a char array by using split, it works so far. The problem is when any character in the input string is preceded by \ . What happens is \ doesn't get considered as a char as it ...
KrOo Pine's user avatar
  • 111
4 votes
3 answers
520 views

Two ssh output as awk input

I have two remote servers that I am trying to ssh and cat some files.. I want to input the output of the ssh to a awk command. This is what I have got ssh username@host1 “cat /tmp/test/*” ssh ...
NecroCoder's user avatar
-1 votes
1 answer
177 views

Create new CSV files based on latest timestamp data

I have a script that accesses multiple dat files and produce csv files based on previous day’s data. these DAT files are updated on per minute bases with data from various instruments. Script snippet: ...
dante_t's user avatar
  • 151
5 votes
4 answers
401 views

Replace single quote at beginning of columns

I receive input files that are outside my control that contain a leading single quote in some columns, such as 'foo|'012|that's nice|bar I want to remove all leading single quotes from each field, to ...
enharmonic's user avatar
2 votes
4 answers
202 views

awk appends column in .gz file as seen with cat -A, but changes column name in regular output

Using awk, to this table I want to add a column where first row is "INFO" and rest of the rows are all "1". $ gunzip -c foo.gz | head SNPID CHR BP Allele1 Allele2 Freq1 Effect ...
jay.sf's user avatar
  • 227
0 votes
4 answers
105 views

Copy matching lines to the next line and modify?

Given a file like this: a foo 1 b bar 2 c foo 3 d baz 4 e bar 5 I want to modify that file as follows: copy every line containing bar to the following line and change bar to bar2, so that the result ...
Nathan Long's user avatar
  • 1,623
-1 votes
1 answer
46 views

Returning lines from a JSON log file which contain IP and timestamps read from a CSV file?

so I'm new to Shell Scripting and could use some help. So in my CSV file, I want to see if IP and timestamp values exists in some way in the line entries of the JSON log file, and if so, return that ...
Linuxnoob's user avatar

1
2 3 4 5
161