Questions tagged [awk]
A pattern-directed scanning and processing language.
6,574
questions
0
votes
0answers
8 views
awk regular expression replace any figure by “”, but do not so for numbers between brackets
Input shows: (TJ)-8.5(o12)-3.2(p1)-15.3(ik)-1.2(1)
Result has to be: (TJ)-.(o12)-.(p1)-.(ik)-.(1)
Figures of any number of digits contained by brackets should'nt be replaced, but all other figures ...
0
votes
2answers
12 views
awk gensub regex matching group between square brackets
Input from record $0:
-0.005 Tc 0.005 Tw [(T)-8.5(o)-3.2(p)-15.3(ik)]TJ
Output into /1 with gensub please:
(T)-8.5(o)-3.2(p)-15.3(ik)
7
votes
5answers
98 views
How do I remove the first 300 million lines from a 700 GB txt file on a system with 1 TB max disk space?
The answers I found use sed, tail, head:
How do I delete the first n lines of an ascii file using shell commands?
https://askubuntu.com/questions/410196/remove-first-n-lines-of-a-large-text-file
But ...
1
vote
1answer
29 views
sed + how to append complex line after match specific line
We want to append with sed , the following complex line after matching the line that start with – number_of_cars_are
more /tmp/orig_file
number_of_cars_are=$number_of_cars_are
the line that we want ...
5
votes
2answers
21 views
awk: split file by column name and add header row to each file
I have a pipe delimited file a.txt which includes a header row. The first column holds a filename.
I would like to split a.txt into several different files - the name of which is determined by the ...
2
votes
1answer
199 views
Why does “awk … file | tee file” truncates the file?
Why does this pipeline truncate the file
awk '...' file | tee file
while cating it first doesn't ?
cat file | awk '...' | tee file
The file is just a regular config file with 100s of lines.
-2
votes
3answers
35 views
Print line in reverse order if condition is meet
Suppose we have the following file:
John FRIEND_OF Jose
Aspirin INTERACTS_WITH Penicilin
Tree COEXISTS_WITH Grass
Andrew FIEND_OF Jane
John INTERACTS_WITH Paula
I need to print the line in reverse ...
0
votes
4answers
52 views
Changing content format
I have a text file which has 1000 rows/entries. These 1000 entries correspond to the column names for my dataframe. These are the new features that need to be added to the dataframe for model ...
5
votes
6answers
303 views
How to sort a first file (csv) based on the second file keys
I am trying to find a solution where I can sort my first file using the first column based on the second file keys
First file example (file1.csv)
COLUMN1 COlUMN2
apple fruit
dog animal
cat animal
cow ...
2
votes
6answers
316 views
How to make the first character following a semicolon uppercase, for each line of a text file, using terminal
I have a txt file like this
John Doe;john Doe is ...;he lives in ...
Mike Nelson;mike Nelson works for ...;he makes ...
Marcy William;marcy's mother is ...;marcy travels a lot...
I want to convert ...
0
votes
1answer
55 views
Cannot replace multiple whitespaces with one
I'm trying to remove double spaces and replace them with just one. So far I've tried:
sed -i 's/ / /g' ...
sed -i 's/[[:space:]][[:space:]]*/ /g' ...
tr -s ' '
But none work. Then when I go to the ...
3
votes
2answers
111 views
Awk: Scripts on the path?
Is there a way to put little awk scriptoids on the path?
For example, I have this really useful collation operation:
// collate-csv.awk
FNR > 1 || NR == 1
And I can use it in all sorts of great ...
-2
votes
4answers
42 views
when found blank line change to new column
I have a data as:
q
w
e
e
r
r
t
23
21
1
Whenever we find blank line change it to new column.
Output
q,e,23
w,r,21
e,r,1
,t,
I have tried but neither with sed nor I ...
1
vote
2answers
27 views
I'd like to extract specific fields from syslogs using grep or awk
Here's an example of syslogs that I'm working on. I'm trying to extract the account_id and version_apk fields from the syslogs:
Sep 16 06:59:16 as09 janus-server[as09][21840]: INFO: janus.application....
-1
votes
3answers
26 views
How to grep lines for a specific profile starting with [default]
I'm trying to parse the lines of an .aws/config or .aws/credentials file of a specific profile such as [default] or [prod].
File example
[default]
region = us-east-1
output = json
[nonprod]
region = ...
7
votes
2answers
200 views
regular awk - easily sort array indexes to output them in the chosen order
[edit: clarified that I need an in awk solution, and corrected that I need to sort 'indexes' (or rather, output them in a sorted way) instead of the ambiguous 'values')]
In awk, I often count things, ...
-2
votes
1answer
45 views
How to replace a column with Awk with a column from ANOTHER file
I need to replace the $6 from file A with the $147 from file B. Can anyone help me with the syntax?
I tried with
cut -d, f6 fileA | paste -d, f157 fileB
but it doesnt work. I tried also
awk '$6 fileA ...
-1
votes
1answer
20 views
How to output a regex in a second file using awk
I have a template file A.txt (sep = \t ):
File Name novaprime-ct043904-TB_2140.pcrd
Created By User admin
Notes
ID
Run Started 09/17/2020 01:19:12 UTC
Run Ended 09/17/2020 03:12:22 UTC
Sample ...
0
votes
0answers
18 views
Passing directory path as arguments and extract columns from files to make a new file with horizontal concatenation
I need a shell script which takes in the path of directories as arguments. The directory contains similar named files for eg. file1.dat, file2.dat etc. I need to take file1.dat from all the ...
2
votes
1answer
28 views
Replace value on a particular column in pipe delimited file at specific row – sed or awk?
I have a requirement to replace a specific column value
(identified by location)
with another value on a pipe-delimited file at specific row.
sample file:
a|abc|val|sum|temp
bs|run|win|can|done
jump|...
1
vote
2answers
32 views
How to remove duplicate paragraphs
Given a file with paragraphs (group of lines separated by a blank line).
It is guaranteed that the line breaks within equivalent paragraphs are conserved.
Is there a way of using uniq to remove the ...
1
vote
1answer
14 views
Compare two files and get the matching rows based on two columns
I have two pipe delimited file like below
file1.txt
A1234|JESSIE|OPTED
A1224|JOHN|OPTED
L1212|RAMSAY|OPTED
L1832|TIZEN|TESTED
file2.txt
A1234|B1465
G1211|L1211
G1241|L1212
G1271|L1232
Desired output
...
0
votes
3answers
26 views
Find columns based on contents of respective first rows
So I have a TSV file that looks something like this:
Hello world how are you
1 2 3 4 5
6 7 8 9 0
(I've separated everything within each row with a single whitespace in the hand-written toy example ...
0
votes
2answers
23 views
Uniq based on last field, keeping last line, and append number of duplicates
I need some help to implement awk command in a bash script for implementing below logic. Tried sort/uniq combination, but no luck.
I have a long list with time stamp and some number in front of it ...
-2
votes
2answers
31 views
Print value between commas from text file
How to print a specific value between commas from text file? There are several lines of this type in the file:
0.9999899864,0.6666600108,0.00,0.00,0.00,36988,140920,1,150.00,1500.00,1400.00,1300.00,1,...
1
vote
2answers
53 views
How to combine two files by matching two parameters with awk
I have a file A.txt (sep = \t ):
Cycle Well Value Target
1 A1 5.07368111264623 EC
1 A1 3.06982862746599 FT
1 A1 2.46545646544623 EC
And a second file B.txt (sep = \t, first ...
-1
votes
4answers
66 views
File transformation
I have a two column file of the format;-
A,val1
A,val2
A,val3
B,val1
B,val2
B,val3
What I need is to be able to transform the output of this so that the second column is horizontal for each unique ...
1
vote
3answers
66 views
How to transform multiple column into successive lign in awk
I have a file A.txt (sep = "\t" , first column is empty):
Cycle A1 A2 B1
1 4 5 2
2 7 3 4
3 3 2 5
And I want to transform this file into a new ...
0
votes
1answer
53 views
Awk to filter content of a file using the line number of another file
I am trying to use awk to filter some content from a file based on the content of another file. But it is not working.
I got two files, orig_data and noghest_data. Both files have same number of ...
0
votes
3answers
83 views
Improvement of a bash script to make some basic statistics on files [closed]
I have some files like this:
file_1
chrV 20924149
chrX 17718866
chrIV 17493793
chrII 15279345
chrI 15072423
chrIII 13783700
chrM 13794
file_2
chrI 230218
chrII 813184
chrIII ...
3
votes
5answers
183 views
Comparing two tab delimited files on the basis of first column of the both the files and print matching and non matching entries in output file
I have two tab-delimited files(fileA.txt and fileB.txt), I have to compare the first column of fileA.txt with the first column of fileB.txt and also I want to print the values present in the second ...
-1
votes
1answer
31 views
sort - prepend each group that start with the same letter in sorted list with an arbitrary string
Given the following list (input):
apple
banana
blueberry
mango
melon
I want the following output:
== A ==
apple
== B ==
banana
blueberry
== M ==
mango
melon
Is this possible to do in a shell ...
-1
votes
0answers
26 views
select rows based on column values and skip values to next higher values if desired values are unavailable
I understand how to select rows in a file based on specific values of a particular column. The code is below. Thanks to @terdon.
awk 'NR==FNR{a[$1]++; next} $1 in a' wantedNums infile.txt > ...
0
votes
1answer
27 views
How to use awk with patterns for specific fields?
I am going through a file that has lines like the below:
project: random_project_name
rf1 rf2 rf3 rf4 rf5 rf6 rf7
rv1 rv2 rv3 rv4 rv5 rv6 rv7
other random lines
where
2., 3. and 4. always come ...
0
votes
1answer
39 views
Escaping nested double-quotes in a corrupted CSV file
I have a large corrupted "CSV" file with many nested double-quotes. For instance:
123,"I wonder how to escape "these" quotes with backslashes.",123,456
456,"I wonder ...
0
votes
1answer
35 views
select rows based on specific column values [duplicate]
How can I select all rows in a text file based on different values of a particular column? The desired values are not in a specific interval.
A short example of what I expect is given below.
My data ...
-2
votes
7answers
117 views
Add '/' around every character of a string
How can we use sed or awk to translate unix to /u/n/i/x/.
I was able to do this with Python as follows:
separator='/'
separator.join(string)+separator
But how can we achieve this in Shell?
1
vote
2answers
40 views
Awk script not catching hash symbol “#”
How can I get awk to ignore records begining with a comment while keeping all other records? The below awk script does not ignore the records beginning with a hash symbol in this sample test file. It ...
8
votes
11answers
515 views
How do I print the last sequence of lines between a start and an end pattern?
Answers to this question:
How to grep lines between start and end pattern?
are not concerned with multiple sequences of lines which fall between the match patterns. Thus, for example, sed -n '/...
0
votes
3answers
61 views
Extract a list of strings from a file, only from first column
I wanted to extract a list of numbers (string.txt) from masterFile.list. masterFile.list is separated by | and contained more than one column. I am only interested with the line, where its first ...
0
votes
3answers
49 views
extract virtualhosts to individual files based on virtual hostname [closed]
I want to extract each VirtualHost from /etc/apache2/sites-enabled/default-ssl.conf and put then into their own config files
there are lot of entries in 1 single file and managing them is becoming ...
2
votes
2answers
105 views
awk using time difference in miliseconds in the report
I need to parse a log file that looks as follows.
09/03/2020 00:05:03.364 Aggregated 1000 NMEs at a rate of 0 NMEs/sec
09/03/2020 00:05:03.366 Scheme S20_SessionClassAggregation tree contained 1000 ...
0
votes
0answers
21 views
weird errors at boot and shutdown
I am running Adelie Linux, an independent distro based on musl and openRC as service manager, on a x86_64 laptop. My desktop environment is KDE Plasma 5.18. When the system is shutting down, I notice ...
-1
votes
2answers
45 views
duplicate rows in fileds
i have a file like this one
4123 4179 3275 3317
4137 4193 3331 3373
4151 4207 3387 3429
3443 3485
3499 3541
...
-1
votes
1answer
27 views
How to merge data based on first column?
I have some simple data:
digital 5434
ero 3464
3es 69955
digital 1948
ero 5783
ride 5884
I want to process this input file and sum all 2nd field values where the first field is the same. My final ...
-1
votes
6answers
70 views
Change all positive values to negative and vice versa but in two different files
I have two data files one.txt and two.txt on a Linux system. I want to convert all positive values to negative in one.txt and vice versa for two.txt, but only on the first column. Please note that ...
0
votes
2answers
56 views
No manual entry for awk
How comes running man awk gives No manual entry for awk? I frankly have never personally installed Awk on my Archlinux systems, but I've just found on the current one, the last I installed, there's no ...
0
votes
2answers
63 views
Parse a YAML section using shell
I have a YAML that looks like
something:
- whatever:
- something
- toc: 4
- body: assets/footer.html
pkg:
- pkg_a_1:
- Shass
- AJh55
- ASH7
- pkg_b_1:
- Kjs6
- opsaa
other:
...
0
votes
2answers
26 views
Print only one value from duplicates [duplicate]
I have following content in a file.
$ cat file.txt
code-coverage-api
jsch
cloudbees-folder
apache-httpcomponents-client-4-api
apache-httpcomponents-client-4-api
jsch
apache-httpcomponents-client-...
0
votes
4answers
244 views
Remove comments from file while ignoring quoted comment signs
I would like to remove comments starting with # from a file. I have tried the simpler approaches described in How can I remove all comments from a file? but I have a few additional rules:
A # does ...
