Questions tagged [awk]

A pattern-directed scanning and processing language.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

awk regular expression replace any figure by “”, but do not so for numbers between brackets

Input shows: (TJ)-8.5(o12)-3.2(p1)-15.3(ik)-1.2(1) Result has to be: (TJ)-.(o12)-.(p1)-.(ik)-.(1) Figures of any number of digits contained by brackets should'nt be replaced, but all other figures ...
0
votes
2answers
12 views

awk gensub regex matching group between square brackets

Input from record $0: -0.005 Tc 0.005 Tw [(T)-8.5(o)-3.2(p)-15.3(ik)]TJ Output into /1 with gensub please: (T)-8.5(o)-3.2(p)-15.3(ik)
7
votes
5answers
98 views

How do I remove the first 300 million lines from a 700 GB txt file on a system with 1 TB max disk space?

The answers I found use sed, tail, head: How do I delete the first n lines of an ascii file using shell commands? https://askubuntu.com/questions/410196/remove-first-n-lines-of-a-large-text-file But ...
1
vote
1answer
29 views

sed + how to append complex line after match specific line

We want to append with sed , the following complex line after matching the line that start with – number_of_cars_are more /tmp/orig_file number_of_cars_are=$number_of_cars_are the line that we want ...
5
votes
2answers
21 views

awk: split file by column name and add header row to each file

I have a pipe delimited file a.txt which includes a header row. The first column holds a filename. I would like to split a.txt into several different files - the name of which is determined by the ...
2
votes
1answer
199 views

Why does “awk … file | tee file” truncates the file?

Why does this pipeline truncate the file awk '...' file | tee file while cating it first doesn't ? cat file | awk '...' | tee file The file is just a regular config file with 100s of lines.
-2
votes
3answers
35 views

Print line in reverse order if condition is meet

Suppose we have the following file: John FRIEND_OF Jose Aspirin INTERACTS_WITH Penicilin Tree COEXISTS_WITH Grass Andrew FIEND_OF Jane John INTERACTS_WITH Paula I need to print the line in reverse ...
0
votes
4answers
52 views

Changing content format

I have a text file which has 1000 rows/entries. These 1000 entries correspond to the column names for my dataframe. These are the new features that need to be added to the dataframe for model ...
5
votes
6answers
303 views

How to sort a first file (csv) based on the second file keys

I am trying to find a solution where I can sort my first file using the first column based on the second file keys First file example (file1.csv) COLUMN1 COlUMN2 apple fruit dog animal cat animal cow ...
2
votes
6answers
316 views

How to make the first character following a semicolon uppercase, for each line of a text file, using terminal

I have a txt file like this John Doe;john Doe is ...;he lives in ... Mike Nelson;mike Nelson works for ...;he makes ... Marcy William;marcy's mother is ...;marcy travels a lot... I want to convert ...
0
votes
1answer
55 views

Cannot replace multiple whitespaces with one

I'm trying to remove double spaces and replace them with just one. So far I've tried: sed -i 's/ / /g' ... sed -i 's/[[:space:]][[:space:]]*/ /g' ... tr -s ' ' But none work. Then when I go to the ...
3
votes
2answers
111 views

Awk: Scripts on the path?

Is there a way to put little awk scriptoids on the path? For example, I have this really useful collation operation: // collate-csv.awk FNR > 1 || NR == 1 And I can use it in all sorts of great ...
-2
votes
4answers
42 views

when found blank line change to new column

I have a data as: q w e e r r t 23 21 1 Whenever we find blank line change it to new column. Output q,e,23 w,r,21 e,r,1 ,t, I have tried but neither with sed nor I ...
1
vote
2answers
27 views

I'd like to extract specific fields from syslogs using grep or awk

Here's an example of syslogs that I'm working on. I'm trying to extract the account_id and version_apk fields from the syslogs: Sep 16 06:59:16 as09 janus-server[as09][21840]: INFO: janus.application....
-1
votes
3answers
26 views

How to grep lines for a specific profile starting with [default]

I'm trying to parse the lines of an .aws/config or .aws/credentials file of a specific profile such as [default] or [prod]. File example [default] region = us-east-1 output = json [nonprod] region = ...
7
votes
2answers
200 views

regular awk - easily sort array indexes to output them in the chosen order

[edit: clarified that I need an in awk solution, and corrected that I need to sort 'indexes' (or rather, output them in a sorted way) instead of the ambiguous 'values')] In awk, I often count things, ...
-2
votes
1answer
45 views

How to replace a column with Awk with a column from ANOTHER file

I need to replace the $6 from file A with the $147 from file B. Can anyone help me with the syntax? I tried with cut -d, f6 fileA | paste -d, f157 fileB but it doesnt work. I tried also awk '$6 fileA ...
-1
votes
1answer
20 views

How to output a regex in a second file using awk

I have a template file A.txt (sep = \t ): File Name novaprime-ct043904-TB_2140.pcrd Created By User admin Notes ID Run Started 09/17/2020 01:19:12 UTC Run Ended 09/17/2020 03:12:22 UTC Sample ...
0
votes
0answers
18 views

Passing directory path as arguments and extract columns from files to make a new file with horizontal concatenation

I need a shell script which takes in the path of directories as arguments. The directory contains similar named files for eg. file1.dat, file2.dat etc. I need to take file1.dat from all the ...
2
votes
1answer
28 views

Replace value on a particular column in pipe delimited file at specific row – sed or awk?

I have a requirement to replace a specific column value (identified by location) with another value on a pipe-delimited file at specific row. sample file: a|abc|val|sum|temp bs|run|win|can|done jump|...
1
vote
2answers
32 views

How to remove duplicate paragraphs

Given a file with paragraphs (group of lines separated by a blank line). It is guaranteed that the line breaks within equivalent paragraphs are conserved. Is there a way of using uniq to remove the ...
1
vote
1answer
14 views

Compare two files and get the matching rows based on two columns

I have two pipe delimited file like below file1.txt A1234|JESSIE|OPTED A1224|JOHN|OPTED L1212|RAMSAY|OPTED L1832|TIZEN|TESTED file2.txt A1234|B1465 G1211|L1211 G1241|L1212 G1271|L1232 Desired output ...
0
votes
3answers
26 views

Find columns based on contents of respective first rows

So I have a TSV file that looks something like this: Hello world how are you 1 2 3 4 5 6 7 8 9 0 (I've separated everything within each row with a single whitespace in the hand-written toy example ...
0
votes
2answers
23 views

Uniq based on last field, keeping last line, and append number of duplicates

I need some help to implement awk command in a bash script for implementing below logic. Tried sort/uniq combination, but no luck. I have a long list with time stamp and some number in front of it ...
-2
votes
2answers
31 views

Print value between commas from text file

How to print a specific value between commas from text file? There are several lines of this type in the file: 0.9999899864,0.6666600108,0.00,0.00,0.00,36988,140920,1,150.00,1500.00,1400.00,1300.00,1,...
1
vote
2answers
53 views

How to combine two files by matching two parameters with awk

I have a file A.txt (sep = \t ): Cycle Well Value Target 1 A1 5.07368111264623 EC 1 A1 3.06982862746599 FT 1 A1 2.46545646544623 EC And a second file B.txt (sep = \t, first ...
-1
votes
4answers
66 views

File transformation

I have a two column file of the format;- A,val1 A,val2 A,val3 B,val1 B,val2 B,val3 What I need is to be able to transform the output of this so that the second column is horizontal for each unique ...
1
vote
3answers
66 views

How to transform multiple column into successive lign in awk

I have a file A.txt (sep = "\t" , first column is empty): Cycle A1 A2 B1 1 4 5 2 2 7 3 4 3 3 2 5 And I want to transform this file into a new ...
0
votes
1answer
53 views

Awk to filter content of a file using the line number of another file

I am trying to use awk to filter some content from a file based on the content of another file. But it is not working. I got two files, orig_data and noghest_data. Both files have same number of ...
0
votes
3answers
83 views

Improvement of a bash script to make some basic statistics on files [closed]

I have some files like this: file_1 chrV 20924149 chrX 17718866 chrIV 17493793 chrII 15279345 chrI 15072423 chrIII 13783700 chrM 13794 file_2 chrI 230218 chrII 813184 chrIII ...
3
votes
5answers
183 views

Comparing two tab delimited files on the basis of first column of the both the files and print matching and non matching entries in output file

I have two tab-delimited files(fileA.txt and fileB.txt), I have to compare the first column of fileA.txt with the first column of fileB.txt and also I want to print the values present in the second ...
-1
votes
1answer
31 views

sort - prepend each group that start with the same letter in sorted list with an arbitrary string

Given the following list (input): apple banana blueberry mango melon I want the following output: == A == apple == B == banana blueberry == M == mango melon Is this possible to do in a shell ...
-1
votes
0answers
26 views

select rows based on column values and skip values to next higher values if desired values are unavailable

I understand how to select rows in a file based on specific values of a particular column. The code is below. Thanks to @terdon. awk 'NR==FNR{a[$1]++; next} $1 in a' wantedNums infile.txt > ...
0
votes
1answer
27 views

How to use awk with patterns for specific fields?

I am going through a file that has lines like the below: project: random_project_name rf1 rf2 rf3 rf4 rf5 rf6 rf7 rv1 rv2 rv3 rv4 rv5 rv6 rv7 other random lines where 2., 3. and 4. always come ...
0
votes
1answer
39 views

Escaping nested double-quotes in a corrupted CSV file

I have a large corrupted "CSV" file with many nested double-quotes. For instance: 123,"I wonder how to escape "these" quotes with backslashes.",123,456 456,"I wonder ...
0
votes
1answer
35 views

select rows based on specific column values [duplicate]

How can I select all rows in a text file based on different values of a particular column? The desired values are not in a specific interval. A short example of what I expect is given below. My data ...
-2
votes
7answers
117 views

Add '/' around every character of a string

How can we use sed or awk to translate unix to /u/n/i/x/. I was able to do this with Python as follows: separator='/' separator.join(string)+separator But how can we achieve this in Shell?
1
vote
2answers
40 views

Awk script not catching hash symbol “#”

How can I get awk to ignore records begining with a comment while keeping all other records? The below awk script does not ignore the records beginning with a hash symbol in this sample test file. It ...
8
votes
11answers
515 views

How do I print the last sequence of lines between a start and an end pattern?

Answers to this question: How to grep lines between start and end pattern? are not concerned with multiple sequences of lines which fall between the match patterns. Thus, for example, sed -n '/...
0
votes
3answers
61 views

Extract a list of strings from a file, only from first column

I wanted to extract a list of numbers (string.txt) from masterFile.list. masterFile.list is separated by | and contained more than one column. I am only interested with the line, where its first ...
0
votes
3answers
49 views

extract virtualhosts to individual files based on virtual hostname [closed]

I want to extract each VirtualHost from /etc/apache2/sites-enabled/default-ssl.conf and put then into their own config files there are lot of entries in 1 single file and managing them is becoming ...
2
votes
2answers
105 views

awk using time difference in miliseconds in the report

I need to parse a log file that looks as follows. 09/03/2020 00:05:03.364 Aggregated 1000 NMEs at a rate of 0 NMEs/sec 09/03/2020 00:05:03.366 Scheme S20_SessionClassAggregation tree contained 1000 ...
0
votes
0answers
21 views

weird errors at boot and shutdown

I am running Adelie Linux, an independent distro based on musl and openRC as service manager, on a x86_64 laptop. My desktop environment is KDE Plasma 5.18. When the system is shutting down, I notice ...
-1
votes
2answers
45 views

duplicate rows in fileds

i have a file like this one 4123 4179 3275 3317 4137 4193 3331 3373 4151 4207 3387 3429 3443 3485 3499 3541 ...
-1
votes
1answer
27 views

How to merge data based on first column?

I have some simple data: digital 5434 ero 3464 3es 69955 digital 1948 ero 5783 ride 5884 I want to process this input file and sum all 2nd field values where the first field is the same. My final ...
-1
votes
6answers
70 views

Change all positive values to negative and vice versa but in two different files

I have two data files one.txt and two.txt on a Linux system. I want to convert all positive values to negative in one.txt and vice versa for two.txt, but only on the first column. Please note that ...
0
votes
2answers
56 views

No manual entry for awk

How comes running man awk gives No manual entry for awk? I frankly have never personally installed Awk on my Archlinux systems, but I've just found on the current one, the last I installed, there's no ...
0
votes
2answers
63 views

Parse a YAML section using shell

I have a YAML that looks like something: - whatever: - something - toc: 4 - body: assets/footer.html pkg: - pkg_a_1: - Shass - AJh55 - ASH7 - pkg_b_1: - Kjs6 - opsaa other: ...
0
votes
2answers
26 views

Print only one value from duplicates [duplicate]

I have following content in a file. $ cat file.txt code-coverage-api jsch cloudbees-folder apache-httpcomponents-client-4-api apache-httpcomponents-client-4-api jsch apache-httpcomponents-client-...
0
votes
4answers
244 views

Remove comments from file while ignoring quoted comment signs

I would like to remove comments starting with # from a file. I have tried the simpler approaches described in How can I remove all comments from a file? but I have a few additional rules: A # does ...

1
2 3 4 5
132