New answers tagged

5 votes

How to check a value and change other key in JSON with jq?

To apply an expression to all elements of an array, and get an altered array out, use map(expression). Extracting all entries that have off status: $ jq 'map(select(.status == "off"))' file [...
  • 281k
1 vote

How to cross check one column in a file in another and print the missing values?

I would first sort the two files (excluding the header of course). I would then use awk or cut to select the first column (minus the header) and use comm to select the ones from file1 that are not in ...
  • 2,771
1 vote

How to cross check one column in a file in another and print the missing values?

If you are using a system that supports process substitution, you can use grep with the -v flag (show non-matching lines) and -f flag (read patterns from file), where the "file" is a command ...
  • 221k
0 votes

combine two text files with adding some separator between?

Using find's arguments -printf and -exec, combined with cat you can place text before and after a files contents. find . -type f -name "*.txt" -printf "/n This is before! \n" -...
  • 101
0 votes

Missing column after merging two files

Assuming the data in the two input files are "simple" CSV records without fields containing embedded delimiters or newlines, that the | characters are the delimiters, and that the files ...
  • 281k
3 votes
Accepted

Find highest value amongst columns and print value in a new column

It should be fairly easy: You can simply iterate over all fields and check which is the highest in this line. Then, append that highest value as (NF+1)th field. awk 'BEGIN{FS=OFS="\t"} {max=$...
  • 19.4k
1 vote

Missing column after merging two files

You could rely on join: join -t\| -j 1 -o 1.2,2.2,2.3,1.3 file2 file1 Where the format(-o) is defined as FILE.FIELD, so just a selection of which field to take from which input file, -t is used to ...
  • 11.3k
3 votes

Missing column after merging two files

The answer is rather simple: You use a[$3] to refer to the third column of file1. However you use the array a to store the second column of file1, and never the third column, and you only ever use ...
  • 19.4k
1 vote

Missing column after merging two files

awk 'BEGIN{FS=OFS="|"} NR==FNR {a[$1]=$2 OFS $3;next} ($1 in a) {print $2,a[$1],$3}' file1.txt file2.txt > file3.txt You never saved a[$3] before in your script so it doesn't find it. You ...
1 vote
Accepted

How to perform a "grouped merge" of three files, while omitting the first few lines of each?

Applying the DSU Idiom, using any versions of the mandatory POSIX tools awk, sort, and cut: $ cat tst.sh #!/usr/bin/env bash awk -v OFS='\t' ' FNR == 1 { fileNr++ } FNR >= 4 { print FNR-3, ...
  • 24.6k
2 votes
Accepted

Replace contents of one column in txt file by custom strings

If the other file has one string per line, like this: $ cat file2 John1 Mike2 Tom33 WolfN and you want to add them in the order they are found, you can do this: $ awk 'NR == FNR{a[NR]=$1;next}; {$(NF-...
  • 221k
0 votes

How to perform a "grouped merge" of three files, while omitting the first few lines of each?

If you don't mind using something else other than awk: for f in $(ls *.txt) ; do awk 'FNR >=4' $f | egrep "." -n ; done | sort -n | uniq | cut -d: -f2- would do the trick explanation: ...
0 votes

How to perform a "grouped merge" of three files, while omitting the first few lines of each?

i did save the patterns you gave above in three files. I got the desired output completting your awk-filtering on this way: for i in {4..15}; do awk "FNR == $i" *.txt | sort -u; done
1 vote

How to generate a new file by extracting some parts?

Using Raku (formerly known as Perl_6) ~$ raku -ne 'put $<> if m/ "Final SoC is " <(\d* \. \d*)> /;' file #OR ~$ raku -ne 'put $0 if m/ "Final SoC is " (\d* \. \d*) /...
  • 1,494
0 votes

Print ONLY the next line after a match with awk

Using Raku (formerly known as Perl_6) ~$ raku -e 'put $<> if m{ "Artifacts were created:" \n <( \N* } for lines.join: "\n";' file #OR ~$ raku -e 'put $0 if m{ "...
  • 1,494
2 votes

Print ONLY the next line after a match with awk

If sed is an option $ sed -En '/Artifacts were created:/{n;s/[^ ]* (.*)/\1/p}' <<< "$file" build-1.tar.gz
  • 2,164
2 votes
Accepted

Print ONLY the next line after a match with awk

Please try: awk '/Artifacts were created:/{p=1} NF{out=$2} END{if(p==1){print out}}' infile Although the pipe you presented is extracting the second filed of the line just following the match, not ...
2 votes

Print ONLY the next line after a match with awk

The original requirement asked for a value from the last non-whitespace line after a match. Here, when you find a match trigger, set a flag. When the flag is set wait for the end of file and then ...
  • 93.9k
1 vote
Accepted

How to generate a new file by extracting some parts?

Using sed on Monterey 12.6 $ sed -En '/.*🪫 SocEvaluator: 🟢 Final SoC is ([0-9.]+)%.*$/s//\1/pwfinal.csv' input_file 64.5537 $ cat final.csv 64.5537
  • 2,164
1 vote

How to generate a new file by extracting some parts?

This is using sed to match the wanted line(s) and then chop off everything up to the last space: sed -e '/SocEvaluator:.*Final SoC is [[:digit:]]/!d' -e 's/.* //' file For the given data, this would ...
  • 281k
1 vote

add a new line after a specific string in a file?

I'm going to assume that echo is just a typo because unless there's something specific with your environment, that same command just gives the output sed '/Random/a\ newly added line' info.txt If you ...
  • 9,841
0 votes

Binary search in a sorted text file

From a very large, sorted log file I often want to extract all records after a given date. Reading the whole file looking for the date linearly takes much too long. A dozen years ago I hastily ...
0 votes

How to print out one part of a regex match, and a list of all other matches

Using Raku (formerly known as Perl_6) ~$ raku -ne 'state %h; state $i; ++$i; \ if m/ ^ \h* at \h .*? \( (src .*? \.test \.js) \: (\d+ \: \d+) \) / -> { %h.push: $0 =&...
  • 1,494
1 vote

concat thousands of huge json files by jq

Sounds like you don't even need jq, just to replace the trailing } with , in all but the last file, and remove the leading { from all but the first file. In zsh: autoload zargs files=( *.json(Nn) ) ...
2 votes
Accepted

concat thousands of huge json files by jq

jq -c -s add file* This will read all files that match the pattern file* into an array in jq. The -s (--slurp) option causes the single array to be created from all input files. Each element of ...
  • 281k
1 vote

How to print out one part of a regex match, and a list of all other matches

That's more of a job for perl: perl -lne ' if (/^\h*at\h.*?\((src.*?\.test\.js):(\d+:\d+)\)/) { $total{$1}++; $lines{$1}->{$2}++ } END { for $file (sort {$total{$b} <=> $...
1 vote

Delete the Nth line from a matched string (which occurs many times – but delete only first occurrence)

Using Raku (formerly known as Perl_6) ~$ raku -e 'my @a = slurp.split("\n\n"); .subst(" IGNORE\n").put if .match(/AGENT_1 | AGENT_3/) for @a;' file Briefly, the code above ...
  • 1,494
0 votes

inserting data into a particular section of a configuration file with ansible

You can use ini_file module from general collection: - name: Adding Entries in "/etc/cinder/cinder.conf" community.general.ini_file: path: "/etc/cinder/cinder.conf" ...
0 votes

Search in specific column for pattern and output entire line

Using Raku (formerly known as Perl_6) ~$ raku -ne '.put if .split("|")[3].starts-with("5");' file Sample Input: 100|20151010|K|5001 695|20151010|K|1010 309|20151010|R|5005 410|...
  • 1,494
1 vote

How to prepend lines with serial number in descending order using awk

Using Raku (formerly known as Perl_6) $ raku -e 'my @a = lines(); put (@a.elems - $++, $_) for @a;' file Sample Input: Abcd Bcde Cdef Defg Efgh Fghi Ghij Sample Output: 7 Abcd 6 Bcde 5 Cdef 4 Defg 3 ...
  • 1,494
4 votes
Accepted

How to display process with status -R

Add the r option: ps auxr This limits the selection to running processes only. Compared to post-processing the output, this has the advantage of being handled entirely within ps, and works even if ...
4 votes

How to prepend lines with serial number in descending order using awk

Not using awk but instead employing tools that would let you avoid doing your own (more detailed) scripting. Using the non-standard tac utility to output the lines of a file in reverse order, and cat ...
  • 281k
2 votes

How to prepend lines with serial number in descending order using awk

Note: This answer assumes that the goal is to have "down-counting" line numbers while keeping the original order of the text lines. The phrasing of the post is ambiguous in thar respect, so ...
  • 19.4k
1 vote
Accepted

How to prepend lines with serial number in descending order using awk

Using awk $ awk '{ a[i++]=NR" "$0 } END { for (n=i-1;n>=0;) print a[n--] }' input_file
  • 2,164
1 vote

Delete the Nth line from a matched string (which occurs many times – but delete only first occurrence)

Task: Remove IGNORE lines for specified CPU names If you want to remove the IGNORE line for AGENT_1 and AGENT_3 you can use awk '$1 == "CPUNAME" && ($2 == "AGENT_1" || $2 ==...
  • 5,633
0 votes

Search in specific column for pattern and output entire line

If you want answers using tools that are CSV-aware to account for CSV files containing fields with embedded | characters and newlines, then here's how you do it with mlr (Miller): mlr --csv --fs '|' -...
  • 281k
4 votes

Split a record by record type and report any unexpected record type

Using any awk and any sort: $ cat tst.sh #!/usr/bin/env bash awk ' BEGIN { split("E20 I47 I49 I50 Q11 R11 W55",tmp) for ( i in tmp ) { expected[tmp[i]] ...
  • 24.6k
0 votes

Collapse multi-line records into single CSV lines containing the "value" parts only

If only the input data looked a bit more like this (rec-formatted): Name: Ami Admin: 2 Oper: 1 Name: Sum Admin: 3 Total: 2 Name: Tar Admin: 1 Oper: 2 ... then we could easily use rec2csv | csvcut -...
  • 281k
5 votes
Accepted

Split a record by record type and report any unexpected record type

Here's one awk way. First, create a file with the "good" records, one per line: $ cat goodRecs E20 I47 I49 I50 Q11 R11 W55 Then: gawk 'FNR==NR{good[$1]; next} { rec=substr($...
  • 221k
0 votes

Collapse multi-line records into single CSV lines containing the "value" parts only

[Not knowing whether the Name>... record can be missing at times instead of systematically appearing at regular interval in the data to be processed, I will assume it's always there.] A simple ...
  • 2,334
1 vote

Collapse multi-line records into single CSV lines containing the "value" parts only

It seems you want to collapse multi-line records into single CSV lines, only print the values of the record attributes Name, Admin and Oper and print "explicit" empty fields where one of ...
  • 19.4k
0 votes

no slashes needed in date string provided via zenity

According to the documentation for zenity (or man zenity) this is handled directly (the example even shows how to do this directly): --date-format=format Specifies the format that is returned from ...
  • 93.9k
0 votes

Collapse multi-line records into single CSV lines containing the "value" parts only

If I understood what you want then you should use xargs and awk commands: xargs -n3 < your_file.txt | awk '{gsub(/(Name|Admin|Oper)>/,""); print $1","$2","$3}' | ...
1 vote

Splitting a file every time the content of a given column changes

awk -F'|' '$2 != out{close(out); out=$2} {print > out}' If you don't close each output file as you go then once you get past a threshold of concurrently open files the awk script will either fail ...
  • 24.6k
3 votes
Accepted

Splitting a file every time the content of a given column changes

Using awk $ awk -F"|" '{print > $2}' input_file $ head gene_cluster* ==> gene_cluster:GC_00001105 <== >00000000|gene_cluster:GC_00001105|genome_name:r7534_20160316|...
  • 2,164
5 votes
Accepted

Regular expression not matching the expected data

The issue is the regular expression. You would need to replace each * by .* to match any number of any character between the substrings. awk -v RS='' -v ORS='\n\n' '/func: peak.*down.*HIL/' file ...
  • 281k
1 vote
Accepted

How can I obtain a full tabular list of the running process on my system (with their command name and PID at least) for text processing consumption?

I have been pretty successful with the ps -o output format field names. The output specifiers are mentioned a couple of times early in the Linux ps(1) man page: First a short ways into the EXAMPLES ...
  • 1,725
0 votes
Accepted

Merging two non-git-based text files with similar semantics to the depiction of git merge conflicts

Due to the limitations of the solutions I originally posted in my first "Answer" here which involve diff -D ... and diff -U ..., I decided to write a solution in python, using python's ...
  • 405
0 votes

Merging two non-git-based text files with similar semantics to the depiction of git merge conflicts

It sounds like you don't REALLY care about the output format and instead just want to know how to identify which lines are from each file or are common. Give that, how about: $ diff --old-line-format=$...
  • 24.6k
0 votes

How can I obtain a full tabular list of the running process on my system (with their command name and PID at least) for text processing consumption?

Eventually I used the output of ps -e.
  • 1,066

Top 50 recent answers are included