New answers tagged text-processing
5
votes
How to check a value and change other key in JSON with jq?
To apply an expression to all elements of an array, and get an altered array out, use map(expression).
Extracting all entries that have off status:
$ jq 'map(select(.status == "off"))' file
[...
- 281k
1
vote
How to cross check one column in a file in another and print the missing values?
I would first sort the two files (excluding the header of course). I would then use awk or cut to select the first column (minus the header) and use comm to select the ones from file1 that are not in ...
- 2,771
1
vote
How to cross check one column in a file in another and print the missing values?
If you are using a system that supports process substitution, you can use grep with the -v flag (show non-matching lines) and -f flag (read patterns from file), where the "file" is a command ...
- 221k
0
votes
combine two text files with adding some separator between?
Using find's arguments -printf and -exec, combined with cat you can place text before and after a files contents.
find . -type f -name "*.txt" -printf "/n This is before! \n" -...
- 101
0
votes
Missing column after merging two files
Assuming the data in the two input files are "simple" CSV records without fields containing embedded delimiters or newlines, that the | characters are the delimiters, and that the files ...
- 281k
3
votes
Accepted
Find highest value amongst columns and print value in a new column
It should be fairly easy: You can simply iterate over all fields and check which is the highest in this line. Then, append that highest value as (NF+1)th field.
awk 'BEGIN{FS=OFS="\t"} {max=$...
- 19.4k
1
vote
Missing column after merging two files
You could rely on join:
join -t\| -j 1 -o 1.2,2.2,2.3,1.3 file2 file1
Where the format(-o) is defined as FILE.FIELD, so just a selection of which field to take from which input file, -t is used to ...
- 11.3k
3
votes
Missing column after merging two files
The answer is rather simple: You use a[$3] to refer to the third column of file1. However
you use the array a to store the second column of file1, and never the third column, and
you only ever use ...
- 19.4k
1
vote
Missing column after merging two files
awk 'BEGIN{FS=OFS="|"} NR==FNR {a[$1]=$2 OFS $3;next} ($1 in a) {print $2,a[$1],$3}' file1.txt file2.txt > file3.txt
You never saved a[$3] before in your script so it doesn't find it. You ...
- 86
1
vote
Accepted
How to perform a "grouped merge" of three files, while omitting the first few lines of each?
Applying the DSU Idiom, using any versions of the mandatory POSIX tools awk, sort, and cut:
$ cat tst.sh
#!/usr/bin/env bash
awk -v OFS='\t' '
FNR == 1 { fileNr++ }
FNR >= 4 { print FNR-3, ...
- 24.6k
2
votes
Accepted
Replace contents of one column in txt file by custom strings
If the other file has one string per line, like this:
$ cat file2
John1
Mike2
Tom33
WolfN
and you want to add them in the order they are found, you can do this:
$ awk 'NR == FNR{a[NR]=$1;next}; {$(NF-...
- 221k
0
votes
How to perform a "grouped merge" of three files, while omitting the first few lines of each?
If you don't mind using something else other than awk:
for f in $(ls *.txt) ; do awk 'FNR >=4' $f | egrep "." -n ; done | sort -n | uniq | cut -d: -f2-
would do the trick
explanation:
...
- 53
0
votes
How to perform a "grouped merge" of three files, while omitting the first few lines of each?
i did save the patterns you gave above in three files. I got the desired output completting your awk-filtering on this way:
for i in {4..15}; do awk "FNR == $i" *.txt | sort -u; done
- 141
1
vote
How to generate a new file by extracting some parts?
Using Raku (formerly known as Perl_6)
~$ raku -ne 'put $<> if m/ "Final SoC is " <(\d* \. \d*)> /;' file
#OR
~$ raku -ne 'put $0 if m/ "Final SoC is " (\d* \. \d*) /...
- 1,494
0
votes
Print ONLY the next line after a match with awk
Using Raku (formerly known as Perl_6)
~$ raku -e 'put $<> if m{ "Artifacts were created:" \n <( \N* } for lines.join: "\n";' file
#OR
~$ raku -e 'put $0 if m{ "...
- 1,494
2
votes
Print ONLY the next line after a match with awk
If sed is an option
$ sed -En '/Artifacts were created:/{n;s/[^ ]* (.*)/\1/p}' <<< "$file"
build-1.tar.gz
- 2,164
2
votes
Accepted
Print ONLY the next line after a match with awk
Please try:
awk '/Artifacts were created:/{p=1} NF{out=$2} END{if(p==1){print out}}' infile
Although the pipe you presented is extracting the second filed of the line just following the match, not ...
- 1,858
2
votes
Print ONLY the next line after a match with awk
The original requirement asked for a value from the last non-whitespace line after a match. Here, when you find a match trigger, set a flag. When the flag is set wait for the end of file and then ...
- 93.9k
1
vote
Accepted
How to generate a new file by extracting some parts?
Using sed on Monterey 12.6
$ sed -En '/.*🪫 SocEvaluator: 🟢 Final SoC is ([0-9.]+)%.*$/s//\1/pwfinal.csv' input_file
64.5537
$ cat final.csv
64.5537
- 2,164
1
vote
How to generate a new file by extracting some parts?
This is using sed to match the wanted line(s) and then chop off everything up to the last space:
sed -e '/SocEvaluator:.*Final SoC is [[:digit:]]/!d' -e 's/.* //' file
For the given data, this would ...
- 281k
1
vote
add a new line after a specific string in a file?
I'm going to assume that echo is just a typo because unless there's something specific with your environment, that same command just gives the output
sed '/Random/a\
newly added line' info.txt
If you ...
- 9,841
0
votes
Binary search in a sorted text file
From a very large, sorted log file I often want to extract all records after a given date. Reading the whole file looking for the date linearly takes much too long.
A dozen years ago I hastily ...
- 827
0
votes
How to print out one part of a regex match, and a list of all other matches
Using Raku (formerly known as Perl_6)
~$ raku -ne 'state %h; state $i; ++$i; \
if m/ ^ \h* at \h .*? \( (src .*? \.test \.js) \: (\d+ \: \d+) \) /
-> { %h.push: $0 =&...
- 1,494
1
vote
concat thousands of huge json files by jq
Sounds like you don't even need jq, just to replace the trailing } with , in all but the last file, and remove the leading { from all but the first file.
In zsh:
autoload zargs
files=( *.json(Nn) ) ...
- 482k
2
votes
Accepted
concat thousands of huge json files by jq
jq -c -s add file*
This will read all files that match the pattern file* into an array in jq. The -s (--slurp) option causes the single array to be created from all input files. Each element of ...
- 281k
1
vote
How to print out one part of a regex match, and a list of all other matches
That's more of a job for perl:
perl -lne '
if (/^\h*at\h.*?\((src.*?\.test\.js):(\d+:\d+)\)/) {
$total{$1}++;
$lines{$1}->{$2}++
}
END {
for $file (sort {$total{$b} <=> $...
- 482k
1
vote
Delete the Nth line from a matched string (which occurs many times – but delete only first occurrence)
Using Raku (formerly known as Perl_6)
~$ raku -e 'my @a = slurp.split("\n\n"); .subst(" IGNORE\n").put if .match(/AGENT_1 | AGENT_3/) for @a;' file
Briefly, the code above ...
- 1,494
0
votes
inserting data into a particular section of a configuration file with ansible
You can use ini_file module from general collection:
- name: Adding Entries in "/etc/cinder/cinder.conf"
community.general.ini_file:
path: "/etc/cinder/cinder.conf"
...
- 101
0
votes
Search in specific column for pattern and output entire line
Using Raku (formerly known as Perl_6)
~$ raku -ne '.put if .split("|")[3].starts-with("5");' file
Sample Input:
100|20151010|K|5001
695|20151010|K|1010
309|20151010|R|5005
410|...
- 1,494
1
vote
How to prepend lines with serial number in descending order using awk
Using Raku (formerly known as Perl_6)
$ raku -e 'my @a = lines(); put (@a.elems - $++, $_) for @a;' file
Sample Input:
Abcd
Bcde
Cdef
Defg
Efgh
Fghi
Ghij
Sample Output:
7 Abcd
6 Bcde
5 Cdef
4 Defg
3 ...
- 1,494
4
votes
Accepted
How to display process with status -R
Add the r option:
ps auxr
This limits the selection to running processes only.
Compared to post-processing the output, this has the advantage of being handled entirely within ps, and works even if ...
- 371k
4
votes
How to prepend lines with serial number in descending order using awk
Not using awk but instead employing tools that would let you avoid doing your own (more detailed) scripting.
Using the non-standard tac utility to output the lines of a file in reverse order, and cat ...
- 281k
2
votes
How to prepend lines with serial number in descending order using awk
Note: This answer assumes that the goal is to have "down-counting" line numbers while keeping the original order of the text lines. The phrasing of the post is ambiguous in thar respect, so ...
- 19.4k
1
vote
Accepted
How to prepend lines with serial number in descending order using awk
Using awk
$ awk '{ a[i++]=NR" "$0 } END { for (n=i-1;n>=0;) print a[n--] }' input_file
- 2,164
1
vote
Delete the Nth line from a matched string (which occurs many times – but delete only first occurrence)
Task: Remove IGNORE lines for specified CPU names
If you want to remove the IGNORE line for AGENT_1 and AGENT_3 you can use
awk '$1 == "CPUNAME" && ($2 == "AGENT_1" || $2 ==...
- 5,633
0
votes
Search in specific column for pattern and output entire line
If you want answers using tools that are CSV-aware to account for CSV files containing fields with embedded | characters and newlines, then here's how you do it with mlr (Miller):
mlr --csv --fs '|' -...
- 281k
4
votes
Split a record by record type and report any unexpected record type
Using any awk and any sort:
$ cat tst.sh
#!/usr/bin/env bash
awk '
BEGIN {
split("E20 I47 I49 I50 Q11 R11 W55",tmp)
for ( i in tmp ) {
expected[tmp[i]]
...
- 24.6k
0
votes
Collapse multi-line records into single CSV lines containing the "value" parts only
If only the input data looked a bit more like this (rec-formatted):
Name: Ami
Admin: 2
Oper: 1
Name: Sum
Admin: 3
Total: 2
Name: Tar
Admin: 1
Oper: 2
... then we could easily use rec2csv | csvcut -...
- 281k
5
votes
Accepted
Split a record by record type and report any unexpected record type
Here's one awk way. First, create a file with the "good" records, one per line:
$ cat goodRecs
E20
I47
I49
I50
Q11
R11
W55
Then:
gawk 'FNR==NR{good[$1]; next}
{
rec=substr($...
- 221k
0
votes
Collapse multi-line records into single CSV lines containing the "value" parts only
[Not knowing whether the Name>... record can be missing at times instead of systematically appearing at regular interval in the data to be processed, I will assume it's always there.]
A simple ...
- 2,334
1
vote
Collapse multi-line records into single CSV lines containing the "value" parts only
It seems you want to
collapse multi-line records into single CSV lines,
only print the values of the record attributes Name, Admin and Oper
and print "explicit" empty fields where one of ...
- 19.4k
0
votes
no slashes needed in date string provided via zenity
According to the documentation for zenity (or man zenity) this is handled directly (the example even shows how to do this directly):
--date-format=format Specifies the format that is returned from ...
- 93.9k
0
votes
Collapse multi-line records into single CSV lines containing the "value" parts only
If I understood what you want then you should use xargs and awk commands:
xargs -n3 < your_file.txt | awk '{gsub(/(Name|Admin|Oper)>/,""); print $1","$2","$3}' | ...
- 937
1
vote
Splitting a file every time the content of a given column changes
awk -F'|' '$2 != out{close(out); out=$2} {print > out}'
If you don't close each output file as you go then once you get past a threshold of concurrently open files the awk script will either fail ...
- 24.6k
3
votes
Accepted
Splitting a file every time the content of a given column changes
Using awk
$ awk -F"|" '{print > $2}' input_file
$ head gene_cluster*
==> gene_cluster:GC_00001105 <==
>00000000|gene_cluster:GC_00001105|genome_name:r7534_20160316|...
- 2,164
5
votes
Accepted
Regular expression not matching the expected data
The issue is the regular expression. You would need to replace each * by .* to match any number of any character between the substrings.
awk -v RS='' -v ORS='\n\n' '/func: peak.*down.*HIL/' file
...
- 281k
1
vote
Accepted
How can I obtain a full tabular list of the running process on my system (with their command name and PID at least) for text processing consumption?
I have been pretty successful with the ps -o output format field names. The output specifiers are mentioned a couple of times early in the Linux ps(1) man page:
First a short ways into the EXAMPLES ...
- 1,725
0
votes
Accepted
Merging two non-git-based text files with similar semantics to the depiction of git merge conflicts
Due to the limitations of the solutions I originally posted in my first "Answer" here which involve diff -D ... and diff -U ..., I decided to write a solution in python, using python's ...
- 405
0
votes
Merging two non-git-based text files with similar semantics to the depiction of git merge conflicts
It sounds like you don't REALLY care about the output format and instead just want to know how to identify which lines are from each file or are common. Give that, how about:
$ diff --old-line-format=$...
- 24.6k
0
votes
How can I obtain a full tabular list of the running process on my system (with their command name and PID at least) for text processing consumption?
Eventually I used the output of ps -e.
- 1,066
Top 50 recent answers are included