New answers tagged text-processing
0
votes
Printing every Nth line out of a large file into a new file
I'm working on Mac OS, and I was getting the following error (probably because it's not GNU sed):
sed: 1: "1~4p": invalid command code ~
Thankfully, I was able to use my vim knowledge to ...
- 111
0
votes
Tool to create text files from a template
Here's a dirty hack with "read" and "HEREDOC". ðŸ¤ðŸ«£
#!/bin/bash
MY_HEREDOC_CONTENT='// Variables to keep track of the number of iterations.
var kwinScriptLoops = 0;
var ...
- 694
0
votes
Is there a robust command line tool for processing csv files?
This is a fix for Alpaca's earlier answer which was on the right track, but didn't actually work.
$ printf 'c0,c1,c2\n"this, is the first entry", this is the second, 34.5' |
sqlite3 -cmd '....
- 547
0
votes
Is there a robust command line tool for processing csv files?
DuckDB can query CSV files directly, including from standard input on Unix like operating systems:
$ echo '"this, is the first entry", this is the second, 34.5' |
duckdb -c 'SELECT * FROM ...
- 547
0
votes
Is there a robust command line tool for processing csv files?
csvtk can read and process CSV files and csvtk cut in particular can select specific columns from an input CSV file:
$ echo '"this, is the first entry", this is the second, 34.5' |
csvtk ...
- 547
1
vote
Use 'fold' to wrap lines at 72 column
If you want your second result, instead of using fold, use fmt
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmo
tempor incididunt ut labore et dolore magna aliqua. Ut enim minim ...
- 7,737
5
votes
Accepted
Use 'fold' to wrap lines at 72 column
Because fold just folds. It doesn't know anything about words, really, it just has that nice option to break on spaces. However, if you look at the output closely, you will see that while it does ...
- 245k
1
vote
Large file manipulation
duckdb cli is a great tool to run this kind of task for large file.
If the input file is in example this TSV
Postcode Latitude Longitude Lorem Ipsum
AB1 0AA 57.101474 -2.242851 e 2
AB1 ...
- 2,915
1
vote
Large file manipulation
This might work for your Ubuntu box:
gawk -v OFS=, '{print $2,$3,q $1 q}NR==1{FS=","; q="\""}' pc0.csv
The idea is to reset FS="," and specify the quotes after the ...
- 121
4
votes
Large file manipulation
This, using any awk, will convert the first sample file you show to the second sample file you show:
$ awk '
BEGIN { FS=OFS="," }
{ sub(/ +$/, "") }
NR == 1 { gsub(/ /, ...
- 33k
0
votes
How to use Awk to format numbers with a thousands separator
In bash if num is a whole number (or in fact any string), you can insert a comma every third position from the right like this:
num=1234567
rem='' # the separator is here ↘
while [[ "${#num}&...
1
vote
How to use grep to get the matching part only, without introducing extra newlines
Using Raku (formerly known as Perl_6)
~$ raku -ne 'put join "", m:g/ ban <[a..z]>* /;' file
#OR
~$ raku -ne 'put join "", m:g/ ban <alpha>* /;' file
Raku is a ...
- 3,495
0
votes
How to remove comma from a csv file with quotes?
Using Raku (formerly known as Perl_6)
~$ raku -MText::CSV -e 'my @a = csv(in => $*IN, sep => ", ");
for ^@a[0].elems -> $i {
@a&...
- 3,495
0
votes
Parse colon-separated value pairs (nmcli output) and convert to JSON fromat
Since the stupid --terse flag will break the bssid parsing (i wonder how nobody thought about this?) I just use the normal form:
# Use either subprocess module or asyncio.create_subprocess_shell ...
- 119
0
votes
Find duplicated column value in CSV
Using Miller (mlr) to read the data as headerless CSV, adding a count to each record, signifying the number of times the first field occurs in the input, then filtering out (extracting) all records ...
- 341k
0
votes
Find duplicated column value in CSV
Using Raku (formerly known as Perl_6)
Raku is a Perl-family programming language, with a small (but growing) module ecosystem. This problem might be tackled with a CSV parser such as Raku's Text::CSV ...
- 3,495
0
votes
Move line matching string to top of the file
$ sed -n '/PRM:TRG/!H;//p;${g;s/\n//;p;}' << EOF
FOOBAR:SCN:NO_TRG
FIZZBAR:SCN:NO_TRG
BEBAR:PRM:TRG
EOF
BEBAR:PRM:TRG
FOOBAR:SCN:NO_TRG
FIZZBAR:SCN:NO_TRG
For non-matches (!), append to Hold-...
- 4,835
0
votes
Print lines between two patterns where first pattern appears more than once before second pattern
For fun here's an alternative solution. Source data is in file:
# Temporary directory
t=$(mktemp --directory 'vhnames.XXXXXXXXXX')
# Split the source data file into parts, each starting with a line ...
- 120k
0
votes
Print lines between two patterns where first pattern appears more than once before second pattern
Using any awk:
$ awk '/^</{s=""} /Server/{s=s $0 ORS} /Redirect/{printf "%s", s}' file
ServerName domain1.com:
ServerAlias www.domain1.com:
ServerName domain3.com:
...
- 33k
3
votes
Print lines between two patterns where first pattern appears more than once before second pattern
Yet another awk solution. This one sets three variables:
block - pattern match that starts a block
need - pattern match we need to trigger any output
result - pattern match for lines to be output
...
- 120k
1
vote
Print lines between two patterns where first pattern appears more than once before second pattern
If we reduce this to printing each block if it contains a Redirect, the straightforward solution is to collect lines to a variable and check what we got at the end (end of file or start of next block)....
- 142k
0
votes
Print lines between (and excluding) two patterns
Using Raku (formerly known as Perl_6)
~$ raku -ne '.put if /firstmatch/^fff^/secondmatch/;' file
Raku is a programming language in the Perl-family. Like Perl, Raku provides many operators to help ...
- 3,495
3
votes
Move line matching string to top of the file
With perl:
perl -ne 'if (/PRM:TRG/) {print} else {push @saved, $_}
if (eof) {print for @saved}' your-file
Add a -i option to edit the file in place, or -i.back to save the original with a ....
- 558k
1
vote
Print lines between two patterns where first pattern appears more than once before second pattern
If you are looking to retrieve all domain names/aliases under a VirtualHost which has a Redirect directive, you can try reading each Virtual Host as a record (RS="<VirtualHost ") and set ...
- 121
1
vote
Print lines between two patterns where first pattern appears more than once before second pattern
Using Raku (formerly known as Perl_6)
First, generate code to break into full desired records:
~$ raku -e 'my @a = slurp.split(/^^ <?before \<VirtualHost > /, :skip-empty );
.put ...
- 3,495
0
votes
How to transpose a text file on character basis
Using Raku (formerly known as Perl_6)
~$ raku -e 'my @a = lines.map: *.comb; my @b;
my $max1 = @a.map(*.elems).max;
for ^@a -> $i {
for ^$max1 -> $j {...
- 3,495
1
vote
Move line matching string to top of the file
Using Raku (formerly known as Perl_6)
~$ raku -ne 'BEGIN my @non-match;
/PRM\:TRG/ ?? .put !! @non-match.push($_);
END .put for @non-match;' file
#OR:
~$ raku -e 'my ...
- 3,495
2
votes
Move line matching string to top of the file
If you have sed then you also have awk so - using any awk and a 2-pass approach just reading 1 line at a time into memory:
$ awk 'NR==FNR { if (/PRM:TRG/) { print; skip=FNR } next } FNR!=skip' file ...
- 33k
8
votes
Accepted
Move line matching string to top of the file
With the ed line editor, the command
/PRM:TRG/ m 0
... would move the next line matching the regular expression PRM:TRG to the first line (0 meaning "after line zero"). "The next line ...
- 341k
0
votes
Cut and replace Nth character on every row
Using Raku (formerly known as Perl_6)
With sed-like code (using Raku's -pe autoprinting flags):
~$ raku -pe 's/ \,\h /x/;' file
#OR
~$ raku -pe 's:10th/ .. /x/;' file
#OR
~$ raku -pe ' $_.=subst(...
- 3,495
1
vote
sed: To match a newline and spaces
Using Raku (formerly known as Perl_6)
~$ raku -ne 'put $_ unless / "<title>" .* "</title>" /;' file
#OR:
~$ raku -ne ' .put unless / "<title>" .* &...
- 3,495
0
votes
Remove last character from line
Using Raku (formerly known as Perl_6)
~$ printf '123\n456\n789' | raku -pe '.=chop;'
12
45
78
#OR:
~$ printf '123\n456\n789' | raku -ne '.chop.put;'
12
45
78
OR:
~$ printf '123\n456\n789' | raku -...
- 3,495
-2
votes
Remove last character from line
df -h | awk {'print $5'}
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4,0M 0 4,0M 0% /dev
tmpfs 16G 1,6G 15G 10% /dev/shm
tmpfs 6,3G 2,3M 6,3G 1%...
1
vote
Fastest way to get all the string value of 4th column in a command output
I would follow up @jubilation1's answer with another recommendation for Raku. If you want to separate on any number of whitespaces you can use the .words method for strings:
cat file > raku -e 'say ...
- 111
0
votes
Reverse the effect of endianness on a memory dump
here is the non-endiannes related binary data reversing for the MAC address recovery and presenting human readable way:
recovery of the MAC address binary data from the OTP and converting it to the ...
- 511
Top 50 recent answers are included