New answers tagged awk
1
vote
awk appends column in .gz file as seen with cat -A, but changes column name in regular output
Your input seems to be some sort of TSV file with Microsoft line endings.
Then you could use mlr instead of awk that supports 2 sorts of tsv and can specify the record delimiter.
with --tsv values ...
- 533k
2
votes
Replace single quote at beginning of columns
Using GNU awk for gensub() (also supported in some other awks but not required by POSIX yet):
$ awk '{$0=gensub(/(^|\|)\047/,"\\1","g")} 1' file
foo|012|that's nice|bar
I could ...
- 30.1k
2
votes
Replace single quote at beginning of columns
You would have to iterate over the fields and replace the quote character in each. Stéphane shows how to do this using awk in their answer.
$ mlr --csv --fs pipe -N put 'for (k,v in $*) { $[k] = sub(...
- 327k
5
votes
Replace single quote at beginning of columns
No need for awk, sed can do it with:
sed -E "s/(^|\|)'/\1/g"
The -E option to switch to extended regular expressions will be in the next version of the POSIX standard but is already ...
- 533k
1
vote
Copy matching lines to the next line and modify?
Using any awk:
$ awk '1; sub(/bar/,"&2")' file
a foo 1
b bar 2
b bar2 2
c foo 3
d baz 4
e bar 5
e bar2 5
$ awk '1; sub(/bar/,"hello world")' file
a foo 1
b bar 2
b hello world ...
- 30.1k
4
votes
Accepted
awk appends column in .gz file as seen with cat -A, but changes column name in regular output
As already mentioned you have DOS line endings.
See why-does-my-tool-output-overwrite-itself-and-how-do-i-fix-it for a description of the issue and possible solutions, for example using any awk:
...
- 30.1k
1
vote
awk appends column in .gz file as seen with cat -A, but changes column name in regular output
As @steeldriver noted in comments, the file appeared to be Windows contaminated with carriage return (CR) and line feed (LF) characters, we can remove setting record separators RS="\r\n" but ...
- 227
1
vote
Copy matching lines to the next line and modify?
Using awk:
$ awk '/bar/{print; sub(/bar/,"&2")}1'
As the sed answer suggests, the awk command prints the record with pattern bar and then substitutes it with bar2.
- 1,740
3
votes
Accepted
Copy matching lines to the next line and modify?
As Stéphane suggests, just print the line and then substitute, sed will autoprint the result:
sed '/bar/{p;s//&2/;}' infile
2
votes
Returning lines from a JSON log file which contain IP and timestamps read from a CSV file?
Assuming the two input files, file.csv and file.jsonl, are correctly formatted as CSV and JSONL files, respectively, we can use Miller (mlr) to extract the client IP from the value of the log key in ...
- 327k
0
votes
identify content inside base64 encoded content and replace the whole line
So far, the only working solution seems to be to just accept that the encoded command is UTF16LE and manually include the null bytes \u0000 on the search in the jq script that generates the content...
...
- 111
2
votes
awk print between lines when "/" is part of the name
/regex/ is short for $0 ~ /regex/, which you can also write $0 ~ "regex", so besides escaping the / by prefixing it with \ or using \octal with the / character value (\57 on ASCII-based ...
- 533k
2
votes
awk print between lines when "/" is part of the name
How can I print lines between those patterns?
By replacing / with \57 records between patterns (and records with those patterns) can be printed.
But please read this before using range expressions.
$ ...
- 1,740
1
vote
Accepted
awk print between lines when "/" is part of the name
When used inside an awk regular expression constant, the forward slash character must be escaped to distinguish it from the surrounding delimiters. For example with GNU awk:
$ echo 'foo/bar was here' |...
- 79.6k
3
votes
Accepted
shell one-liners for bulk-renaming multiple files
If you can install zsh:
XX - title.flac to XX title.flac regular files.
zsh <<\EOF
autoload zmv
zmv -n '(**/)(<-> )- (*.(#i)(ogg|mp3|flac))(#q.)' '$1$2$3' < /dev/tty
EOF
GENRE - ...
- 533k
1
vote
AWK match exact value from comma separated column in brackets
Using pcregrep:
$ pcregrep -o1 '(\S*).*(\[(\d+,)*22(,\d+)*])' file
Using awk:
The following command may be used:
$ nawk '$(NF-2) ~ /\[([[:alnum:]]*,)*22(,[[:alnum:]]*)*]/{print $1}' file
If ...
- 1,740
2
votes
AWK match exact value from comma separated column in brackets
Using any awk:
$ awk '$(NF-2) ~ /[[,]22[],]/{print $1}' file
33.xx
24.1b
1.52
With some more comprehensive sample input that includes more than just the sunny day cases where we find a match where we ...
- 30.1k
1
vote
AWK match exact value from comma separated column in brackets
Something like can do the work:
awk -F"[][]" '$2~"22"{print }' <input file>
here the delimiter is set to be [ or ] and check if second field contain 22
If you want the value ...
- 17.1k
1
vote
identify content inside base64 encoded content and replace the whole line
Assuming you want to:
find base64 encoded data after any occurrence of -encodedcommand (case insensitive) followed by whitespace
decode it into something assumed to be a multiline UTF16-LE encoded ...
- 533k
1
vote
Find all lines between two patterns, EXCLUSIVE of the second pattern?
TXR Lisp's awk macro supports this directly; the rng (range) operator has nine variants for various ways of excluding records from the start or end of a range:
$ txr -e '(awk ((rng- #/Word A/ #/Word ...
- 7,894
1
vote
Accepted
Find all lines between two patterns, EXCLUSIVE of the second pattern?
GNU sed only:
sed '/Word A/!d;:1;n;/Word D/d;b1' file
In more complex cases - invalid blocks:
sed -n '/Word A/!b;:1;/Word A/h;n;/Word D/{g;p;d};H;b1' file
- 2,430
4
votes
Find all lines between two patterns, EXCLUSIVE of the second pattern?
Using AWK:
awk '/Word A/ { m = 1 } /Word D/ { m = 0 } m'
- 423k
0
votes
Find all lines between two patterns, EXCLUSIVE of the second pattern?
Using awk:
$ awk '
$0 == "Word A" { f=1; rec=$0; next }
{ if ( $0 == "Word D" ) { print rec; f=0 } }
f{rec = rec ORS $0}'
# For regex pattern
$ awk '
(/...
- 1,740
-2
votes
Find all lines between two patterns, EXCLUSIVE of the second pattern?
sed lets one do arithmetic on line specifications:
sed -n -e '/Word A/,/Word D/-1p' The_File
Read man sed.
- 4,709
2
votes
Find all lines between two patterns, EXCLUSIVE of the second pattern?
Here's an awk solution
awk \
-vstart='Word A' \
-vend='Word D' \
'{
if ($0==end ) {flag=0;next};
if ($0==start) {flag=1};
if (flag==1) {print $0};
}'
Only a minor change ...
- 4,776
1
vote
merge some line start with special character
Using the ed command below, we join any line starting with a # character with the previous line:
g/^#/ -,. j
What that command does is to apply the command -,. j to each line matching the regular ...
- 327k
1
vote
merge some line start with special character
Using any awk:
$ awk '
/^#/ { rec = rec OFS $0; next }
NR>1 { print rec }
{ rec = $0 }
END { print rec }
' file
2023-11-15T08:59:28.000000+00:00 database-1 # Time: 231115 8:59:28 # ...
- 30.1k
0
votes
merge some line start with special character
aws logs tail /aws/rds/instance/database-1/slowquery --log-stream-names database-1 --follow | sed ':a;N;$!ba;s/\n#/#/g'
- 131
3
votes
Accepted
merge some line start with special character
Using awk:
$ awk '
{
printf "%s", ((/^#/ || NR==1) ? "" : ORS )$0
}
END {
if (NR) print ""
}'
- 1,740
0
votes
Add two new columns in tab delimited file
Method 1: Use a Perl one-liner:
perl -lpe '$_ = join "\t", $_, qw(0 -);' in_file > out_file
Or change the file in-place:
perl -i.bak -lpe '$_ = join "\t", $_, qw(0 -);' in_file
...
- 458
1
vote
Accepted
Grep a log file for SQL queries and their execution time
Assuming your lines will always have the same number of fields, and that sql: will only appear on the lines you want, you can do this to sort on the 10th field which seems to be the time it took to ...
- 238k
2
votes
Grep a log file for SQL queries and their execution time
Extract the relevant fields, printing only lines that match, and sort the result by descending time of execution:
sed -En 's/^.*(time: [0-9.]+) .* (sql: .*)/\1 | \2/p' logfile | sort -k2,3nr
Output
...
- 111k
0
votes
bash + how to verify folders under specific path are ended with number/s
Another variant:
[ -z "$(LC_ALL=C find /var/kafka -mindepth 1 -maxdepth 1 -type d \( -name '*[0-9]' -o -print -quit \))" ]
This returns true ($? is 0) if all directory names end in a digit
- 111k
1
vote
bash + how to verify folders under specific path are ended with number/s
Using find:
LC_ALL=C find /var/kafka -mindepth 1 -maxdepth 1 -type d -regex '.*[^0-9]$' | grep '^' \
&& echo "Error in folder name"
I'm "abusing" grep to get a sensible ...
- 6,439
4
votes
Accepted
bash + how to verify folders under specific path are ended with number/s
With zsh:
files=( /var/kafka/*[^0-9](ND) )
if (( $#files )); then
print -rlu2 -- "There are files whose name doesn't end in an ASCII digit:" ' - '$^files
exit 1
fi
Same with bash:
shopt ...
- 533k
5
votes
Accepted
Can awk be told to count the character string length rather than byte string length for '%10s' printf formats?
GNU awk (and possibly some other awk variants):
$ echo 'Ü X' | LC_ALL='en_US.UTF-8' awk '{printf "|% 2s|% 2s|\n", $1, $2}'
| Ü| X|
Bash 3.0+ (and possibly some other shells, possibly with ...
- 30.1k
5
votes
Accepted
Why doesn't gawk -d create awkvars.out in my current working directory?
The manual is admittedly misleading, or at least unclear. The issue is that you still need to give a program to gawk so it has something to run. At least in the case of gawk, you can pass it an empty ...
- 238k
0
votes
Create Symlink Glob of directory read from a file
With zsh:
typeset -U prefixes=( ${(f)"$(cut -f5 < ExtensionList.tsv)"} )
typeset -U regular_files_with_those_prefixes=( ~/$^prefixes*(N.) )
if (( $#regular_files_with_those_prefixes )); ...
- 533k
1
vote
Retrieve the 1st and 5th column of a tab-separated file, convert the spaces in the 5th to tabs
FWIW in reality I'd use @PrapjhotSingh's solution but for some alternatives that may provide useful information on manipulating fields in general, read on...
To get the output you show from the input ...
- 30.1k
1
vote
Retrieve the 1st and 5th column of a tab-separated file, convert the spaces in the 5th to tabs
Using awk:
$ awk -F '\t' '{gsub(OFS,FS,$5); print $1 FS $5}' file
- 1,740
0
votes
Retrieve the 1st and 5th column of a tab-separated file, convert the spaces in the 5th to tabs
Using Raku (formerly known as Perl_6)
~$ raku -ne 'put join "\t", .words.[0,4..*];' file
Above is an answer coded in Raku, a member of the Perl-family of programming languages. Here I ...
- 2,583
2
votes
Accepted
Retrieve the 1st and 5th column of a tab-separated file, convert the spaces in the 5th to tabs
If what you want is the 1st and 5th tab-delimited field from some input file input.tsv, and then change the embedded spaces in those fields into tabs, you can do that with cut and tr like so:
cut -f 1,...
- 327k
0
votes
Parse text file add semicolons before and after numbers; join all other words that are before/ between numbers with hyphens
Using Raku (formerly known as Perl_6)
First see if you can get all lines to parse:
~$ raku -ne '.match(/ ^ \d* \s? <.alpha>+ \s \d+ \s [ \d**2 \/ \d**2 \/ \d**4 ] \s [[<.graph>+]+ % " ...
- 2,583
Top 50 recent answers are included