6
Here’s a generic solution using AWK, adapting to the number of common components:
BEGIN {
FS = "-"
}
NF == 2 {
split($1, start, /\./)
split($2, end, /\./)
printf $1
for (i = 1; i <= length(start); i++) {
if (start[i] != end[i]) {
for (j = i; j <= length(end); j++)
printf "%s%s"...
6
POSIXly, you could do:
awk -F '] ' '{
out = $1
for (i = 2; i <= NF; i++)
out = out FS toupper(substr($i, 1, 1)) substr($i, 2)
print out}' < infile
Or:
awk '
BEGIN {FS = OFS = "] "}
{
for (i = 2; i <= NF; i++)
$i = toupper(substr($i, 1, 1)) substr($i, 2)
print
}' < infile
That defines "] " as the ...
5
You need to either use extended regex with sed -E if available or escape the braces and parenthesis:
:a s/^\(.\{4\}x\{0,\}\)[^x ]/\1x/;ta
5
With GNU Sed,
md5sum input.txt | sed 's/./x/5g'
This simply skips substituting the 4 first characters of the string and performs the substitution for all other characters.
A POSIX alternative with Awk (although there is probably something simpler),
md5sum xad | awk '{
four=substr($0, 1, 4)
rest=substr($0, 5)
gsub(/./, "x", rest)
print four,...
5
If (md5sum) hashes; then this would do the job (in a shell that supports brace-expansion)
sed -e's/./x/'{5..32} infile
else for any character length until first character space seen; then:
Using sed, and as a general solution:
sed -E ':a s/^(.{4}x{0,})[^x ]/\1x/;ta' infile
:a is a sed label we named a
s/ substitute below matches
^ is start of line anchor
(...
5
Awk scripts consist of pattern {action} pairs.
if pattern is empty, then {action} will be applied to all records
if {action} is empty, then the default action {print} will be applied to all records matching pattern
awk '$1' will apply the default action {print} when pattern $1 evaluates true. Since a non-empty string is true, it will print the whole of any ...
5
You're not printing. Try
awk '{print $1}' input.txt > output.txt
When you just give an expression (the way you tried), awk works somewhat like default grep: completely print any matching lines:
awk '/regexp/' file.txt - print lines matching regexp
awk 'NR==3' file.txt - print line 3
awk '1' file.txt - print all lines where 1 is true, i.e. all (okay, an ...
5
$ cat script.awk
NR == FNR {
a[$0]
next
}
{
for (i = 2; i <= NF; i++) {
for (k in a) {
if ($i == k) {
a[k] = a[k] ? a[k] "," $1 : $1
}
}
}
}
END {
for (k in a) {
print a[k] "\t" k
}
}
Run it as:
$ cat search.txt
2.2.2.3
3.3.3.3
4.4.4.4
5.5.5.5
$ awk -F, -f script.awk search.txt data.csv
A,...
4
An option with awk:
awk -F'[.-]' '{ print $1"."$2"."$3"."$4"-"$8 }'
Output:
10.1.1.1-3
10.100.100.11-31
4
Here's a sed approach. It will look for everything up to the last - on the line, save that as \1, then save the last occurrence of a . followed by one or more digits as \2. The entire line is then replaced by \1\2:
$ sed -E 's/(.*-).*\.([0-9]+)/\1\2/' file
10.1.1.1-3
10.100.100.11-31
Alternatively, in awk, you could set the input field separator to . or -, ...
4
If on a GNU system, you can use sed:
sed 's/] ./\U&/g' <infile
stream editor 'substitute/replace-what/replace-with/globally' <inputfile
Here "replace-what" is a literal ] followed by a single space then a single character (. matches a single character except \newline but would match a \newline character if it was ever found in the ...
4
First note that the 2.2.2.3 regexp will match 2.2.2.3 but also 212.243 (as . is the regexp operator that matches any character) and also inside 22.2.2.36.
Here, I'd use perl:
<data.csv perl -F, -lane '
BEGIN {for (@l = qw{2.2.2.3 3.3.3.3 4.4.4.4 5.5.5.5}) {$v{$_} = []}}
for (grep $v{$_}, @F[1..$#F]) {push @{$v{$_}}, $F[0]}
END {for (@l) {print(join(&...
4
md5sum always produces a 32-character output for the hash. Instead of detecting a space, you could look for 32-characters then a space and replace the last 28 characters with an X.
md5sum input.txt | sed -E 's/^([[:xdigit:]]{4})[[:xdigit:]]{28} /\1xxxxxxxxxxxxxxxxxxxxxxxxxxxx /'
35c9xxxxxxxxxxxxxxxxxxxxxxxxxxxx input.txt
Breaking down the statement:
's/^([...
3
With perl if GNU sed isn't available:
md5sum input.txt | perl -pe 's/^.{4}(*SKIP)(*F)|./x/g'
^.{4}(*SKIP)(*F) will prevent replacement of first four characters
|. specifies the alternate pattern that has to be replaced
To change only the checksum:
md5sum ip.txt | perl -pe 's/(^.{4}|\h.*$)(*SKIP)(*F)|./x/g'
3
POSIXly (I think), you could use a sed loop to repeatedly replace the first non-x character following the 4-character prefix:
$ md5sum input.txt | sed '
:a
s/^\(....x*\)[^x]/\1x/
ta
'
Replace [^x] with [^x ] if you only want to do the substitution in the first field (the checksum).
3
A Perl approach:
$ md5sum file | perl -ple 's/^(.{4})(\S+)(\s+.*)/$1 . "x" x length($2) . $3/e'
9191xxxxxxxxxxxxxxxxxxxxxxxxxxxx file
That will find the first 4 characters (^(.{4})), then as many non-whitespace characters as possible ((\S+)) and then the first one or more spaces followed by anything until the end of the line ((\s+.*)). That is ...
3
grep -o '\w\+' file.txt | sort | uniq -c
Explanation:
grep -o will output each match on a separate line.
\w\+ matches any run of consecutive alphanumeric characters and _.
uniq -c will output a count of occurrences for each consecutive run
of repeated lines.
The sort before uniq -c is necessary in order
to group each unique word in a single consecutive run ...
3
Capture the first digit, replace it with | and the captured digit:
sed -e 's/\([0-9]\)/| \1/'
3
If you want to extract the lines that has the smallest value in the 11th column grouped by gene, then use the gene identifier as the key in an associative array in awk to keep track of the smallest value for that gene:
$ awk '{ gene = $1 } min[gene] == "" || min[gene] > $11 { min[gene] = $11; line[gene] = $0 } END { for (gene in line) print line[...
3
From comments it's clear that you're working with a file containing Windows text data. Convert it to a Unix text file using dos2unix:
dos2unix fl.txt
Once it's a Unix text file, you may remove the first line of the file, which contains the Data header, using sed 1d fl.txt.
The resulting lines may be converted to space-delimited columns on a single line by ...
2
Using only standard tools:
$ tr -sc '[:alpha:]' '\n' <file | sort | uniq -c
3 I
2 am
8 groot
1 love
2 me
1 so
This first replaces every non-word character with a newline character. We define a non-word character as "any character that is not an alphabetic character" (this is what -c together with [:alpha:] and \n does on the ...
2
awk -F "+" 'NR%2{nf=NF;l=$0;next}{sep=(nf==NF?" + ":RS);printf "%s%s%s\n",l,sep,$0}' file
For a line by line analysis, let's unwind it:
awk -F "+" '
NR%2{nf=NF;l=$0;next}
{
sep=(nf==NF?" + ":RS)
printf "%s%s%s\n",l,sep,$0
}
' file
Set field separator to +.
If the line number is odd (...
2
One easy method is as follows. The keys to look for are listed one key per line in the file named keys.txt and put on the stdin of perl command whilst the data to be searched is placed as an argument.
$ < keys.txt \
perl -F, -lane '
$.==1 && chomp(@keys = <STDIN>);
$_ .= "$F[0]," for @h{
grep { my $k = $_; grep(($_ eq $k), ...
2
$ md5sum somefile
d68610fdffd770de94818268899d6abb somefile
$ md5sum somefile | { read -r hash pathname; printf '%.4sxxxxxxxxxxxxxxxxxxxxxxxxxxxx %s\n' "$hash" "$pathname" }
d686xxxxxxxxxxxxxxxxxxxxxxxxxxxx somefile
This reads the hash and the pathname into two shell variables and then prints these out with the 28 last characters of ...
2
While this is an awk question, I'd nevertheless throw cut in here as good alternative for this specific task:
cut -d' ' -f1 input.txt > output.txt
The default delimiter (tab) has been replaced with space by -d' ' and the first field selected by -f1.
2
Another perl variant to replace all but the first 4 bytes of the checksum (and the checksum only) with x:
$ md5sum input.txt | perl -pe 's{.{4}\K\S+}{$& =~ s/./x/gr}e'
d632xxxxxxxxxxxxxxxxxxxxxxxxxxxx input.txt
2
If your concat_ws(...) is never nested, that is if there is never something like concat_ws(... foo() ...), so there is never a second set of parentheses within the concat_ws(), then you can match all occurrences of concat_ws(, followed by non-( until a ${hiveconf:RUNDATE} and replace only those. Like this:
$ sed "s/\(concat_ws([^)]*\)\('\${hiveconf:...
1
The MD5 hash is always 32 characters long. The first four characters may be had by cutting with cut -c -4, and the 28 x required to pad out the rest of the hash may be added with printf:
$ md5sum somefile
d68610fdffd770de94818268899d6abb somefile
$ printf '%sxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n' $(md5sum somefile | cut -c -4)
d686xxxxxxxxxxxxxxxxxxxxxxxxxxxx
You ...
1
The problem with Quasímodo's answer is that it also replaces the filename with x's. OP posted a followup question about that. Here's a sed solution that stops at the space:
md5sum always produces a 32-character output for the hash. Instead of detecting a space, you could look for 32-characters then a space and replace the last 28 characters with an X.
...
1
First, you can set FS: -v FS="[]][[:space:]]+" i.e. []] followed by [[:space:]]+. With this you can get the field for making changes:
$ awk -v FS="[]][[:space:]]+" '{print $2}' file
john is bla bla
blue house in Chicago...
accessing the safe... bla bla
Here you "want to convert the first letter found after the space to uppercase&...
Only top voted, non community-wiki answers of a minimum length are eligible
