New answers tagged text-processing
0
$ join -t, <( sort file1 ) <( sort file2 )
burnj01,Joe Burns,0001
burnj01,Joe Burns,0002
burnj01,Joe Burns,0010
burnj01,Joe Burns,2000
steves02,Santosh Steve,2048
steves02,Santosh Steve,2049
steves02,Santosh Steve,2091
This sorts the two files and passes the sorted contents to the join utility. The join utility does a relational JOIN operation ...
0
You should specify the field separator FS and OFS is used for output field separator.
awk 'BEGIN { FS=OFS="," }
NR==FNR { saving[$1]=$2; next }
($1 in saving){ print $1, saving[$1], $2 }' file1 file2
4
grep is the simplest command for this.
Assuming you have files in one folder then
grep 'Total Energy =' dir/* > results.txt
Without more information on the way your files are named / organized, I can't be more precise. Hopefully this will let you solve it
Other solutions might use sed or awk, but grep is probably the simplest
0
Throwing another one on the pile...
csvprintf
It works like printf(1).
For example, with this input:
FirstName,LastName,IdNumber
George,Washington,1
Betsy,Ross,2
And this command line:
$ cat input.csv | csvprintf -i '%{IdNumber}04d: %{LastName}s, %{FirstName}s\n'
You get this output:
0001: Washington, George
0002: Ross, Betsy
Disclaimer: I wrote this
0
awk '{if ($0 ~ /puppies/){x=$0;gsub("puppies","bunnies",$0);gsub("I","you",$0);print $0"\n"x}else{print $0}}' filename
output
you have bunnies cute
I have puppies cute
I have two kitties
you have three bunnies cute
I have three puppies cute
I have four kitties
Python
#!/usr/bin/python
import re
k=re....
0
Test your crashing file with the file-command in Linux and compare it with non-crashing files.
Most probable, there is a mismatch in file-encoding, and maybe some unwanted auto-detection involved.
Linux offers UTF-8 on most systems since 20 years, and you shouldn't change that. Windows has been later to the party, but more recent installations should use utf-...
3
Some more approaches:
sed '
/^1234/!b
:1
s/^\(.\{10,19\}\) /\10/
t1' < file
perl -pe 'substr($_, 10, 10) =~ y/ /0/ if /^1234/' < file
perl -pe 's{^1234.{6}\K.{10}}{$& =~ y/ /0/r}e' < file
gawk -v FIELDWIDTHS='10 10 *' -vOFS= '
/^1234/ {gsub(/ /, "0", $2)}; {print}' < file
(With perl, add the -C option or -Mopen=...
3
You were almost there...a slight change in your regex plus encasing the s/// in a do-while loop until there are no more spaces left in string position from 11 to 20.
sed -e '
:loop
s/^\(1234.\{6\}[0-9]\{0,9\}\) /\10/
t loop
' file
12345678909876456000qwerty
12345678909870000000qwerty
23456789011234567 qwerty
1
if a bash script is valid, you coud use the sed quit command:
#!/bin/bash
sed -E '
# If we find an username
/username/ {
# make the sobstitution of the username
s/^([[:space:]]*username:)/\1toto/g
n # Take the next line with the password
# Sobstitute the password
s/^([[:space:]]*password:).*$/\1totopw/g
...
1
With gnu awk
awk -F '^\\s+|:' '
BEGIN {
a["username"] = "toto"
a["password"] = "totopsw"
}
($2 in a) {
sub(/:.*/, ": " a[$2])
delete a[$2]
};1
' file
set fields separator to be colon or a run of whitespace
initilaize array a with keywords to search for n their corresponding ...
5
Another sed alternative:
sed '1,/username/{/username/ s/:.*/:toto/};
1,/password/{/password/ s/:.*/:totopsw/}' infile
1,/regex/ start from the first line up-to first line that matched with given regex (here it's username string), change the "username"; doing the same for "password" part.
4
If the file is longer, here is an awk-based solution that processes it line-wise:
awk '/^[[:space:]]+username/ && !u_chng{sub(/:.+$/,": toto"); u_chng=1}
/^[[:space:]]+password/ && !p_chng{sub(/:.+$/,": totospw"); p_chng=1} 1' input.yml
This will check each line whether it starts with username or password, ...
4
If your file is small enough to fit into memory, you can try reading the entire thing as a single record. That way, sed will only make the substitution for the first time it sees the pattern on the "line" (record):
$ sed -Ez 's/(username:)[^\n]*/\1toto/; s/(password:)[^\n]*/\1totopsw/' file.yaml
spring:
datasource:
url: url
username:toto
...
0
A few different approaches:
Select whatever is between host " and " is:
$ sed -En 's/.*host "([^"]*)" is.*/\1/p' file
ADMK05786HTREDIRECT_JA3A
KOPYUTRCEPORT_JA3A
ADMK05786HTSRM_JA3A
ADMK05786HTIAS_JA3A
ADMK05786HTOTN_JA3A
ADMK05786HTWPM
OCAIDEKD_JA3A
ADMK05786HTSC_JA3A
or, using grep with perl compatible regular expressions (-P) ...
1
you are mixing grep's regex and sed's substitution.
use either
sed -e '/db\.username/s/"[^"]*"/"foo"/' txt
db.username="foo"
db.password="toto"
where
/db\.username/ tell sed to operate on line whith db.username (. is special caracter in sed)
s replace
"[^"]*" a quote, any number of non quote, a ...
0
grep -o '"[^"]\+"'
This will return the info between the double quotes. There are definitely more robust ways to do this but if data is always in that format it will work and even if names change it will work.
3
With perl:
perl -Mopen=locale -pe 's/\h*\[\d+(\p{dash}\d+)?\]//g' < your-file
Where \p{dash} matches characters in the dash punctuation category (such as -֊־᐀᠆‐‑‒–—―⁓⁻₋−⸗⸚⸺⸻⹀〜〰゠︱︲﹘﹣－).
That deletes the horizontal whitespace before but not after those [x]. That means however that on your sample, in vivo [47] . Due becomes in vivo . Due. You could improve ...
1
You need to match [ then one or more digits, then ].
The [ and ] are considered special characters in regular expressions so you have to escape them with a leading backslash
A digit can be represented either as [0-9] or as [[:digit:]]
You need multiple digits so use + to indicate one or more
So
sed -E 's/\[[[:digit:]]+\]//g'
To match a number range, such ...
1
Try this:
# it also removes the extra spaces
$ sed -e 's/\[[0-9–]*\]//g;s/[[:blank:]]\+/ /g;s/[[:blank:]]\([\.?!:;,]\+\)/\1/g' file
Output:
Local delivery of a wide selection of growth factors (e.g.
platelet-derived growth factor and bone morphogenetic proteins, for
example) from electrospun membranes/scaffolds has demonstrated to
enhance cell activity in ...
0
gnu sed with branches:
sed '
s/puppies/&/g; tfind
n
:find
h
s/puppies/bunnies/g
s/^I/You/
G
'
gawk :
awk '$0~/puppies/ {
original=$0
sub(/puppies/, "bunnies")
sub(/^I/, "You")
print $0, original
next
}
{ print }
'
0
Using the sed editor :
sed -e '
h;/puppies/G
s//bunnies/
/\n/s/^I/You/
' file
Alternatively with awk
awk '{h=$0}
sub(/puppies/, "bunnies") {
sub(/^I/, "You")
$0 = $0 ORS h
};1
' file
1
With GNU sed:
sed 'h; s/puppies/bunnies/g; T; s/I/You/; G' < your-file
1
Using sed, and assuming that the trigger is the substring cute at the end of the line:
sed '/cute[[:blank:]]*$/ { h; s/^I/You/; s/[[:alpha:]]* cute/bunnies cute/; G; }' file
The [[:blank:]]* in the pattern allows for trailing spaces or tabs at the end of the line (the first line has a trailing space in the question).
This would produce the text
You have ...
1
Using awk:
awk 'function prnt() { print edit ORS hold; hold=edit="" }
hold { prnt() }
/puppies/ { hold=$0; $1="you"; gsub("puppies", "bunnies"); edit=$0; next }
END{ if(hold) prnt() }1' infile
0
I came up with a simple example that calculates the SHA256 hash of each file as they appear in the directory. Harmless and easy to verify if it got that wrong. You could use any other processing.
#!/bin/sh
inotifywait -m "$@" -e create | while read dir event file; do
if [ -f "${dir}${file}" ]; then
sha256sum "${dir}${file}...
0
Use a soft link ln -s blabla_3300.dat .next.
Tell your script to read .next, then rm .next;ln -s blabla3400.dat .next.
0
Gilles had the solution:
# xxd -g4 fsl,mf-mix-wakeup-irq
00000000: 07c00000 00007d00 00000000 01400640 ......}......@.@
Note that various od options are invalid in busybox:
# od -tx4 fsl,mf-mix-wakeup-irq
od: invalid option -- 't'
BusyBox v1.31.0 (2021-02-24 17:49:57 UTC) multi-call binary.
Usage: od [-aBbcDdeFfHhIiLlOovXx] [FILE]
2
gawk -F, '
{ gsub(/"/,"") }
FNR==1{
delete timestamp;
start=strftime("%Y %m %d 00 00 00", systime()-86400);
for(min=0; min<1440; min++)
timestamp[strftime("%F %H:%M", mktime(start)+min*60)]
$3="Info"; $5="Output"; hdr=$1 FS $3 FS $5;
yday=strftime("%...
5
You need to specify the field separator and ensure that you count the fields right. In your case, the field separator is |. But this also implies an "empty" field at the beginning of each line, hence your "actual" field number is 5:
~$ sort -t'|' -k5 file
| 2020-10-20T05:46:28.000Z | | eb82a308-543b-4e1a-8f08-e17d2655e5c7 | 2020-10-20T06:...
0
Using any awk in any shell on every UNIX box:
From Ed Morton answer, but replacing each character only once (much faster):
$ echo 'aaabefhhhhhthkkd' |
awk '{
while ( length() ) { # process all letters.
let = substr($0,1,1) # extract the first letter
if ( gsub(let,"") == 1 ) { # remove ...
1
The configuration file for the Whisker Menu is saved in your xfce4 panel directory:
~/.config/xfce4/panel/whiskermenu-1.rc
The actions defined at the bottom of the file contain the same properties as in the "Search Actions" dialog, i.e. name, pattern, command and a boolean regex flag.
$ tail -18 ~/.config/xfce4/panel/whiskermenu-1.rc
[action2]
...
2
Alternative awk:
awk 'BEGIN{ FS=OFS="\t" }
NR==FNR{ mutations[$1] =$2; next }
FNR>1 {
split($3, muts, "," );
for(x in muts) {
if (gsub( ",?"muts[x]",?", "", mutations[$2])>0) delete muts[x] }
}
FNR==1 { $4="missing_mutation"; $5="...
3
The following awk program, which uses GNU awk for convenience (in form of "true double-index arrays"), should do the trick. It is intended to be called with your two files as argument in the order B.txt A.txt (as you already did in your code example).
#!/bin/awk -f
BEGIN{FS=OFS="\t"}
NR==FNR {
n_tmp[$1]=split($2,buff,/,/)
for (i=...
-1
Command:
awk 'NR <5 ||NR>4 && !/^$/' filename
output
1
2
3
4
5
5
6
7
8
9
1
There are a number of problems with your command, fortunately they're all easy to resolve:
As Quasimodo mentioned in their comment, ’ is not a valid character for delimiting strings within Bourne Again Shell (aka "BASH", the default shell interpreter on Kali Linux). Replacing that with single quotes (') or double quotes (") will be required. ...
0
This uses GNU awk for multi-dimensional arrays:
gawk '
BEGIN {
FS = "[\t,]"
OFS = "\t"
}
FILENAME == ARGV[1] {
for (i = 2; i <= NF; i++)
B[$1][$i] = 1
next
}
FNR == 1 {
print $0, "number", "total_mut"
next
}
!($2 in B) {
...
4
sed '1,3b;/./!d'
Branch in range 1,3 or delete if empty.
1
With GNU awk for the \< and \> (Word-boundaries anchors):
awk 'BEGIN{ FS=OFS="\t" }
NR==FNR{ mutations[$1] =$2; next }
{
split($3, muts, "," );
for(x in muts) { tmp=mutations[$2]; c+=sub( "\\<"muts[x]"\\>", "", tmp) }
}
FNR==1 { c="number" }
{ print $0, (c?c:""); c=0 }...
7
Another awk solution:
awk 'NR<=3||NF' file
This will print all lines where either the line number is smaller or equal 3, or which have at least one "field", defined by default as at least one non-blank character. This way, you can also exclude lines that contain no "text" but are only "visually" empty, i.e. that still ...
3
You were almost there. Just needed to add the following steps.
sed -ne '4,${s~^$~~;t};p' < "${File}"
select the range 4,$
try the dummy substitution s/^$//
in case the s/// succeeds, skip to next record t
else fall back n print for either these in-range nonempty lines or all out-of-range lines };p
2
an awk solution:
awk '((NR > 3 && $0!~/^$/) || NR <= 3) { print }' file
1
awk 'BEGIN{ OFS=FS="\t" }
NR==FNR{ clade[$1]=$2; next } # save clade, mutation of B.tsv in array
FNR==1{ print $0, "number"; next } # print header
!($2 in clade){ print; next } # no match -> print record
{ # else...
split($3 "," clade[$2], tmp, ",") ...
8
try:
sed '4,$ {/^$/d}' infile
start from the 4th line until end of the file, delete the empty lines.
The problem with your command is that you replace empty line again with empty string in s~^$~~g (which is same as s~^$~~) and you are not deleting it.
Note: also since you use different delimiter other than default slash /, to use this style ~^$~d you need ...
0
The method is to make an array indexed by clade and mutation, from the B file. Then iterate the mutations from the A file.
Somewhat tricky to deal with a tab-separated file, especially keeping the number of columns where there is no clade.
We define the necessary column numbers for the A file as cClade and cMut, and changed these to match the full data ...
3
Try:
awk -F, -v yesterday="$(date -d'-1day' +'%F')" '
BEGIN{ for(min=0; min<1440; min++){
mins = "date +%F\" " "\"%T -d\"" min "minutes" yesterday"\""
mins |getline yday_tmp; close(mins);
timestamp["\"" yday_tmp "\""] }
...
0
A brute-force approach, to fill in the gaps from your existing solution: it tests 1440 files, but, hey, it's only once a day, and it's yesterday date. In Bash:
(a) Get yesterday's date in both formats: dt=20210103 and 2021-01-03.
(b) Run a for loop: for hm in {00..23}{00..59}.
(c) For any file ${dt}${hm}00.csv not present: printf the required header and ...
0
Using Emacs, you could first select all text in your buffer. Then invoke
M-x downcase-region
1
$ cat tst.awk
BEGIN { FS=OFS="\t" }
{ key = (FNR>1 ? $1 : RS) }
NR == FNR {
$1 = ""
map[key] = $0
next
}
{ print $0 map[key] }
$ awk -f tst.awk B.tsv A.tsv
Sample ID Internal Control Result Consensus clade substitutions deletions
4686427 Pass Not Detected Not Available 161 no yes
...
3
Use join
join -t, -a1 B.csv C.csv
The -a1 means left outer join (i.e show lines from file1 that are not in file2)
If commas at the end of unpaired lines really matter
(join -t, B.csv C.csv ; join -t, -v1 B.csv C.csv | perl -pe "s/$/,/" ) | sort
3
Using awk without dis-ordering lines of the original files, but require to loading the first file into the memory and you would need to care to don't run on big file that cannot fit into your memory.
awk 'BEGIN { FS=OFS="," }
NR==FNR { hold[$1]=$2; next }
{ print $0, hold[$1] }' fileC fileB
for such a case when there was a key ...
Top 50 recent answers are included
