2

I have two files, one file contains a list of strings.

+stringa +Dog +Cat
+cat +Tux +elephant

and the second file (csv) contains something like:

"123456 Abc","+Stringx +123","something"
"23456 dEf","+cat +Tux +elephant","Other something"
"34524 xyz","+stringa +Dog +Cat","third something"

the result should be:

"123456 Abc","+Stringx +123","something"
"23456 dEf","+cat +tux +elephant","Other something"
"34524 xyz","+stringa +dog +cat","third something"

How I can change the strings, that match my list of patterns, to lowercase?

My comma-separated values file have about 30 columns and about 1500 rows.

  • 1
    can you add some code you tried? what tools do you have? GNU sed? perl? awk? – Sundeep Nov 27 '17 at 10:34
  • Despite newer versions having an upper- & lower-case variable operator, bash is not a text editor. – Jeff Schaller Nov 27 '17 at 10:48
  • @Sundeep I can not find the right approach. I do not understand perl but it is available. I use Debian 8.9 (Jessie). The point is i like change only the pattern, not the whole file. I have sed, awk, tr – FaxMax Nov 27 '17 at 10:54
3

With GNU sed, assumes that you do not have any meta character in list of strings, + is not a meta character with default BRE

$ # create substitute command for each line
$ sed 's/.*/s|"&"|\\L\&|gi/' f1
s|"+stringa +Dog +Cat"|\L&|gi
s|"+cat +Tux +elephant"|\L&|gi

$ # pass those commands as sed script
$ sed -f <(sed 's/.*/s|"&"|\\L\&|gi/' f1) ip.csv
"123456 Abc","+Stringx +123","something"
"23456 dEf","+cat +tux +elephant","Other something"
"34524 xyz","+stringa +dog +cat","third something"

$ # or save them in a file and use
$ sed 's/.*/s|"&"|\\L\&|gi/' f1 > f2
$ sed -f f2 ip.csv 
  • \L to convert string to lowercase
  • g for replacing all occurrences in a line, i for case-insensitive matching


If you don't have GNU sed

$ # \Q to quote metacharacters
$ # but will have issues if you have \ or $ or @
$ sed 's/.*/s|\\Q"&"|\\L$\&|gi;/' f1
s|\Q"+stringa +Dog +Cat"|\L$&|gi;
s|\Q"+cat +Tux +elephant"|\L$&|gi;

$ perl -p <(sed 's/.*/s|\\Q"&"|\\L$\&|gi;/' f1) ip.csv 
"123456 Abc","+Stringx +123","something"
"23456 dEf","+cat +tux +elephant","Other something"
"34524 xyz","+stringa +dog +cat","third something"


As noted by Stéphane Chazelas, this could lead to code injection vulnerabilities if contents of f1 is not under control

  • 1
    It may be worth noting that those amount to command injection vulnerabilities if the content of f1 is not under your control. – Stéphane Chazelas Nov 27 '17 at 11:23
  • good observation as always :) added a note – Sundeep Nov 27 '17 at 11:31
2

With perl, assuming you want each word in the first file to be turned to lowercase:

perl -pe '
 BEGIN {local $/ = undef; $regex = join "|", map qr{\Q$_\E}i, split " ", <>}
 s/$regex/\L$&/g' file1.words file2.csv

local $/ = undef makes the record separator for the BEGIN block undefined so that the one invocation of <> there, slurps the whole first file (file1.words) in. We split that on whitespace (split " " is special in perl in the same way as awk -F " " is in awk), and join the resulting words with | after having regex-quoted them and made them case insensitive.

So we have a huge regexp that is something like (?i:word1)|(?i:word2)|... which we apply on each line of the second file in the rest of the code.

If it's each string in each line of the first file, then that can be simplified to:

perl -pe '
 BEGIN {chomp (@strings = <STDIN>); $regex = join "|", map qr{\Q$_\E}i, @strings}
 s/$regex/\L$&/g' < file1.strings file2.csv

There, we open the first file on stdin instead of passing it as argument. <STDIN> returns a list of its lines from which we remove the delimiters with chomp, and join with | as above.

If you don't want it to be limited to ASCII characters, add the -Mopen=locale option.

  • 1
    @Sundeep. That was my initial interpretation, but now that you mention it, you're probably right the lines of file1 are probably to be the strings to be matched. See edit. – Stéphane Chazelas Nov 27 '17 at 11:45
2

AWK solution (for your current input):

Assuming that the 2nd field is of main interest and values in search file are double-quoted.

awk 'NR==FNR{ $0="\042"$0"\042"; a[$0]; next }
     $2 in a{ $2=tolower($2) }1' patterns FS=',' OFS=',' file.csv
  • $0="\042"$0"\042" - wrap a pattern line with double quotes while iterating through the lines of patterns file

  • a[$0] - capturing a pattern line into array a

  • $2 in a{ $2=tolower($2) } - if the 2nd field value from line of file.csv file is in the list of patterns(i.e. array a) - convert all characters in it to lowercase $2=tolower($2)


The output:

"123456 Abc","+Stringx +123","something"
"23456 dEf","+cat +tux +elephant","Other something"
"34524 xyz","+stringa +dog +cat","third something"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.