3

I have a script which uses sed in a loop to delete lines containing words from a list:

for i in $list; do
  sed -i "/$i/d" file
done

$list is created from an SQL query:

list=$(psql -d dbname -t -c "SELECT foo from table ORDER BY column DESC")

The problem is my company recently implemented tighter auditd rules which include all fchown or fchmod syscalls. Using strace, I see every sed -i includes one fchown and fchmod syscall.

With a large list variable, iowait skyrockets and the system actually bogs down because auditd is writing a ridiculous amount of lines to its log.

I'm trying to figure out if there's a way to pass the list variable to sed in one line so the file is only opened and closed once.

I've already asked, any changes to the auditd rules is a non-starter.

Thanks.

  • 2
    What's $list ? Is it a list of words saved in a file, one per line or an array or what ? – don_crissti Jan 23 at 17:12
  • $list is one word per line. – keyboard_banger Jan 23 at 17:15
  • From where do you get list? If all the words or patterns are in a file line by line you could use fgrep -v -f patternfile inputfile > file && mv tmpfile file of grep ... if you have regex patterns. – Bodo Jan 23 at 17:22
  • You could try to convert your list in a string in that way, that the final sed command that is exceuted looks like this: sed -i -E "/(word1|word2|word3)/d" file – finswimmer Jan 23 at 17:25
  • I added how the list is created to the post. I was wrong earlier, it's not one word per line, rather words separated by a single space. – keyboard_banger Jan 23 at 17:35
2

By only running sed once and using a single sed script rather than invoking a zillion in-line commands one-by-one:

sedscript=$(mktemp)
for i in $list; do
  echo "/$i/d" >> "$sedscript"
done
sed -f "$sedscript" -i /path/to/file
rm -f "$sedscript"
  • I'm liking this solution, but I'm getting sed: can't find label for jump to 'mp.Vbqh8BAFGv' The 't' is getting dropped from the mktemp filename somewhere. – keyboard_banger Jan 23 at 17:50
  • This works without creating a temp file: sedscript=$(for i in $list; do echo "/$i/d"; done) && sed -e "$sedscript" -i /path/to/file' – keyboard_banger Jan 23 at 18:15
  • My mistake; sed -e should have been sed -f; -e is "run this command right here" (which is implied by default), while -f is "run the commands in this file". – DopeGhoti Jan 23 at 18:19
0

If you can carefully assign the words in list, you could use bash parameter expansion to replace every space with the regex alternator |, then ask grep to do the work for you:

list='auditd the to'
cp file temp &&  
grep -Ev "${list// /|}" temp > file && 
rm temp 
  • This relies on the assumption (also present in your sed example) that none of the "list" values contain regular expression tokens (that you didn't expect). – Jeff Schaller Jan 23 at 17:27
  • "system-calls", seriously? – poige Jan 24 at 2:08
  • you've added that tag. q-n is irrelevant to this, it's only text data after all – poige Jan 24 at 2:09
  • "Questions concerning the details of how a program uses system calls to interact with the kernel API, what calls are available, how they work etc. View tag" – poige Jan 24 at 2:11
0

If your shell supports arrays, you could do something like

mapfile -t arr < <(psql -d dbname -t -c "SELECT foo from table ORDER BY column DESC")

to put the output of the command into a shell array, then

printf '/%s/d\n' "${arr[@]}" | sed -i.bak -f - somefile

to apply the sequence of deletions to somefile, in place. The usual caveats apply if the fields returned from your psql command might contain characters that are special to sed.

This is similar to @DopeGhoti's solution, except that it avoids the temporary script file (and should handle whitespace in table fields, should that be necessary).

0

If this: psql -d dbname -t -c "SELECT foo from table ORDER BY column DESC" gives you a white-space delimited list of words, try this:

$ psql -d dbname -t -c "SELECT foo from table ORDER BY column DESC" \
| awk -v FS=" " -v OFS="|" '{$1=$1; print "("$0")"}' \
| xargs -I {} sed -i -E '/{}/d' file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.