7

I have several text files. All these text files need to undergo a set to editing which can be done in vim. I want to automate this. vim has several commands for replacement. Suppose the text files need to undergo the following replacements:

  1. replace boy by Boy: %s/boy/Boy/g
  2. replace girl by Girl: %s/girl/Girl/g
  3. delete empty lines: g/^$/d

This is just a simple example, Is there a way to write all these rules and then automate this on several files?

  • I changed your question from vi to vim since I assumed that's what you were using. Please confirm this. – slm Sep 17 '14 at 15:57
  • @braiam - what about using vim? – slm Sep 17 '14 at 15:57
  • @slm I presume the tag is enough... the original title didn't reflect what was the task at hand and was actually misleading. – Braiam Sep 17 '14 at 15:58
13

Yes there is a way to automate this. And it starts with selecting the right tool, for the job.
In this case you should be using e.g sed and not try to bend vi which was designed for interactive use (and not for automation).

The replacement syntax for sed is largely the same as the one for vi.

 sed -i.backup 's/boy/Boy/g' file-name-1 file-name-2 ...
  • 2
    Apparently there's an echo in here, 4 others felt to say exactly what you did 8-) – slm Sep 17 '14 at 11:45
  • 3
    sed is a stream editor, vi is a visual text file editor, built on top of ex/ed (and the s command is an ex command in vi). Most sed implementations don't have a -i. ex and ed are standard Unix commands. So yes, it can be done with some sed implementations, but that is not to say it is the right tool. – Stéphane Chazelas Sep 17 '14 at 12:13
  • 3
    I didn't write sed is "the right tool". There are multiple tools, hence the "e.g.". As far as I remember ed didn't operate on multiple files and you would have to provide read and write operations on its stdin. However the last time I can remember I used ed was because vi was not available on a PDP-11 in the mid 80s, so I might be out of date with its capabilities – Anthon Sep 17 '14 at 12:31
10

Vim scripts (gvim,vim) can be elegant and are very easy to adapt

vi -s edit.vim  test.txt

where edit.vim contains (the :wq is optional)

:%s/boy/Boy/g
:%s/girl/Girl/g
:g/^$/d
:wq

where test.txt contains

boys & girls

boys & girls
boy & girl

boys & girls

here's a generic vim script to clean up a "mucky" text file

:" clean.vim
:" clean up a text file
:" delete DOS returns (actually superfluous because of non-ASCII
deletion)
:%s/\r//eg
:" delete any non-ASCII including invisibles
:%s/[\x00-\x1f\x80-\xff]/ /eg
:" compress multiple spaces
:%s/\s\s\+/ /eg
:" delete end of line spaces
:%s/\s\+$//e
:" compress multiple blank lines
:v/./,/./-j
:" sort eliminating duplicate lines
:%sort -u

remember you can also source the script from within vim

:source clean.vim
3

Why not use sed ?

With sed you could easily loop over the files in the directory:

for f in *.txt
do
sed 's/boy/Boy/g;s/girl/Girl/g;/^\s*$/d' $f > tmp
mv tmp $f
done

Above example would change boy->Boy, girl->Girl and delete empty the lines.

  • 2
    You do not need to do the for loop, give sed a list of files directly as in Anthon's answer below or the glob, *.txt directly. Also the mv is unnecessary. Use sed's -i.ext switch to create a backup. – slm Sep 17 '14 at 12:00
  • 1
    I didn't know that trick I will try to remember it the next time. Thanks for the tip and explanation @slm – janrpn Sep 17 '14 at 18:06
2

And an inline vi script:

$ vi test.txt -c '%s/boy/Boy/g | %s/girl/Girl/g | g/^$/d | wq'
2

NOTE: Using sed is the appropriate way to do this, as shown by @Anthon's answer!


Just to show you how you could do this in vim. You can use the argdo command within vim. From the :help argdo in vim:

:argdo[!] {cmd}         Execute {cmd} for each file in the argument list.
Example:
    :args *.c
    :argdo set ff=unix | update

This sets the 'fileformat' option to "unix" and writes the file if it is now changed. This is done for all *.c files.

Example:
    :args *.[ch]
    :argdo %s/\<my_foo\>/My_Foo/ge | update

This changes the word "my_foo" to "My_Foo" in all *.c and *.h files. The "e" flag is used for the :substitute command to avoid an error for files where "my_foo" isn't used. :update writes the file only if changes were made.

The command :args <pattern> tells :argdo which files you want to run the following {cmd} against. Once defined you run :argdo {cmd} | update where {cmd} can be your s/boy/Boy/g substitution.

Are the files all opened during this operation?

It would appear so. I did the following test to confirm this.

$ for i in {1..3};do echo "dog" > file${i}.txt;done

$ head file*.txt
==> file1.txt <==
dog

==> file2.txt <==
dog

==> file3.txt <==
dog

Now go into vim and do the following commands:

:args file*.txt

You can confirm that they've all been opened:

:ls
  2 %a   "file1.txt"                    line 1
  3      "file2.txt"                    line 0
  4      "file3.txt"                    line 0

Do :argdo ...:

:argdo %s/dog/cat/g | update
"file1.txt" 1L, 4C written
"file2.txt" 1L, 4C written
"file3.txt" 1L, 4C written

So keep this in mind if you intend to use this approach. It inefficiently opens every file that matches your pattern to :args and applies the command to them in turn.

0

sed is the tool for you. Most of the commands that you will use in vi will be available in sed and so, there is not much learning curve. The commands in vi that are started with a : are based on the underlying editor ed and they are also available in sed. You will use those commands without the : and they are, by default, applied to the entire file. For example, for your example, you will do

sed -i 's/boy/Boy/g
s/girl/Girl/g
/^$/d' file1 file2 ...
0

Assuming your vi is actually vim or gvim, it can certainly be implemented in vimscript, but it may be complicated to work on multiple files at once.

Better the right tool for the job: sed, the stream editor, which used a very similar syntax like vi, but is intended to work on files as a stream line by line, without interaction.

With the option -i, sed can do changes in multiple files. -i.backup moves an original filefoo.txttofoo.txt.backup, and applies changes in place tofoo.txt`:

sed -i.backup -e 's/boy/Boy/g' -e 's/girl/Girl/g' -e '/^$/d' f1.txt f2.txt ...

You can run this inside vi too with :!:

:!sed -i.backup -e 's/boy/Boy/g' -e 's/girl/Girl/g' -e '/^$/d' f1.txt f2.txt ...
0

I second @Anthon's answer. Use sed:

$ sed -e "s/boy/Boy/g" -e "s/girl/Girl/g" -e '/^\s*$/d' file

NOTE: (This also deletes lines with only whitespaces)

Add -i to sed if you want to replace in-place (your files are under version control, right?). For multiple files:

$ for $f in $(ls mydir); do
   sed -e "s/boy/Boy/g" -e "s/girl/Girl/g" -e '/^\s*$/d' $f
done
0

Alternatives

Unless you really need special Vim capabilities, you're probably better off using non-interactive tools like sed, awk, or Perl / Python / Ruby / your favorite scripting language here.

That said, you can use Vim non-interactively:

Silent Batch Mode

For very simple text processing (i.e. using Vim like an enhanced 'sed' or 'awk', maybe just benefitting from the enhanced regular expressions in a :substitute command), use Ex-mode.

REM Windows
call vim -N -u NONE -n -es -S "commands.ex" "filespec"

Note: silent batch mode (:help -s-ex) messes up the Windows console, so you may have to do a cls to clean up after the Vim run.

# Unix
vim -T dumb --noplugin -n -es -S "commands.ex" "filespec"

Attention: Vim will hang waiting for input if the "commands.ex" file doesn't exist; better check beforehand for its existence! Alternatively, Vim can read the commands from stdin. You can also fill a new buffer with text read from stdin, and read commands from stderr if you use the - argument.

Full Automation

For more advanced processing involving multiple windows, and real automation of Vim (where you might interact with the user or leave Vim running to let the user take over), use:

vim -N -u NONE -n -c "set nomore" -S "commands.vim" "filespec"

Here's a summary of the used arguments:

-T dumb           Avoids errors in case the terminal detection goes wrong.
-N -u NONE        Do not load vimrc and plugins, alternatively:
--noplugin        Do not load plugins.
-n                No swapfile.
-es               Ex mode + silent batch mode -s-ex
                Attention: Must be given in that order!
-S ...            Source script.
-c 'set nomore'   Suppress the more-prompt when the screen is filled
                with messages or output to avoid blocking.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.