How to append multiple lines to a file, if these lines doesn't exist in that file?

For example, to add multiple global aliases to /etc/bash.bashrc I use an heredocument:

cat <<-"BASHRC" >> /etc/bash.bashrc
    alias rss="/etc/init.d/php*-fpm restart && systemctl restart nginx.service"
    alias brc="nano /etc/bash.bashrc"

I was criticized that this operation doesn't include a way to check if the lines are already there, and if mistakenly rexecute the heredocument, I could cause redundancy, as well as conflict.


Please ensure not to miss the Bounty message.

  • 2
    How sure can you be that the lines in the file and the new ones are exactly the same? – nohillside Jan 13 '18 at 10:58
  • I cannot, or maybe I can but it will be an extremely complicated mechanism. – Arcticooling Jan 13 '18 at 11:01
  • 1
    Well, you could grep for alias rss= and only add the line of you don‘t get a match. Same for each other alias you want to add. This still doesn‘t cover situations where a user has a separate file for aliases which gets sourced by .bashrc – nohillside Jan 13 '18 at 11:04
  • This was cross-posted at Ask Ubuntu, and I wrote an answer which was already accepted by the OP there. – Mukesh Sai Kumar Jan 13 '18 at 11:25
  • You are quite correct @MukeshSaiKumar. Although there I asked only on aliases to find out if my approach of adding aliases is wrong. Here I desired a general one-line solution and brought aliases only as an example. But given your answer there it's likely I was wrong to publish this question here and answers should be migrated to there. – Arcticooling Jan 13 '18 at 11:44

Simple shell script to add lines from the file newdata to datafile. It should be straightforward to change newdata to a here-doc. This is really not very effective since it calls grep for every (new) input line:

while IFS= read -r line ; do
    if ! grep -Fqxe "$line" "$target" ; then
        printf "%s\n" "$line" >> "$target"
done < newdata 

For each line, we use grep to see if it already exists in the target file, -F for fixed-string match (no regexes), -x for full line match, and -q to suppress output of matched lines. grep returns a falsy error code if it doesn't find a matching line, so append to target file if the negated result is truthy.

More effectively, in awk. This relies on awk being able handle arbitrary lines as keys to an array.

$ awk 'FNR == NR { lines[$0] = 1; next } ! ($0 in lines) {print}' datafile newdata 

The first part FNR == NR { lines[$0] = 1; next } loads all lines of the first input file as keys into the (associative) array lines. The second part ! ($0 in lines) {print} runs on following input lines, and prints the line if it's not in the array, i.e. the "new" lines.

The resulting output contains the new lines, only, so it needs to be appended to the original file, e.g. with sponge:

$ awk 'FNR == NR { lines[$0] = 1; next } ! ($0 in lines) {print}' datafile newdata | sponge -a datafile

Or we could have awk append the lines to the final line, it just requires passing the file name to awk:

$ target=datafile 
$ awk -vtarget="$target" 'FNR == NR { lines[$0] = 1; next } 
                        ! ($0 in lines) {print >> target}' "$target" newdata

To use a here-doc with awk, we'll need to add - (stdin) as an explicit source file, in addition to setting the redirection, so awk ... "$target" - <<EOF


This solution is probably a little different than what you had in mind but I'd check whether an alias is actually defined and only add it to bashrc if it isn't. Assuming you are using a relatively current version of Bash...

# instead of 'alias x=y' form put them in an associative array
declare -A aliases

for key in "${!aliases[@]}"; do
    if ! alias "$key" > /dev/null 2>&1; then
        printf "alias %s='%s'\n" "$key" "${aliases[$key]}" >> /etc/bash.bashrc

unset aliases key

Source this . ./scriptname rather than run it as a normal script.

Note: If you have a bunch of the candidate aliases and are wary of manually converting them to associative array form then you c edit the file with vim and run this commmand: :%s/\valias *([^=]+)\=['"]?([^'"]+)['"]?$/aliases['\1']='\2'/.

OR put your candidate aliases (and nothing else) in a file (e.g. /tmp/aliases.txt) and replace the aliases[..]='..' lines in the script with this:

while IFS= read -r a; do 
    eval "$a"; 
done < <(sed -E "s/alias  *([^=]+)=['\"]?([^'\"]+)['\"]?$/aliases['\1']='\2'/" /tmp/aliases.txt)

Of course, there are other ways besides using AAs but they're clean and easy to work with...and I'm all in at this point. :)


A generic solution for exact matches (not optimized for performance); the file input contains the lines to be checked for:

awk 'FNR==NR { lines[NR]=$0; next; };
    { for(i=1;i<=length(lines);i++) if ($0==lines[i]) matches[i]=1; print; };
    END { for(i=1;i<=length(lines);i++)
        if (matches[i]==0) print lines[i]; }' input file


:> cat input
alias rss="/etc/init.d/php*-fpm restart && systemctl restart nginx.service"
alias brc="nano /etc/bash.bashrc"

:> cat file
alias rss="/etc/init.d/php*-fpm restart && systemctl restart nginx.service"

Output of the awk command:

alias rss="/etc/init.d/php*-fpm restart && systemctl restart nginx.service"
alias brc="nano /etc/bash.bashrc"

Does the order of appended lines matter?

If no, you can try such a solution:

comm -1 -3 <(sort /etc/bash.bashrc) <(sort aliases) >> /etc/bash.bashrc

assuming you have previously saved lines to add in the file aliases.

comm is GNU coreutil (you can see the man page for it), which allows comparing two sorted files line by line.


Hmmmm. How about this...

Put the aliases to add in file new. Then...

cp bashrc bashrc.tmp && comm -23 <(sort -u new) <(sort -u bashrc.tmp) >> bashrc && rm -f bashrc.tmp

We unique sort both the new aliases and the bashrc contents (put in a temp file to avoid race condition when we append to bashrc) and run them through comm. comm does line-by-line comparisons of sorted files and shows unique and common lines. We suppress columns 2 and 3 (-23) so the result is simply those lines that are unique to new and we append those to bashrc. That's it.

(This is a totally different approach than my other answer, focusing on your quest for more of a "one-liner".)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.