0

I have two text files containing a list of URLs. I want to compare the two files and print out the differences. I know that we can use 'diff' but here's the problem. The urls on one file are full e.g. http://google.com while on the other file they don't have the http://, e.g. google.com

File 1:

http://google.com

File 2:

google.com

I want these to be ignored and only actually different URLs to be printed. Is it possible?

5
  • 1
    Can you show us a slightly longer sample of both files, or at least a longer list of examples? Commented Jan 21, 2016 at 13:02
  • Yes , you can write a python script to do that.
    – Ijaz Ahmad
    Commented Jan 21, 2016 at 13:03
  • 2
    Still not enough - do the two files have a sorted list of URL's or are they a random assortment of URLs that may not be in the same order in each file? Are the full URLs or just domain names prefixed by http:// (i.e. do the URLs contain slashes, punctuation, numbers, etc.) Does the order matter, or is it purely 'does this URL exist in this other file'. We need to see a real sample set or a more detailed description and as @don_crissti suggests, we need to know what output you're expecting. Commented Jan 21, 2016 at 13:24
  • Isn't this what diff does? Commented Jan 21, 2016 at 17:26
  • Unfortunately, i don't have a longer list of examples. The list are not in the same order, so we would need to sort it first. Some of the URLs do contain slashes and punctuation. The order doesn't matter, i just need to check whether a URL exists in the other file. Thanks!
    – Koshur
    Commented Jan 24, 2016 at 7:02

4 Answers 4

3

Yes, you can e.g. use sed to remove certain things before piping it into diff:

$ diff file1 file2
1,3c1,3
< http://google.com/search
< http://www.google.com
< http://example.com
---
> google.com/search
> google.com
> example.com

$ < file1 sed 's|https\{0,1\}://||g' | diff - file2
2c2
< www.google.com
---
> google.com
2
  • @StéphaneChazelas please explain your edit. I'd like to understand this < file1 sed magic.
    – s3lph
    Commented Jan 21, 2016 at 16:24
  • 1
    That's just redirection. Redirections can be anywhere on the line. cmd arg < file, cmd < file arg, < file cmd arg. Some people do UUOCs because they like to have the input on the left. If you're more comfortable with the redirection on the right, you can change it to sed 's|https\{0,1\}://||g' < file | diff - file2` or since sed can also open the file by itself: sed 's|https\{0,1\}://||g' file1 | diff - file2. Commented Jan 21, 2016 at 16:33
0

Take, for example, these two files:

$ cat file1
http://google.com
example.com
http://foobar.org

$ cat file2
example.com
google.com
foobar.org
unique.url

I would just use a tool like sed to remove everything up to the last occurrence of //. Also, in order to use diff, both files need to be sorted. Combining the two gives us:

$ diff <(sed 's#.*//##' file1 | sort) <(sort file2) 
3a4
> unique.url

Alternatively, use comm -3:

$ comm -3 <(sed 's#.*//##' file1 | sort) <(sort file2) 
    unique.url

You can remove the leading whitespace with sed again:

$ comm -3 <(sed 's#.*//##' file1 | sort) <(sort file2) | sed 's/^\s*//'
unique.url

Another approach would be to remove the http:// from both files and print what remains, then pass it through uniq -u which will only print unique lines. Since those URLs which are present in both files won't be unique, this will only print the ones that are present on one of the two files:

$ sed 's#.*//##' file1 file2 | sort | uniq -u
unique.url
2
  • Is it possible to check each line against all the lines in other file? I mean what if even after sorting, some hosts exist in different lines?
    – Koshur
    Commented Jan 25, 2016 at 12:12
  • @Koshur I don't understand. This does check everything. By the way, if one of the answers here solved your issue, please take a moment and accept it by clicking on the check mark to the left. That will mark the question as answered and is the way thanks are expressed on the Stack Exchange sites.
    – terdon
    Commented Jan 25, 2016 at 12:14
0

Yes, you can skip the part (e.g. http://) with awk's field separator.

from manpage:

NAME
       awk - pattern scanning and processing language

-F fs
       --field-separator fs
              Use fs for the input field separator (the value of the FS predefined variable).

Example:

$ cat file1
http://google.com
http://gnu.org
http://fsf.org
http://linux.stackexchange.com

$ cat file2
google.com
gnu.org
fsf.org
unix.stackexchange.com

$ cat file1 | awk -F "http://" '{print $2}'
google.com
gnu.org
fsf.org
linux.stackexchange.com

$ cat file1 | awk -F "http://" '{print $2}' | diff - file2
4c4
< linux.stackexchange.com
---
> unix.stackexchange.com

Note:

  • You can also use -F "://" generally for https:// or http://
  • The files should be sorted before comparision
0

For anyone interested here's a script for comparing URLs that contain the protocol (e.g. https://) in two or more files, no matter where the URLs are positioned in the files (not necessarily at the beginning of a line):

#!/bin/sh

# uniql.sh

# UNIQue Links - a "uniq" and "diff" utilities-like .sh 
# script (bash, zsh compatible) that can find 
# different/common URL web links in one file compared to 
# a group of files

# Please note that: for simplicity, in this script, only 
# URLs containing "://" are taken into consideration, 
# although there can be URLs that do not contain it 
# (such as mailto:[email protected])

GetOS () {
    
    OS_kernel_name=$(uname -s)
    
    case "$OS_kernel_name" in
        "Linux")
            eval $1="Linux"
        ;;
        "Darwin")
            eval $1="Mac"
        ;;
        "CYGWIN"*|"MSYS"*|"MINGW"*)
            eval $1="Windows"
        ;;
        "")
            eval $1="unknown"
        ;;
        *)
            eval $1="other"
        ;;
    esac
    
}

DetectShell () {
    eval $1=\"\";
    if [ -n "$BASH_VERSION" ]; then
        eval $1=\"bash\";
    elif [ -n "$ZSH_VERSION" ]; then
        eval $1=\"zsh\";
    else
        eval $1=\"undetermined\";
    fi
}

PrintInTitle () {
    printf "\033]0;%s\007" "$1"
}

PrintJustInTitle () {
    PrintInTitle "$1">$where_to_print
}

trap1 () {
    CleanUp
    printf "\nAborted.\n">$where_to_print
}

CleanUp () {
    
    #Restore "INTERRUPT" (CTRL-C) and "TERMINAL STOP" (CTRL-Z) signals:
    trap - INT
    trap - TSTP
    
    #Clear the title:
    printf "\033]0;%s\007" "">/dev/tty
    
    #Restore initial IFS:
    #IFS=$old_IFS
    unset IFS
}

ProcFindLinks () {

    if [ -z "$count" ]; then
        #For files_1 and files_2: initialise next variables
        count="0"
        mask="00000000000000000000"
    fi
    
    #extract number from the end of files_N:
    file_index="$1"
    file_index="${file_index##*"_"}"
    
    #index_amplifier is "1" for "--subshell1" and "--subshell3" flags or "2" for "--subshell2" and "--subshell4" flags, and otherwise "1"
    if [ "$file_index" = "1" ]; then
        #files_1:
        title_index="$(( 1 + ( $subshell_flag * ( $index_amplifier - 1 ) ) * 2 ))"
        file_group_index="1"
    else
        #files_2..N
        title_index="$(( 2 + ( $subshell_flag * ( $index_amplifier - 1 ) ) * 2 ))"
        file_group_index="2"
    fi
    
    for link in $(\
        eval cat "\$$1" |\
        eval "$sed_command1" |\
        eval "$sed_command2"\
    ); do
        count_prev=$count
        count=$((count+1))
        if [ "${#count_prev}" -lt "${#count}" ]; then
            mask="${mask%?}"
        fi
        number="$mask$count"
        printf '%s\n' "$file_group_index $number $link"
        printf "\033]0;%s\007" "Links found [$title_index]: $((count))...">$where_to_print
    done;
    #Prepare initialisation for files_2:
    if [ "$1" = "files_1" ]; then count=""; fi
}

ProcFindLinksShowLineNumbers () {
    
    if [ -z "$count" ]; then
        #For files_1 and files_2: initialise next variables:
        count=0
    fi
    
    #extract number from the end of files_N:
    file_index="$1"
    file_index="${file_index##*"_"}"
    
    if [ "$file_index" = "1" ]; then
        #files_1:
        file_group_index="1"
    else
        #files_2..N
        file_group_index="2"
    fi
    
    for link in $(\
        eval cat "\$$1" |\
        eval "$sed_command1" |\
        eval "$sed_command2"\
    ); do
        count=$((count+1))
        number="$count"
        printf '%s\n' "File: $file_group_index Link: $number"
        printf '%s\n' "$link"
        printf "\033]0;%s\007" "Links found [$file_group_index]: $((count))...">$where_to_print
    done
    #Prepare initialisation for files_2:
    if [ "$1" = "files_1" ]; then count=""; fi
}

DisplayHelp () {
    printf "\n"
    printf "uniql - UNIQue URL web Links - find different/common URL web links in one file compared to a group of files and viceversa\n"
    printf "\n"
    printf "    What it does:\n"
    printf "        - compares the URL web links in the files provided as parameters - <file1>, <file2>, ..., <fileN> - and shows the different/common web links in <file1> compared to the group of files <file2>, ..., <fileN> and viceversa (see flags below):\n"
    printf "    Syntax:\n"
    printf "        <caller_shell> '/path/to/uniql.sh' <file1> <file2> ... <fileN> [flags]\n"
    printf "        - where:\n"
    printf "            - <caller_shell> can be any of the shells: bash, zsh\n"
    printf "            - '/path/to/uniql.sh' represents the path of this script\n"
    printf "            - <file1>, <file2>, ..., <fileN> represent the files to be compared: the web links in <file1> are compared with all the web links in <file2>, ... <fileN> and viceversa\n"
    printf "            - [flags] can be:\n"
    printf "                --help or -h\n"
    printf "                    Displays this help information\n"
    printf "                --common or -c\n"
    printf "                    - compare the URL web links in the files provided as parameters - in the group of files: <file1>, <file2>, ..., <fileN> - and show the common web links that are found in all files\n"
    printf "                --different or -d\n"
    printf "                    - compare the URL web links in the files provided as parameters - <file1> and the group of files: <file2>, ..., <fileN> - and show the missing web links that are found in <file1> but not in the group <file2>, ..., <fileN>, and viceversa\n"
    printf "                --preserve-order or -p\n"
    printf "                    - preserve the order and the occurences in which the links appear in files 1..N in this order (where file 1 is the first file given as parameter and file N is the last file given as parameter)\n"
    printf "                --domains\n"
    printf "                    - compare and print only the URL domains (plus subdomains) not the full URLs\n"
    printf "                --domains-full\n"
    printf "                    - compare only the URL domains (plus subdomains) but print the full URLs\n"
    printf "    Output:\n"
    printf "        For the \"-d\" flag:\n"
    printf "            - lines starting with '<' signify web links from <file1>\n"
    printf "            - lines starting with '>' signify web links from <file2>, ..., <fileN>\n"
    printf "    Notes:\n"
    printf "               - for simplicity, in this script, only URLs containing \"://\" are taken into consideration, although there can be URLs that do not contain it (such as mailto:[email protected])\n"
    printf "\n"
}

GetOS OS

#################################################################################
##    Uncomment the next line if your OS is not Linux or Mac (and eventually   ##
##    modify the commands used (sed, sort, uniq) according to your system):    ##
#################################################################################
#OS="userdefined"

DetectShell current_shell

if [ "$current_shell" = "bash" ]; then
    current_script_path="${BASH_SOURCE}"
elif [ "$current_shell" = "zsh" ]; then
    current_script_path="${(%):-%N}"
elif [ "$current_shell" = "undetermined" ]; then
    printf "\Error: This script was designed to work with bash and zsh shells.\n\n">$where_to_print
    CleanUp & exit 1
fi

#Get the program parameters into the array "params":
params_count=0
for i; do
    params_count=$((params_count+1))
    eval params_$params_count=\"\$i\"
done
params_0=$((params_count))

if [ "$params_0" = "0" ]; then #if no parameters are provided: display help
    DisplayHelp
    CleanUp && exit 0
fi


#Create a flags array. A flag denotes special parameters:
different_flag="0"
common_flag="0"
domains_flag="0"
domains_full_flag="0"
preserve_order_flag="0"
help_flag="0"
subshell_flag="0"
subshell1_flag="0"
subshell2_flag="0"
subshell3_flag="0"
subshell4_flag="0"
i=1;
j=0;
while [ "$i" -le "$((params_0))" ]; do
    eval params_i=\"\$\{params_$i\}\"
    case "${params_i}" in
        "--different" | "-d" )
            different_flag="1"
        ;;
        "--common" | "-c" )
            common_flag="1"
        ;;
        "--domains" )
            domains_flag="1"
        ;;
        "--domains-full" )
            domains_full_flag="1"
        ;;
        "--preserve_order" | "-p" )
            preserve_order_flag="1"
        ;;
        "--help" | "-h" )
            help_flag="1"
        ;;
        "--subshell1" )
            subshell1_flag="1"
            subshell_flag="1"
        ;;
        "--subshell2" )
            subshell2_flag="1"
            subshell_flag="1"
        ;;
        "--subshell3" )
            subshell3_flag="1"
            subshell_flag="1"
        ;;
        "--subshell4" )
            subshell4_flag="1"
            subshell_flag="1"
        ;;
        * )
            j=$((j+1))
            eval selected_params_$j=\"\$params_i\"
        ;;
    esac
    
    i=$((i+1))
done

error="false"
if [ "$different_flag" = "0" -a "$common_flag" = "0" ]; then
    printf '%s\n' "ERROR: Expected either -c or -d flag!">/dev/stderr
    error="true"
elif [ "$common_flag" = "1" -a "$different_flag" = "1" ]; then
    printf '%s\n' "ERROR: Invalid combination of flags ( -d and -c )!"
    error="true"
# If the --domains-full flag is set ("1"): the --preserve-order flag is set by default ("1") for both -c and -d flags;
# Otherwise, if the --domains-full flag is not set ("0"):
elif [ ! "$domains_full_flag" = "1" -a "$common_flag" = "1" -a "$preserve_order_flag" = "1" ]; then
    printf '%s\n' "ERROR: The -p flag cannot be used together with the -c flag)!"
    error="true"
fi
if [ "$domains_flag" = "1" -a "$domains_full_flag" = "1" ]; then
    printf '%s\n' "ERROR: Invalid combination of flags ( --domains and --domains-full )!"
    error="true"
fi
if [ "$error" = "true" ]; then
    CleanUp & exit 1
fi

# When "$domains_full_flag" is "1":
#
# Currently: process substitution cannot be waited in bash, 
# and as a result multiple processes from process substitution
# would display at the same time in title:
# i.e. command1 + command2 or command1 + command3
#
# so - as a workarround: 
# we display in the title the most time consuming of the processes
# from process substitution:
# i.e. command1 and command3

# WHEN PROCESS SUBSTITUTION RUNS:

# IN SUBSHELLS 1 and 3 (command 1 and 3):
if [ "$subshell1_flag" = "1" -o "$subshell3_flag" = "1" ]; then
    where_to_print="/dev/tty"
# IN SUBSHELLS 2 and 4 (commands 2 and 4):
elif [ "$subshell2_flag" = "1" -o "$subshell4_flag" = "1" ]; then
    where_to_print="/dev/null"
# OTHERWISE: NO SUBSHELL CONFLICT:
else
    where_to_print="/dev/tty"
fi

# IN SUBSHELLS 1 and 2 (commands 1 and 2):
if [ "$subshell1_flag" = "1" -o "$subshell2_flag" = "1" ]; then
    index_amplifier="1"
# IN SUBSHELLS 3 and 4 (commands 3 and 4):
elif [ "$subshell3_flag" = "1" -o "$subshell4_flag" = "1" ]; then
    index_amplifier="2"
# OTHERWISE: NO SUBSHELL CONFLICT:
else
    index_amplifier="1"
fi


selected_params_0=$j

#Rebuild params array:
for i in $(seq 1 $selected_params_0); do
    eval params_$i=\"\$\{selected_params_$i\}\"
done
params_0=$selected_params_0

if [ "$help_flag" = "1" ]; then
    DisplayHelp
else #Run program:
    
    NL=$(printf '%s' "\n\n"); #final NewLine is deleted
    #or use:
    #NL=$'\n'
    
    error1="false"
    error2="false"
    error3="false"
    { sed --help >/dev/null 2>/dev/null; } || { error1="true"; }
    { sort --help >/dev/null 2>/dev/null; } || { error2="true"; }
    { uniq --help >/dev/null 2>/dev/null; } || { error3="true"; }
    { diff --help >/dev/null 2>/dev/null; } || { error4="true"; }
    { grep --help >/dev/null 2>/dev/null; } || { error5="true"; }
    if [ "$error1" = "true" -o "$error2" = "true" -o "$error3" = "true" -o "$error4" = "true" -o "$error5" = "true" ]; then
        {
            printf "\n"
            if [ "$error1" = "true" ]; then printf '%s\n' "ERROR: Could not run \"sed\" (necessary in order for this script to function correctly)!"; fi
            if [ "$error2" = "true" ]; then printf '%s\n' "ERROR: Could not run \"sort\" (necessary in order for this script to function correctly)"; fi
            if [ "$error3" = "true" ]; then printf '%s\n' "ERROR: Could not run \"uniq\" (necessary in order for this script to function correctly)"; fi
            if [ "$error4" = "true" ]; then printf '%s\n' "ERROR: Could not run \"diff\" (necessary in order for this script to function correctly)"; fi
            if [ "$error5" = "true" ]; then printf '%s\n' "ERROR: Could not run \"grep\" (necessary in order for this script to function correctly)"; fi
            printf "\n"
        }>/dev/stderr
        CleanUp & exit 1
    fi

    #Copy the parameters to the array "files":
    for i in $(seq 1 $selected_params_0); do
        eval files_$i=\"\$selected_params_$i\"
    done
    files_0=$selected_params_0
    
    error="false"
    if [ "$files_0" -lt "2" ]; then
        printf '\n%s\n' "ERROR: Please provide at least two parameters!">/dev/stderr
        error="true"
    fi
    
    if [ "$error" = "true" ]; then
        printf "\n"
        CleanUp & exit 1
    fi
    
    if [ "$OS" = "Linux" -o "$OS" = "Mac" -o "$OS" = "userdefined" ]; then
        
        #Commands to display common/different domains when --domains-full flag is not set:
        if [ "$domains_full_flag" = "0" ]; then
            # Command to add a NewLine character at the beginning of each URL found (an URL is identified by: "...://..."):
            #   command1: sed -E 's/([a-zA-Z]*\:\/\/)/\\${NL}\1/g'
            sed_command1='sed -E '"'"'s/([a-zA-Z]*\:\/\/)/'"\\${NL}"'\1/g'"'";
            
            # Command to strip anything beside the URL body ("...://...") = for each new line, remove anything after (including) the first SPACE/TAB/NEWLINE found:
            if [ "$domains_flag" = "0" ]; then
                # Do not strip the URL domain (keep the full URL):
                #   command2: sed -n 's/\(\(.*\([^a-zA-Z+]\)\|\([a-zA-Z]\)\)\)\(\([a-zA-Z]\)*\:\/\/\)\([^ \|^\t\|^\n]*\).*/\4\5\7/p'
                sed_command2='sed -n '"'"'s/\(\(.*\([^a-zA-Z+]\)\|\([a-zA-Z]\)\)\)\(\([a-zA-Z]\)*\:\/\/\)\([^ \|^\t\|^\n]*\).*/\4\5\7/p'"'"
            elif [ "$domains_flag" = "1" ]; then
                # Filter only URLS that contain "."
                # Strip anything beside the URL domain (the string until the first encountered SLASH/COLON/NEWLINE beside "://"):
                #   command2: sed -n 's/\(\(.*\([^a-zA-Z+]\)\|\([a-zA-Z]\)\)\)\(\([a-zA-Z]\)*\:\/\/\)\([^/\|^\:\|^\n]*\).*/\4\5\7/p'|sed '/\./!D'
                sed_command2='sed -n '"'"'s/\(\(.*\([^a-zA-Z+]\)\|\([a-zA-Z]\)\)\)\(\([a-zA-Z]\)*\:\/\/\)\([^/\|^\:\|^\n]*\).*/\4\5\7/p'"'"\|'sed '"'"'/\./!D'"'"
            fi
            
            #Command to remove URLs that contain "file:///" (URLs that contain the file protocol (file:///...) are automatically skipped):
            sed_command2="$sed_command2"\|'sed '"'"'/file\:\/\/\/./D'"'"
            
            # For each line: Command to remove the second column (containing a padded number which was used to preserve initial find order (after sorting)):
            if [ "$different_flag" = "1" ]; then
                #   command3: sed -E 's/(.) [0-9]* (.*)/\1 \2/g'
                sed_command3='sed -E '"'"'s/(.) [0-9]* (.*)/\1 \2/g'"'";
                #   command4: sed -E 's/^2/>/g;s/^1/</g'
                sed_command4='sed -E '"'"'s/^2/>/g;s/^1/</g'"'"
            elif [ "$common_flag" = "1" ]; then
                #   command3: sed -E 's/(.) [0-9]* (.*)/\2/g'
                sed_command3='sed -E '"'"'s/(.) [0-9]* (.*)/\2/g'"'";
            fi
        #Commands to display common/different domains when --domains-full flag is set:
        elif [ "$domains_full_flag" = "1" ]; then
            # Command to find common domains:
            #     command1: 
            uniql_command1="$current_shell $current_script_path -c --domains $(for i in seq 1 $files_0; do eval printf \'%s \' \'\$files_$i\'; done)"

            # Links that are only in first parameter file (file group 1):
            #     command2: 
            uniql_command2="$current_shell $current_script_path -d '$files_1' \"/dev/null\""
            
            #     command3: 
            uniql_command3="$current_shell $current_script_path -c --domains $(for i in seq 1 $files_0; do eval printf \'%s \' \'\$files_$i\'; done)"
            # Links that are only in 2..N parameter files (file group 2):
            #     command4: 
            uniql_command4="$current_shell $current_script_path -d \"/dev/null\" $(for i in seq 2 $files_0; do eval printf \'%s \' \'\$files_$i\'; done)"
        fi
                    
    else
        printf '\n%s\n\n' "Error: Unsupported OS!">/dev/stderr
        CleanUp & exit 1
    fi
    
    error="false"
    for i in $(seq 1 $files_0); do
        eval current_file=\"\$files_$i\"
        # If current file does not exist or is a directory:
        if [ ! -e "$current_file" -o -d "$current_file" ]; then
            printf '\n%s\n' "ERROR: File \"$current_file\" does not exist or is not a accessible!">/dev/stderr
            error="true"
        else
            # If current file is not readable:
            if [ ! -r "$current_file" ]; then
                printf '\n%s\n' "ERROR: File <file$i> = \"$current_file\" is not accessible!">/dev/stderr
                error="true"
            fi
        fi
    done
    
    if [ "$error" = "true" ]; then
        printf "\n"
        CleanUp & exit 1
    fi
    
    #Proceed to finding and comparing links:
    
    #Trap "INTERRUPT" (CTRL-C) and "TERMINAL STOP" (CTRL-Z) signals:
    trap 'trap1' INT
    trap 'trap1' TSTP
    
    old_IFS="$IFS" #Store initial IFS value
    IFS="
"
    
    if [ "$domains_full_flag" = "0" ]; then
        
        if [ "$different_flag" = "1" -a "$preserve_order_flag" = "0" ]; then
        {
            title_index="$(( 1 + ( $subshell_flag * ( $index_amplifier - 1 ) ) * 2 ))"
            PrintJustInTitle "Searching for links [$title_index]..."
            {
                ProcFindLinks "files_1"
                
                PrintJustInTitle "Sorting results [$title_index]..."
            }|sort -u -k 3
            
            title_index="$(( 2 + ( $subshell_flag * ( $index_amplifier - 1 ) ) * 2 ))"
            PrintJustInTitle "Searching for links [$title_index]..."
            {
                for i in $(seq 2 $files_0); do
                    ProcFindLinks "files_$i"
                done
                
                PrintJustInTitle "Sorting results [$title_index]..."
            }|sort -u -k 3
            
            PrintJustInTitle "Searching for unique links [$((title_index + 1))]..."
        }|{\
            sort -k 3|uniq -u -f 2|sort|eval "$sed_command3"|eval "$sed_command4"
            
            PrintJustInTitle "Done"
        }
        elif [ "$common_flag" = "1" -a "$preserve_order_flag" = "0" ]; then
        {
            title_index="$(( 1 + ( $subshell_flag * ( $index_amplifier - 1 ) ) * 2 ))"
            PrintJustInTitle "Searching for links [$title_index]..."
            {
                ProcFindLinks "files_1"
                
                PrintJustInTitle "Sorting results [$title_index]..."
            }|sort -k 3|uniq -f 2
            
            title_index="$(( 2 + ( $subshell_flag * ( $index_amplifier - 1 ) ) * 2 ))"
            PrintJustInTitle "Searching for links [$title_index]..."
            {
                for i in $(seq 2 $files_0); do
                    ProcFindLinks "files_$i"
                done
                
                PrintJustInTitle "Sorting results [$title_index]..."
            }|sort -k 3|uniq -f 2
            
            PrintJustInTitle "Searching for common links [$((title_index + 1))]..."
        }|{\
            sort -k 3|uniq -d -f 2|sort|eval "$sed_command3"
            
            PrintJustInTitle "Done"
        }
        elif [ "$different_flag" = "1" -a "$preserve_order_flag" = "1" ]; then
            PrintJustInTitle "Searching for links..."
            {
                link_count=0
                current_line=""
                for line in $(\
                        diff \
                        <( \
                            PrintJustInTitle "Searching for links [1]..."; \
                            ProcFindLinksShowLineNumbers "files_1"; \
                            printf '%s\n' "### Sepparator 1";\
                        ) \
                        <( \
                            PrintJustInTitle "Searching for links [2]..."; \
                            for i in $(seq 2 $files_0); do ProcFindLinksShowLineNumbers "files_$i"; done; \
                            printf '%s\n' "### Sepparator 2";\
                        ) \
                    ); do
                    link_count+=1
                    previous_line="$current_line"
                    current_line="$line"
                    #if ( current line starts with "<" and previous line starts with "<" ) OR ( current line starts with ">" and previous line starts with ">" ):
                    if [ \( \( ! "${current_line#"<"}" = "${current_line}" \) -a \( ! "${previous_line#"<"}" = "${previous_line}" \) \) -o \( \( ! "${current_line#">"}" = "${current_line}" \) -a  \( ! "${previous_line#">"}" = "${previous_line}" \) \) ]; then
                        printf '%s\n' "$previous_line"
                    fi
                done
            }

        fi
    elif [ "$domains_full_flag" = "1" ]; then
        
        if [ "$different_flag" = "1" ]; then
        # Find links (second eval) that are not in the common domains list (first eval):
            # Links in the first file given as parameter (second eval):
            grep -F -vf <( eval $uniql_command1 --subshell1; ) <( eval $uniql_command2 --subshell2; )
            # Links in the files 2..N - given as parameters (second eval):
            grep -F -vf <( eval $uniql_command3 --subshell3; ) <( eval $uniql_command4 --subshell4; )
        elif [ "$common_flag" = "1" ]; then
        # Find links (second eval) that are in the common domains list (first eval):
            # Links in the first file given as parameter (second eval):
            grep -F -f <( eval $uniql_command1 --subshell1; ) <( eval $uniql_command2 --subshell2; )
            # Links in the files 2..N - given as parameters (second eval):
            grep -F -f <( eval $uniql_command3 --subshell3; ) <( eval $uniql_command4 --subshell4; )
        fi
        # grep flags explained:
        #    -F = do not interpret pattern string (treat string literally)
        #    -v = select non-matching lines
        #    -f = obtain pattern strings from next file - each string on a new line (containing site domains in this case)
    fi
    
    CleanUp
fi

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .