consider the links

https://unix.stackexchange.com/questions/tagged/linux?page=2&sort=newest&pagesize=15 https://unix.stackexchange.com/questions/tagged/linux?page=3&sort=newest&pagesize=15 https://unix.stackexchange.com/questions/tagged/linux?page=4&sort=newest&pagesize=15

here 'page' is incrementing, if there are 115 pages then the value of page will be 115 for the last page

How to know which part of the link is incrementing by processing 2 sample links?

I need to generate all 115 links using shell script.

The inputs will be 2 links of page 2 and page 3 and total number of pages.

I am on bash shell, python is also possible


In Python you can use SequenceMatcher from difflib:

#!/usr/bin/env python
import difflib

url1 = "http://unix.stackexchange.com/questions/tagged/linux?page=2&sort=newest&pagesize=15"
url2 = "http://unix.stackexchange.com/questions/tagged/linux?page=3&sort=newest&pagesize=15"

matcher = difflib.SequenceMatcher(a=url1, b=url2)
matches = matcher.get_matching_blocks()

prefix = url1[:matches[0][2]]
suffix = url2[matches[1][1]:]

for i in range(2, 116):
    print prefix + str(i) + suffix

SequenceMatcher.get_matching_blocks() will return a list of triples of the form (i, j, n), where a[i:i+n] == b[j:j+n]. Using the first two such triples, we construct the prefix and suffix of the URL surrounding the page number and iterate the range of URLs.

| improve this answer | |

Here's a Perl script just to show how it's possible using Perl's bitwise exclusive or (XOR) operator (^).

The script

I called it cmp.pl.

#!/usr/bin/perl -w

use strict;
use warnings;
# $s1 = "http://unix.stackexchange.com/questions/tagged/linux?page=2&sort=newest&pagesize=15";
# $s2 = "http://unix.stackexchange.com/questions/tagged/linux?page=3&sort=newest&pagesize=15";
# $np = 115

my $s1 = $ARGV[0];
my $s2 = $ARGV[1];
my $np = $ARGV[2];

my $posOfDiff;

my $mask = $s1 ^ $s2;
while ($mask =~ /[^\0]/g) {
  $posOfDiff = $-[0];

for (my $idx = 1; $idx <= $np; $idx++) {
  my $newStr = $s1;
  substr($newStr,$posOfDiff,1) = $idx;
  print "$newStr\n";


The unique feature of this script is the use of Perl's (^) operator. The power of this approach is in this snippet of code:

my $mask = $s1 ^ $s2;
while ($mask =~ /[^\0]/g) {
  $posOfDiff = $-[0];

The above will create a mask ($mask) using the 2 strings. An XOR mask is a vector that will contain a 0 for values that match between $s1 and $s2, and a 1 where they differ. You can add this line of code if you want to convince yourself of this:

my $mask = $s1 ^ $s2;
printf "[$_] is 0x%02x\n", ord($_) for split //, $mask;

Innards of mask

This printf will produce output like this. NOTE: that the characters are unprintable, they're hex values. The 0x00 is the hex value for the null character, 0x01 is a 1.

[] is 0x00
[] is 0x00
[] is 0x00
[] is 0x00
[] is 0x00
[ ] is 0x01
[] is 0x00
[] is 0x00

The value being returned when anything other than 0 means the values are different. Other examples:

$ perl -we '$a="ab"; $b="ac"; $c=$a ^ $b; printf "[$_] is 0x%02x\n", ord($_) for split //, $c;'
[] is 0x00
[] is 0x01

$ perl -we '$a="ab"; $b="ad"; $c=$a ^ $b; printf "[$_] is 0x%02x\n", ord($_) for split //, $c;'
[] is 0x00
[] is 0x06

$ perl -we '$a="ab"; $b="ae"; $c=$a ^ $b; printf "[$_] is 0x%02x\n", ord($_) for split //, $c;'
[] is 0x00
[] is 0x07

Looping through the mask

The other interesting feature of the while loop is that it loops through only the characters of $mask that are not nulls (\0). So in your example, we're actually only executing the while loop 1 time, since there is only 1 difference between the 2 strings. If there were 2 differences it would execute 2 times. So this is a fairly efficient way to do this.

If you need more convincing here's some additional lines of code you can add that show the while loop in action:

while ($mask =~ /[^\0]/g) {
  print "in the loop\n";
  print "what we're looking for:" . $-[0] . "\n";

These lines are only displayed one time:

in the loop
what we're looking for:58

Saving the position of the difference

Once a match is found, the body of the while loop will execute and the position will get recorded in the variable $posOfDiff. How? The beauty here is the use of the variable, $-[0]. This will give us the offset of the position of the last successful match.

$-[0] is the offset of the start of the last successful match.

This match is what is occurring in the control portion of the while loop, we're looking for characters in $mask that are NOT the null character (\0), hence our difference character:

$mask =~ /[^\0]/g

NOTE: The trailing g tells the match function in Perl to do it globally, so it will keep finding matches until it exhausts the string, $mask.

What else?

The rest of this script is pretty much boilerplate Perl, not worth discussing further.


| improve this answer | |
  • 1
    Actually, its 0 where they match, and non-zero where they don't match—not always 1. Its just 1 for you because you compared adjacent characters, and they happened to be part of an even-odd pair. That said, at a quick glance, your script tests for non-0, so its just a problem with your description. – derobert Sep 24 '13 at 15:13
  • e.g., perl -E 'print "ab"^"ad"' | xxd results in 0000000: 0006 .. – derobert Sep 24 '13 at 15:14

Without knowing anything else about the links and assuming only the page number changes, here's my approach. First I thought about diff -e, but it turns out it just creates a whole line ed substitution script, which is useless as a starting point.

So here's a pure bash solution:


function splitOnDigit() {
  url1="${1#$prefix}" # remainder if you strip the prefix
  suffix1="&${1#*&}" # remainder after the number
  num="${url1%$suffix1}" # the number that we just split around
  if [[ $suffix1 == $1 ]]; then
    # if substitutions failed, we're at the end
    echo "$prefix$3"

  if [[ $suffix1 == $suffix2 ]]; then
    echo "$prefix$3$suffix1"
    echo -n "$prefix$num"
    splitOnDigit "$suffix1" "$suffix2" "$3"

splitOnDigit "$url1" "$url2" "$pagenum"

It compares the two strings in chunks by splitting one on the numbers. Just save it somewhere and run it — it is self-contained. The recursive part is there just in case the changing numeric part isn't the first one found like in your example.

To generate a whole range of page numbers put the whole thing in a loop (takes three arguments: url1 url2 pagenumber, just like the function it implements). It's fast enought to work as-is, but the string munging could be done only once and saved as a prefix-suffix pair inbetween which you'd put your iterating numbers.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.