3

Is this possible? I have an HTML file containing a lot of URLs and relative paths. I want to collect all of them. I know that HTML can't be properly parsed with regex, but I'm looking for a solution that doesn't care about what kind of data it's looking at. It should just pull out every occurrence of /[A-Za-z0-9_-]* including paths that are multiple directories deep.

Given /this/is/a/test I should get:

/this
/is
/a
/test
/this/is
/this/is/a
/this/is/a/test
/is/a
/a/test
/is/a/test

I think that's all of them. If I could code a solution for this, I'd be sure!

3
$ sed -n -f script.sed file
/this/is/a/test
/this/is/a
/this/is
/this
/is/a/test
/is/a
/is
/a/test
/a
/test

Where script.sed is

:outer
h
:inner
/./ {
    p
    s:/[^/]\{1,\}$::
    t inner
}
g
s:^/[^/]\{1,\}::
t outer

and where file is a file containing the example pathname that you mentioned.

This script implements a double loop that processes each line of input. It expects to get lines with pathnames only.

The outer loop (labelled outer) copies the current line to the hold space (h). The inner loop (labelled inner), which is completely skipped if the line is empty, then prints the current line if it's not empty. Then the last /<anything> is removed from the current line, and, if that substitution modified the data, t inner loops back to the start of the inner loop.

Otherwise, the saved line is fetched from the hold space (g) and the first /<anything> is removed from it. If that substitution modified the data, the code loops back to the start of the outer loop.

The net result is that you get your complete pathname printed, then you get each pathname that is generated by chopping off successive path components from the end of it.

Then the same thing is repeated with the pathname that you get by removing the first path element from the original pathname. Etc.

If you swap the two substitutions in the code, you get the same thing, but in another order:

/this/is/a/test
/is/a/test
/a/test
/test
/this/is/a
/is/a
/a
/this/is
/is
/this
0

The body of the question seems to ask two different things. This addresses the first, that being extracting anything that looks like a relative URL path that conforms to the stated regex, from an input file.

Using grep:

grep -Eio "(/[a-z0-9_-]+)+" file.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.