0

I need to display the complete address with curl, when it find results with '301' status code.

This is my variable.

search=$(curl -s --head -w %{http_code} https://launchpad.net/~[a-z]/+archive/pipelight -o /dev/null | sed 's#404##g') 

echo $search
301

The above works, but only display if the site exists with '301' status code.

I want

echo $search
https://launchpad.net/~mqchael/+archive/pipelight

UPDATE

This is my new variable, maybe can explain what I need. This variable will allow me to search and install a ppa in Ubuntu o similar.

ppa=$(curl https://launchpad.net/ubuntu/+ppas?name_filter=$packagename | grep '<td><a href="/~' | grep ">$packagename<" )

echo $ppa

Example:

ppa=$(curl https://launchpad.net/ubuntu/+ppas?name_filter=Pipelight | grep '<td><a href="/~' | grep ">Pipelight<" )

echo $ppa 

<td><a href="/~mqchael/+archive/pipelight">Pipelight</a></td>

The problem here is I can't extract mqchael (this name is variable), also pipelight is only a example.

This is the format final when I will apply my variable.

ppa:mqchael/pipelight
  • Does the answer specified work? If you need a more specific answer, please let me know :) – Ramesh Mar 21 '14 at 22:47
  • Thanks @Ramesh the idea is display the search found, https://launchpad.net/~mqchael/+archive/pipelight, because I will making a patron in ppa with your complete name. – davidva Mar 22 '14 at 0:56
  • Cant you just var=$(curl -operands $url_is_last_arg >&2 ; echo ${_##*namefilter=}) ? – mikeserv Mar 23 '14 at 8:29
2

This should do what you want:

curl https://launchpad.net/ubuntu/+ppas?name_filter=Pipelight |  awk -F/ '/>Pipelight</{print $2}'

Explanation:

The -F/ sets the filed delimiter to /, and the />Pipelight</ means "run the commands in the {} only on lines matching >Pipelight<. So, at least in the example you posted, the line with >Pipelight< is:

<td><a href="/~mqchael/+archive/pipelight">Pipelight</a></td>

So, since awk is splitting on /, the first field will be <td><a href=" and the second will be ~mqchael. Which is why {print $2} will print ~mqchael.

If you also want to get rid of the tilde (~), use this:

curl https://launchpad.net/ubuntu/+ppas?name_filter=Pipelight |  
    awk -F/ '/>Pipelight</{print $2}' | sed 's/~//'
  • this is my new variable, maybe can explain what I need. ppa=$(curl https://launchpad.net/ubuntu/+ppas?name_filter=Pipelight | grep '<td><a href="/~' | grep ">Pipelight<" ); echo "ppa=$ppa" > /tmp/lauchpad.txt The problem here is I can't extract ~mqchael is variable this name, pipelight is only a example. – davidva Mar 23 '14 at 0:38
  • @davidva see updated answer. – terdon Mar 23 '14 at 2:25
  • only a problem, the line awk -F/ '/>Pipelight</{print $2}' doesn't read my variable, example: awk -F/ '/>$my_program</{print $2}' – davidva Mar 23 '14 at 8:36
0

I think you are trying to find out the sites which have 301 status code returned. You can actually write the contents of the curl command to a file and do a grep on Location to find out the URLs for the status code 301. Try this one.

curl -s --head -w %{http_code} https://launchpad.net/~[a-z]/+archive/pipelight -o
grep 'Location' file1.txt

The output would be,

Location: https://launchpad.net/~j/+archive/ppa/pipelight
  • Sorry but doesn't work, the idea is display the search found, https://launchpad.net/~mqchael/+archive/pipelight, because I will making a patron in ppa with your complete name. – davidva Mar 22 '14 at 0:54
0
code=$desired_HTML_return_code
url="https://launchpad.net/ubuntu/+ppas?name_filter=Pipelight"
_curl=$( curl -o /dev/stderr -sL -w \
    "%{http_code} %{url_effective}\\n" "$url" ) 
[ ${_curl%%[!0-9]*} -eq  $code ] && {\
    ppa="${_curl##*~}"
    ppa="ppa:${ppa%%/*}/${_curl##*namefilter=}" 
}

So I haven't tested the above, but apparently this guy has, and he also explains pretty well what the various curl operands are doing, not to mention many others you have at your disposal. The point is, though, that you use curl itself to guarantee its output rather than relying on parsers to any great degree.

As written curl should print to its stdout only the http return code of its query and the URL you feed it - which doesn't have to be a variable but is above for readability and to demonstrate that it can be.

So the next thing we do is ${strip%%*} from the %%tail of that output as far forward as we *can until we encounter the first [character] in the string that !isn't a 0-9number.

Then we [test] the resulting numeric string against our desired http return $code.

&&If they -equal we ${strip##*} from the ##head of our stored $_curl output as far forward as we can up to and including the last ~tilde it contains and assign= the results to $ppa.

Then we assign= $ppa again to:

The string "ppa:" plus:

${ppa's} previous value ${less%%*} the first /forward-slash it contains and everything thereafter plus:

Only what remains of $_curl after ${removing##*} from its ##head everything up to and including the string "namefilter="

This offers some advantages over the other solutions.

As already explained, curl guarantees its standard output only to be the short string "$code $url", but, as written, it also sends the html results to your terminal for debugging on standard error. Its results are not consumed by a parsing program.

Only two applications are involved here: curl and whatever POSIX-compatible shell in which you invoke it.

The results are explicitly tested in the current shell environment and are not consumed on the far side of a subshelled pipe nor are they the result of a regular expression.

It has one disadvantage:

It depends upon the "namefilter=$RESULT" being the tail of your URL string. It is possible to handle using the same mechanics applied here if it is not, but it will likely require at least one more shell command. sed and awk both offer more powerful string searches than simple ${parameter##expansion} globs ever could.

BUT because we tailor curl's output to our purpose in the first place you don't need powerful string searches. So long as your desired http code is returned to curl AND your output should be "namefilter=$desired_string" then I don't see how $ppa can ever equal anything it shouldn't.

If you should accept multiple http return codes your [test] should look like this:

codes="$code1 $code2 $code3"
...
[ "${codes#*"${_curl%%[!0-9]*}"} -ne "$codes" ] && ppa=...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.