1

Shell Script to retrieve text from website

Is there a way to extract a specific part of a website and display it in a terminal? There is a couple of sections on this site that displays a Username and Password for people to access their VPN (VPNBook). However they change the passwords, so I was wondering if a shell script could be made to access the website, extract the username and password fields, and displays it.

Right now the username and password fields are:

Username: vpnbook
Password: he2qv5h

If anyone could point me in the right direction to start this project, that would be great.

|improve this question
  • 2
    curl http://www.example.com/vastly-insecure-VPN-credential-exposure | grep -E '(username|password)'`? – DopeGhoti Aug 24 '17 at 20:23
  • running that command with my link replacing yours, only gives me this output: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 10524 0 10524 0 0 5505 0 --:--:-- 0:00:01 --:--:-- 5504 – iamr00t Aug 24 '17 at 20:34
  • Try @DopeGhoti's command but with -E replaced with -Ei so that the search for username and password is case insensitive. – John1024 Aug 24 '17 at 20:41
  • It worked, but It looks like a mess on a terminal (it is displaying html code as well). Is there any way to make it just display the way they are displayed on the website? – iamr00t Aug 24 '17 at 20:43
4

This worked for me:

#!/usr/bin/env bash

url='https://www.vpnbook.com/freevpn'
html=$( curl -# -L "${url}" 2> '/dev/null' )

username=$(
  <<< "${html}" \
  grep -P -o -e '(?<=<li>Username: <strong>)(.*?)(?=<\/strong><\/li>)' |
  head -n 1
)

password=$(
  <<< "${html}" \
  grep -P -o -e '(?<=<li>Password: <strong>)(.*?)(?=<\/strong><\/li>)' |
  head -n 1
)

printf '%s\n' "Username: ${username}" "Password: ${password}"
|improve this answer
  • This works great, however it shows a progress bar above the output (something like: ####################100%) is there any way to not show that? – iamr00t Aug 24 '17 at 20:46
  • nevermind, i added a 'clear' after the url and html part and it got rid of it. – iamr00t Aug 24 '17 at 20:49
  • 1
    That progress bar is shown because of the curl command. It is send to stderr, not stdout, so it's safe to pipe its output to another command. Anyway, if you don't want to see that progress bar, you could add 2> '/dev/null' to the curl command to send stderr to /dev/null. I've edited my answer to reflect that change. – nxnev Aug 24 '17 at 20:58
  • Thanks a lot @nxnev for the answer and explination. Now I just have to figure out what grep is doing lol. This script is really good though, so thanks again! – iamr00t Aug 24 '17 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.