Tag Info

Hot answers tagged

29

wget -qO- 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' | perl -l -0777 -ne 'print $1 if /<title.*?>\s*(.*?)\s*<\/title/si' You can pipe it to GNU recode if there are things like &lt; in it: wget -qO- 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' | perl -l -0777 -ne 'print $1 if /<title.*?>\s*(.*?)\s*<\/title/si' | recode html.. ...


21

You can also try hxselect (from HTML-XML-Utils) with wget as follows: wget -qO- 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' | hxselect -s '\n' -c 'title' 2>/dev/null You can install hxselect in Debian based distros using: sudo apt-get install html-xml-utils. STDERR redirection is to avoid the Input is not well-formed. (Maybe try normalize?) ...


19

If you have bash 2.04 or above with the /dev/tcp pseudo-device enabled, you can download a file from bash itself. Paste the following code directly into a bash shell (you don't need to save the code into a file for executing): function __wget() { : ${DEBUG:=0} local URL=$1 local tag="Connection: close" local mark=0 if [ -z "${URL}" ]; ...


18

Such a thing already exists for Ubuntu: https://apps.ubuntu.com/ You can browse by category and search for packages using the web interface. Each application also displays its rating and any reviews it has received - just like the Software Center in Ubuntu.


11

From the Surfraw website: Surfraw provides a fast unix command line interface to a variety of popular WWW search engines and other artifacts of power. It reclaims google, altavista, babelfish, dejanews, freshmeat, research index, slashdot and many others from the false-prophet, pox-infested heathen lands of html-forms, placing these wonders where they ...


11

If you're looking for an online environment to practice your pattern matching using regex, there are a number of nice resources that allow you to play around with expressions for a given portion of text. Off the top of my head, there is: regex pal or rubular. In terms of exercises, one idea is to search for questions tagged with [regex] on stack overflow. ...


10

For simple cases of downloading the contents of a page, use curl or wget. Both are command line tools designed to download files over HTTP and have many options. In your case, you'll likely need to make these tools look more like a browser; lutzky's answer and penguin359's answer mention some curl and wget options that are useful in that respect. Sometimes, ...


9

Yes, it's fully possible with curl. Most importantly will be to save and reload cookies between uses of curl with --cookie-jar. You can also post form data as needed. I usually use a Firefox add-on called Live HTTP Headers to capture what happens when I navigate a website. It will record any headers, but also any form posts which is very helpful when ...


9

Rendering HTML is a function of the browser, not the operating system. Don't let Microsoft's patently ridiculous marketing of "native support" delude you into thinking otherwise. Install a modern browser. Live happily ever after.


9

You can also use curl and grep to do this. You'll need to enlist the use of PCRE (Perl Compatible Regular Expressions) in grep to get the look behind and look ahead facilities so that we can find the <title>...</title> tags. Example $ curl 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' -so - | \ grep -iPo ...


8

The difference is in how DNS and the HTTP "Host" header work. The site you're going to may have multiple sites hosted using the same server. In DNS, all the names for the sites hosted on that IP list the single server IP address. When you enter the name in the browser, the browser sends the hostname to the server using the "Host" header. If you enter only ...


7

No, it will not help. All it does is set up an encrypted connection to your local machine, and from there, connects to the outside world exactly like it would have without the local ssh - you gain nothing, but performance is going to suffer (after all, ssh does encrypt and decrypt all the X11 messages that firefox and the X11 server pass back and forth). ...


7

The wget redirection problem can be solved by using wget --trust-server-names http://www.example.com/X?1234


6

Note: In your case, the best would be to just drop root privileges for updates and run your scripts with your apache user: su apache -c "./update-script" Otherwise, use chmod g+s /var/www. New files and sub-directories created inside this directory will share the same owner/group as the parent directory, by default. (This spreads recursively.) ...


6

You may use tcpdump. # tcpdump filter for HTTP GET sudo tcpdump -s 0 -A 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420' # tcpdump filter for HTTP POST sudo tcpdump -s 0 -A 'tcp dst port 80 and (tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504f5354)' For a solution using tshark see: ...


5

I would suggest that uzbl is just the right ninja magic for this. It is a scriptable, console-controllable single-purpose browser. Being based on webkit, its rendering and javascript support is first class, but it follows the unix phylosophy of doing one thing and doing it well while allowing other programs to push data in and out. There is a wrapper for it ...


5

My favorite is wget, so I'll give an example with that. What you want to do is replicate your browser session as closely as possible, so use the relevant commandline arguments. The ones which are necessary depend on how thoroughly the site checks your browser. --referer (sic) is usually sufficient, but you might also need --user-agent and --load-cookies. ...


5

LinkChecker. LinkChecker is a free, GPL licensed URL validator. Features recursive and multithreaded checking output in colored or normal text, HTML, SQL, CSV, XML or a sitemap graph in different formats HTTP/1.1, HTTPS, FTP, mailto:, news:, nntp:, Telnet and local file links support restriction of link checking with regular ...


5

Use lynx. It is pretty common for most of Unix/Linux. lynx -dump http://www.google.com -dump: dump the first file to stdout and exit man lynx Or netcat: /usr/bin/printf 'GET / \n' | nc www.google.com 80 Or telnet: (echo 'GET /'; echo ""; sleep 1; ) | telnet www.google.com 80


5

If it doesn't need to be web based, there are various programs that do this (arranged in order of purdiness): The classic: synaptic, you just need to click on the "Sections" button on the bottom left to browse by section. Should work on all Debian based distros and as far as I know is installed by default on all of them as well. You can get Ubuntu debs ...


4

Gwibber has the ability to send to multiple services at once. According to their website it supports the following protocols/services: Twitter Identi.ca/StatusNet Ping.fm Facebook FriendFeed Buzz Digg Flickr Qaiku As far as I know, it has the ability to receive content from all of the listed services, but I'm not sure if there is a way to receive ...


4

Do you have log files turned on for your server? If you do, I'd recommend installing AwStats and running your log files with it to get accurate reporting. If you want to just monitor all incoming/outgoing traffic, you can use WireShark.


4

It does not rely on the desktop environment. What it does rely on (at least in case of graphical-mode browsers like Chrome, Opera or Firefox) is the Xorg server, together with its libraries1. Once you have the core dependencies of a browser installed (including the X server), you can run it under a minimal environment either by: using a very minimalistic ...


4

myparameter=foobar ssh user@server "myscript $myparameter" This will execute myscript foobar on the server.


4

(This is a follow-up to my comment on the accepted answer.) Note that if $myparameter contains spaces, it will split on the server side. Bash's printf has a %q format that you could use. Example: $ myparameter='hello; rm somefile' $ ssh user@server "echo $myparameter" hello rm: cannot remove `somefile': No such file or directory $ ssh user@server "echo ...


4

I don't know if there's online resources, but local tests is fine, e.g with kiki This will help you learn about regex. The package is available in Ubuntu / Arch repository, to install it, do: apt-get install -y kiki Or pacman -S kiki-re on Arch.


4

Linux has the same host file. Just edit: /etc/hosts Depending on your system a command like this will open up that file: $ sudo nano /etc/hosts Just put lines like this in there and save: <IP> <HOSTNAME> <SHORTNAME #1> <SHORTNAME #2> .... For example: 192.168.1.1 www.anothersite.com wwwanother


4

That seems to be an SVN repository. So you can just check it out: svn co https://svn.linuxsampler.org/svn/liblscp/trunk/ You need to have subversion installed (should be available via package manager on most systems)


4

Using simple regex to parse HTML is naive. E.g. with newlines and ignoring special character encoding specified in the file. Do the right thing and really parse the page using any of the other real parsers mentioned in the other answers or use the following one liner: python -c "import bs4, urllib2; print ...


3

A lot of excellent web UIs are mentioned in other answers, however if you are managing VMs, I would recommend using OpenStack. Just for UI, it'll be overkill but trust me requirements are always increasing. OpenStack will provide you an API upon which you can build an interface easily both for customers and the administrators and modify to suit specific ...



Only top voted, non community-wiki answers of a minimum length are eligible