Tag Info

Hot answers tagged

21

You can also try hxselect (from HTML-XML-Utils) with wget as follows: wget -qO- 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' | hxselect -s '\n' -c 'title' 2>/dev/null You can install hxselect in Debian based distros using: sudo apt-get install html-xml-utils. STDERR redirection is to avoid the Input is not well-formed. (Maybe try normalize?) ...


18

Such a thing already exists for Ubuntu: https://apps.ubuntu.com/ You can browse by category and search for packages using the web interface. Each application also displays its rating and any reviews it has received - just like the Software Center in Ubuntu.


17

If you have bash 2.04 or above with the /dev/tcp pseudo-device enabled, you can download a file from bash itself. Paste the following code directly into a bash shell (you don't need to save the code into a file for executing): function __wget() { : ${DEBUG:=0} local URL=$1 local tag="Connection: close" local mark=0 if [ -z "${URL}" ]; ...


11

From the Surfraw website: Surfraw provides a fast unix command line interface to a variety of popular WWW search engines and other artifacts of power. It reclaims google, altavista, babelfish, dejanews, freshmeat, research index, slashdot and many others from the false-prophet, pox-infested heathen lands of html-forms, placing these wonders where they ...


11

If you're looking for an online environment to practice your pattern matching using regex, there are a number of nice resources that allow you to play around with expressions for a given portion of text. Off the top of my head, there is: regex pal or rubular. In terms of exercises, one idea is to search for questions tagged with [regex] on stack overflow. ...


10

For simple cases of downloading the contents of a page, use curl or wget. Both are command line tools designed to download files over HTTP and have many options. In your case, you'll likely need to make these tools look more like a browser; lutzky's answer and penguin359's answer mention some curl and wget options that are useful in that respect. Sometimes, ...


9

Yes, it's fully possible with curl. Most importantly will be to save and reload cookies between uses of curl with --cookie-jar. You can also post form data as needed. I usually use a Firefox add-on called Live HTTP Headers to capture what happens when I navigate a website. It will record any headers, but also any form posts which is very helpful when ...


9

Rendering HTML is a function of the browser, not the operating system. Don't let Microsoft's patently ridiculous marketing of "native support" delude you into thinking otherwise. Install a modern browser. Live happily ever after.


9

You can also use curl and grep to do this. You'll need to enlist the use of PCRE (Perl Compatible Regular Expressions) in grep to get the look behind and look ahead facilities so that we can find the <title>...</title> tags. Example $ curl 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' -so - | \ grep -iPo ...


8

The difference is in how DNS and the HTTP "Host" header work. The site you're going to may have multiple sites hosted using the same server. In DNS, all the names for the sites hosted on that IP list the single server IP address. When you enter the name in the browser, the browser sends the hostname to the server using the "Host" header. If you enter only ...


7

No, it will not help. All it does is set up an encrypted connection to your local machine, and from there, connects to the outside world exactly like it would have without the local ssh - you gain nothing, but performance is going to suffer (after all, ssh does encrypt and decrypt all the X11 messages that firefox and the X11 server pass back and forth). ...


6

Use lynx. It is pretty common for most of Unix/Linux. lynx -dump http://www.google.com -dump: dump the first file to stdout and exit man lynx Or netcat: /usr/bin/printf 'GET / \n' | nc www.google.com 80 Or telnet: (echo 'GET /'; echo ""; sleep 1; ) | telnet www.google.com 80


5

Note: In your case, the best would be to just drop root privileges for updates and run your scripts with your apache user: su apache -c "./update-script" Otherwise, use chmod g+s /var/www. New files and sub-directories created inside this directory will share the same owner/group as the parent directory, by default. (This spreads recursively.) ...


5

You may use tcpdump. # tcpdump filter for HTTP GET sudo tcpdump -s 0 -A 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420' # tcpdump filter for HTTP POST sudo tcpdump -s 0 -A 'tcp dst port 80 and (tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504f5354)' For a solution using tshark see: ...


5

I would suggest that uzbl is just the right ninja magic for this. It is a scriptable, console-controllable single-purpose browser. Being based on webkit, its rendering and javascript support is first class, but it follows the unix phylosophy of doing one thing and doing it well while allowing other programs to push data in and out. There is a wrapper for it ...


5

My favorite is wget, so I'll give an example with that. What you want to do is replicate your browser session as closely as possible, so use the relevant commandline arguments. The ones which are necessary depend on how thoroughly the site checks your browser. --referer (sic) is usually sufficient, but you might also need --user-agent and --load-cookies. ...


5

LinkChecker. LinkChecker is a free, GPL licensed URL validator. Features recursive and multithreaded checking output in colored or normal text, HTML, SQL, CSV, XML or a sitemap graph in different formats HTTP/1.1, HTTPS, FTP, mailto:, news:, nntp:, Telnet and local file links support restriction of link checking with regular ...


5

If it doesn't need to be web based, there are various programs that do this (arranged in order of purdiness): The classic: synaptic, you just need to click on the "Sections" button on the bottom left to browse by section. Should work on all Debian based distros and as far as I know is installed by default on all of them as well. You can get Ubuntu debs ...


4

It does not rely on the desktop environment. What it does rely on (at least in case of graphical-mode browsers like Chrome, Opera or Firefox) is the Xorg server, together with its libraries1. Once you have the core dependencies of a browser installed (including the X server), you can run it under a minimal environment either by: using a very minimalistic ...


4

Gwibber has the ability to send to multiple services at once. According to their website it supports the following protocols/services: Twitter Identi.ca/StatusNet Ping.fm Facebook FriendFeed Buzz Digg Flickr Qaiku As far as I know, it has the ability to receive content from all of the listed services, but I'm not sure if there is a way to receive ...


4

I don't know if there's online resources, but local tests is fine, e.g with kiki This will help you learn about regex. The package is available in Ubuntu / Arch repository, to install it, do: apt-get install -y kiki Or pacman -S kiki-re on Arch.


4

Linux has the same host file. Just edit: /etc/hosts Depending on your system a command like this will open up that file: $ sudo nano /etc/hosts Just put lines like this in there and save: <IP> <HOSTNAME> <SHORTNAME #1> <SHORTNAME #2> .... For example: 192.168.1.1 www.anothersite.com wwwanother


4

That seems to be an SVN repository. So you can just check it out: svn co https://svn.linuxsampler.org/svn/liblscp/trunk/ You need to have subversion installed (should be available via package manager on most systems)


4

Using simple regex to parse HTML is naive. E.g. with newlines and ignoring special character encoding specified in the file. Do the right thing and really parse the page using any of the other real parsers mentioned in the other answers or use the following one liner: python -c "import bs4, urllib2; print ...


3

A lot of excellent web UIs are mentioned in other answers, however if you are managing VMs, I would recommend using OpenStack. Just for UI, it'll be overkill but trust me requirements are always increasing. OpenStack will provide you an API upon which you can build an interface easily both for customers and the administrators and modify to suit specific ...


3

I would suggest xen-admin , it has the capabilities to give your users the ability to mange their VM. And maybe Proxmox is good too, but i don't have any experience with it.


3

I think you are looking for a caching web proxy like polipo or squid. Although it's called a proxy-server you don't need a separate system to run it. It just means that it is a special software that can serve a client a particular service (In this case caching already downloaded data). A client is also a software that can run on the same system as the server ...


3

Yes, of course. You need X11 installed but that is all. You could use something like fvwm, or awesome or xnomad to manage the X11 session. Configuration of those window managers can be done via their own rc files and started via .xinitrc/.xsession. Then from a shell you can start either Chrome or Firefox. If you do not want to install X11, then you can ...


3

"Stealing cookies from the web browser" in order to obtain authentication sounds like exactly the kind of thing malware would get up to in order to gain access to your personal information and login data. If this approach turns out to work, I recommend you inform each browser's development team about a security hole in their software.


3

You can try links or lynx. Both should be available through apt-get. These are text only browsers, and usually get the job done relatively well when all you have is a console. I don't think any console browser has PDF support though, since that format is usually highly graphical in nature.



Only top voted, non community-wiki answers of a minimum length are eligible