Tag Info

Hot answers tagged

32

You can use wget over ssh to open an HTTP connection and output stream data, then mplayer on the local machine to play it. ssh <user>@<host> "wget <stream> -O -" | mplayer -cache 512 - The subtleties: wget's -O (capital letter oh) option sets its output file; - (dash) implies stdout. mplayer's -cache option causes a configurable amount ...


21

You can also try hxselect (from HTML-XML-Utils) with wget as follows: wget -qO- 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' | hxselect -s '\n' -c 'title' 2>/dev/null You can install hxselect in Debian based distros using: sudo apt-get install html-xml-utils. STDERR redirection is to avoid the Input is not well-formed. (Maybe try normalize?) ...


18

I was looking for a way to run a program with modified DNS resolution for testing purposes. For me, the solution was using the HOSTALIASES environment variable: $ echo "foo www.google.com" >> ~/.hosts $ HOSTALIASES=~/.hosts $ wget foo See hostname(7).


18

You can use ssh's -L flag to create a tunnel from your system to a remote: ssh -L <localport>:<streamhost>:<streamport> <user>@<remote> You can then connect to your end of the tunnel with mplayer as you would otherwise: mplayer localhost:<localport>


9

Use tcpdump. tcpdump -w httpdebug.pcap -i eth0 port 80 will sniff all packets heading to or from port 80 on the eth0 interface and output them to httpdebug.pcap, which you can then read at your leisure, either with tcpdump again (with multiple -x options, refer to the tcpdump manpage ) in console if you're feeling masochistic, or with wireshark. I really ...


9

You can also use curl and grep to do this. You'll need to enlist the use of PCRE (Perl Compatible Regular Expressions) in grep to get the look behind and look ahead facilities so that we can find the <title>...</title> tags. Example $ curl 'http://www.youtube.com/watch?v=Dd7dQh8u4Hc' -so - | \ grep -iPo ...


8

Most browsers enable you to get a warning if you have HTTP content on an HTTPS page. This can be very annoying if you visit sites that mix HTTP content on their HTTPS pages. From your question it appears Wikipedia is one of those. When properly set, Firefox warns me visiting this page. A web server is not required to offer HTTPS. Many sites do not ...


8

The difference is in how DNS and the HTTP "Host" header work. The site you're going to may have multiple sites hosted using the same server. In DNS, all the names for the sites hosted on that IP list the single server IP address. When you enter the name in the browser, the browser sends the hostname to the server using the "Host" header. If you enter only ...


6

I think the best way to do this would be to set up a SOCKS5 proxy and tell firefox to send the DNS requests over the SOCKS5 proxy (network.proxy.socks_remote_dns). You could set up a socks5 proxy with openssh fairly easily (the -D option) and have a remote host running with a custom /etc/hosts, or something like DNSMasq for more complex DNS settings. ...


6

With avfs: mkdir ~/.avfs avfsd ~/.avfs ln -s ~/.avfs/'#http:server.com|~me|uber_vimrc' ~/.vimrc For writing, you could use webdav. avfs supports a (great) number of network protocols and other methods to access files and adding more is relatively easy. For mounting over sftp, look at sshfs.


5

Use node.js , fast and lightweight. Or just use simple nc netcat command to start a quick webserver on a port and serve the content of a file including the server response headers. Reference from Wikipedia: http://en.wikipedia.org/wiki/Netcat#Setting_up_a_one-shot_webserver_on_port_8080_to_present_the_content_of_a_file { echo -ne "HTTP/1.0 200 ...


5

This may not be the best solution, but if you use any proxy then it will have a specific host:port so the netcat solution with still work, albeit you'll have to pick apart the proxy meta-data to make sense of it. The easiest way to do this might be to use any random anonymization proxy out there and just channel all the traffic through netcat. (I.e., set ...


5

If you really want to use command line for this there is tcpflow. It saves TCP streams to different files. The HTTP request and responses will be stored separately. tcpflow -i wlan0 tcp port 80 If you can use GUI try Wireshark. You can right click any packet and pick "Follow TCP stream".


5

Check out following question at superuser: http://superuser.com/questions/184643/override-dns-in-firefox If the discussed options and the SO link are not viable solutions then check out: http://superuser.com/questions/100239/hostname-override-in-firefox Especially check out: https://addons.mozilla.org/en-US/firefox/addon/redirector/ It sounds like this ...


5

What you want is a HEAD request, but wget does not support it; curl does. Your distribution most probably has curl in repositories. curl -s -I $URL -I | awk '/Location: (.*)/ {print $2}' | tail -n 1 $ URL=http://unix.stackexchange.com/questions/89282/ $ curl -s -I $URL | awk '/Location: (.*)/ {print $2}' | tail -n 1 ...


5

sshfs is a wonderful tool for cases like this. It hides many a firewall and allows you to use whatever player you choose (though mplayer is still a nice choice). sudo apt-get install sshfs # your favorite packager here sshfs me@mydomain.org:/home/arthur/Music ~/Music I also run Subsonic listening to port 4040 on localhost and then ssh-tunnel home with ...


5

you can simply use curl for it.i have written a simple script for it. url=$1 curl -sL $url -w "%{http_code} %{url_effective}\n" "URL" -o /dev/null where URL is your URL which you have to test output will gives you status. Example


4

Your question is quite subjective, and configuring the port should be very easy with any http server. Though, I would recommend lighttpd with server.port = 1080 (or whatever above 1024 for non-root users) in a configuration file: server.document-root = "/home/foo/www" server.port = 1080 It's a complete enough http server, and for serving static ...


4

The headers in an HTTP request must use CRLF (Windows) line endings. (See Wikipedia or RFC 2616.) Many servers support LF (Unix) line endings as an extension, but not this one. In addition, HTTP 1.1 requires a Host: header line, as Warren Young pointed out. (See Wikipedia or RFC 2616). echo -e "GET ...


4

The debian package dwww give access to all the documentation installed by the packages, included the manual pages. After installing the package with your favorite package manager, you will be able to browse the local documentation with your navigator on http://localhost/dwww/. By default, access to this URL is restricted to local connections but you can ...


4

Both Perl and Python (and probably Ruby as well) have simple kits that you can use to quickly build simple HTTP proxies. In Perl, use HTTP::Proxy. Here's the 3-line example from the documentation. Add filters to filter, log or rewrite requests or responses; see the documentation for examples. use HTTP::Proxy; my $proxy = HTTP::Proxy->new( port => ...


4

wget is a great tool. Use wget -m http://somesite.com/directory -m --mirror Turn on options suitable for mirroring. This option turns on recursion and time-stamping, sets infinite recursion depth and keeps FTP directory listings. It is currently equivalent to -r -N -l inf --no-remove-listing.


4

This is commonly referred to as "port forwarding". Here is a snippet from the official documentation: Go to the Firewall menu, select NAT, then click on the Port Forward tab. Click on the + icon at the top or bottom of the screen. Choose the Interface for the port forward (likely WAN) and if needed, pick a virtual IP address from the External Address ...


4

You haven't set up (or tried to use) a HTTP proxy, nor an ssh tunnel. Instead, you used port-forwarding over ssh. Forwarding TCP ports does not work for HTTP. Visiting a HTTP URL uses the domain of the URL at two different points. 1 - to find the IP address to send messages to. 2 - for the the Host header in the HTTP message. This lets one IP address ...


4

Using simple regex to parse HTML is naive. E.g. with newlines and ignoring special character encoding specified in the file. Do the right thing and really parse the page using any of the other real parsers mentioned in the other answers or use the following one liner: python -c "import bs4, urllib2; print ...


4

Rather than try and do this using wget I'd suggest using a more appropriate tool for downloading complex "sets" of files or filters. You can use httrack to download either entire directories of files (essentially mirror everything from a site) or you can specify to httrack a filter along with specific file extensions, such as download only .pdf files. You ...


3

You can use the iprange module to match a range of addresses. You want to DNAT the packets to your webserver. iptables --table nat --append PREROUTING --match iprange --src-range 10.13.13.10-10.13.13.19 --protocol tcp --dport 80 --jump DNAT --to-destination 1.2.3.4


3

You might be interested in HTTP Strict Transport Security sometimes called STS or HSTS. This is a voluntary header a web server can send out to instruct the web browser that the site must always be secure. Chrome is one of the browsers that implement it. http://www.imperialviolet.org/2011/02/17/hstsui.html I think you can add Wikipedia to the default ...


3

What you are looking for is called Mutual SSL Authentication. In the case of a commonly deployed web server as Apache, and an also commonly deployed SSL/TLS implementation as OpenSSL, the steps would be as follows: There is a handy script distributed alongside openssl, CA.sh to do most of this stuff. Its location is distribution specific. In Debian and ...


3

Try mitmproxy. mitmproxy is an SSL-capable man-in-the-middle proxy for HTTP. It provides a console interface that allows traffic flows to be inspected and edited on the fly. mitmdump is the command-line version of mitmproxy, with the same functionality but without the user interface. Think tcpdump for HTTP. Features Intercept HTTP requests and ...



Only top voted, non community-wiki answers of a minimum length are eligible