What would be the best way to come up with a shell script that grabs one or many Wistia videos?
There's plenty of proof-of-concepts out there for downloading YouTube videos via a shell script (i.e.: http://computing.dcu.ie/~humphrys/Notes/UNIX/lab.youtube.html) - however, given that Wistia uses some obfuscation - it seems a little more complicated.
For arguments sake, let's imagine that I wanted to 'wget' all of the videos (in, .mp4 or .flv format) from https://www.optimizely.com/opticon/sessions/ to a *nix server.
We can determine each of the video links (iFrames) to be:
- https://fast.wistia.net/embed/iframe/xcorh9bx2t?popover=true
- https://fast.wistia.net/embed/iframe/s0yu60meos?popover=true
- https://fast.wistia.net/embed/iframe/iz1icfwrgn?popover=true
- ...
As a starting point, I've started with:
$ wget https://fast.wistia.net/embed/iframe/xcorh9bx2t
Then, in xcorh9bx2t, we have some interesting HTML which can probably be parsed.
Example:
$ cat xcorh9bx2t | grep .flv
Wistia.iframeInit({"assets":[{"type":"original","slug":"original","display_name":"Original file","width":1920,"height":1080,"ext":"mp4","size":3099425793,"bitrate":10139,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/a1706057d64d7d4c1a39f28539a2475f8bd28123.bin","created_at":1435770832},{"type":"iphone_video","slug":"mp4_h264_932k","display_name":"360p","container":"mp4","codec":"h264","width":640,"height":360,"ext":"mp4","size":284937429,"bitrate":932,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/d402b0a53106ca07fac1fa6a403b280640546102.bin","created_at":1435770832,"opt_vbitrate":800},{"type":"flash_video","slug":"flv_h264_819k","display_name":"360p","container":"flv","codec":"h264","width":640,"height":360,"ext":"flv","size":250541036,"bitrate":819,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/33d11cd62475bc8e823f4091e802ee1c48407d02.bin","created_at":1435770832,"opt_vbitrate":700},{"type":"flash_video","slug":"flv_h264_330k","display_name":"224p","container":"flv","codec":"h264","width":400,"height":224,"ext":"flv","size":101132686,"bitrate":330,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/35e479cea895d39f6ffa62a1b30f5417b3bbdbe7.bin","created_at":1435770832,"opt_vbitrate":200},{"type":"mp4_video","slug":"mp4_h264_328k","display_name":"224p","container":"mp4","codec":"h264","width":400,"height":224,"ext":"mp4","size":100518328,"bitrate":328,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/f381cae06fbf0812def4dd25b02d75cf5a755862.bin","created_at":1435770832,"opt_vbitrate":200},{"type":"md_flash_video","slug":"flv_h264_1308k","display_name":"540p","container":"flv","codec":"h264","width":960,"height":540,"ext":"flv","size":400097566,"bitrate":1308,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/e7bb3744f468b6957a66e77646264ba7df192da8.bin","created_at":1435770832,"opt_vbitrate":1200},{"type":"md_mp4_video","slug":"mp4_h264_1306k","display_name":"540p","container":"mp4","codec":"h264","width":960,"height":540,"ext":"mp4","size":399475890,"bitrate":1306,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/96c7c3df12bca82781bb91f828c5a73584d41986.bin","created_at":1435770832,"opt_vbitrate":1200},{"type":"hd_flash_video","slug":"flv_h264_2579k","display_name":"720p","container":"flv","codec":"h264","width":1280,"height":720,"ext":"flv","size":788608424,"bitrate":2579,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/4bdc9ccad52d20fa8ef0758723f462cc0975b058.bin","created_at":1435770832,"opt_vbitrate":2500},{"type":"hd_mp4_video","slug":"mp4_h264_2577k","display_name":"720p","container":"mp4","codec":"h264","width":1280,"height":720,"ext":"mp4","size":787978856,"bitrate":2577,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/c6d2bb8dbe4efcc83bbc83360640b35fff88cf09.bin","created_at":1435770832,"opt_vbitrate":2500},{"type":"hd_flash_video","slug":"flv_h264_3802k","display_name":"1080p","container":"flv","codec":"h264","width":1920,"height":1080,"ext":"flv","size":1162495511,"bitrate":3802,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/842bd773588d0ceedec3bd1b947d203735b7bed2.bin","created_at":1435770832,"opt_vbitrate":3750},{"type":"hd_mp4_video","slug":"mp4_h264_3800k","display_name":"1080p","container":"mp4","codec":"h264","width":1920,"height":1080,"ext":"mp4","size":1161857141,"bitrate":3800,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/6f3a6043d431a58b907503997d7fd276c584cb4b.bin","created_at":1435770832,"opt_vbitrate":3750},{"type":"mp4_video","slug":"mp4_h264_311k","display_name":"360p","container":"mp4","codec":"h264","width":640,"height":360,"ext":"mp4","size":95330863,"bitrate":311,"public":false,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/2eaee4ad203f0a5ad223c2db5a9b5c6506683df1.bin","created_at":1435770832,"opt_vbitrate":600},{"type":"still_image","slug":"still_image_1280x720","display_name":"Image","width":1280,"height":720,"ext":"jpg","size":92160,"bitrate":0,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/be55b81db186b83574c23db7c893e12b5651cc80.bin","created_at":1436925915}],"unnamed_assets":[{"type":"original","slug":"original","display_name":"Original file","width":1920,"height":1080,"ext":"mp4","size":3099425793,"bitrate":10139,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/a1706057d64d7d4c1a39f28539a2475f8bd28123.bin","created_at":1435770832},{"type":"iphone_video","slug":"mp4_h264_932k","display_name":"360p","container":"mp4","codec":"h264","width":640,"height":360,"ext":"mp4","size":284937429,"bitrate":932,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/d402b0a53106ca07fac1fa6a403b280640546102.bin","created_at":1435770832,"opt_vbitrate":800},{"type":"flash_video","slug":"flv_h264_819k","display_name":"360p","container":"flv","codec":"h264","width":640,"height":360,"ext":"flv","size":250541036,"bitrate":819,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/33d11cd62475bc8e823f4091e802ee1c48407d02.bin","created_at":1435770832,"opt_vbitrate":700},{"type":"flash_video","slug":"flv_h264_330k","display_name":"224p","container":"flv","codec":"h264","width":400,"height":224,"ext":"flv","size":101132686,"bitrate":330,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/35e479cea895d39f6ffa62a1b30f5417b3bbdbe7.bin","created_at":1435770832,"opt_vbitrate":200},{"type":"mp4_video","slug":"mp4_h264_328k","display_name":"224p","container":"mp4","codec":"h264","width":400,"height":224,"ext":"mp4","size":100518328,"bitrate":328,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/f381cae06fbf0812def4dd25b02d75cf5a755862.bin","created_at":1435770832,"opt_vbitrate":200},{"type":"md_flash_video","slug":"flv_h264_1308k","display_name":"540p","container" "flv","codec":"h264","width":960,"height":540,"ext":"flv","size":400097566,"bitrate":1308,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/e7bb3744f468b6957a66e77646264ba7df192da8.bin","created_at":1435770832,"opt_vbitrate":1200},{"type":"md_mp4_video","slug":"mp4_h264_1306k","display_name":"540p","container":"mp4","codec":"h264","width":960,"height":540,"ext":"mp4","size":399475890,"bitrate":1306,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/96c7c3df12bca82781bb91f828c5a73584d41986.bin","created_at":1435770832,"opt_vbitrate":1200},{"type":"hd_flash_video","slug":"flv_h264_2579k","display_name":"720p","container":"flv","codec":"h264","width":1280,"height":720,"ext":"flv","size":788608424,"bitrate":2579,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/4bdc9ccad52d20fa8ef0758723f462cc0975b058.bin","created_at":1435770832,"opt_vbitrate":2500},{"type":"hd_mp4_video","slug":"mp4_h264_2577k","display_name":"720p","container":"mp4","codec":"h264","width":1280,"height":720,"ext":"mp4","size":787978856,"bitrate":2577,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/c6d2bb8dbe4efcc83bbc83360640b35fff88cf09.bin","created_at":1435770832,"opt_vbitrate":2500},{"type":"hd_flash_video","slug":"flv_h264_3802k","display_name":"1080p","container":"flv","codec":"h264","width":1920,"height":1080,"ext":"flv","size":1162495511,"bitrate":3802,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/842bd773588d0ceedec3bd1b947d203735b7bed2.bin","created_at":1435770832,"opt_vbitrate":3750},{"type":"hd_mp4_video","slug":"mp4_h264_3800k","display_name":"1080p","container":"mp4","codec":"h264","width":1920,"height":1080,"ext":"mp4","size":1161857141,"bitrate":3800,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/6f3a6043d431a58b907503997d7fd276c584cb4b.bin","created_at":1435770832,"opt_vbitrate":3750},{"type":"mp4_video","slug":"mp4_h264_311k","display_name":"360p","container":"mp4","codec":"h264","width":640,"height":360,"ext":"mp4","size":95330863,"bitrate":311,"public":false,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/2eaee4ad203f0a5ad223c2db5a9b5c6506683df1.bin","created_at":1435770832,"opt_vbitrate":600},{"type":"still_image","slug":"still_image_1280x720","display_name":"Image","width":1280,"height":720,"ext":"jpg","size":92160,"bitrate":0,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/be55b81db186b83574c23db7c893e12b5651cc80.bin","created_at":1436925915}],"distilleryUrl":"https://distillery-main.wistia.com/x","accountKey":"wistia-production_58149","mediaKey":"wistia-production_14783299","type":"Video","progress":1.0,"status":2,"name":"Opticon 2015 Keynote Address","duration":2388.35,"hashedId":"xcorh9bx2t","branding":false,"seoDescription":"an Opticon Session Videos (Public) video from Optimizely","preloadPreference":null,"flashPlayerUrl":"https://embed-ssl.wistia.com/flash/embed_player_v2.0.swf?2015-02-27","showAbout":true,"createdAt":1435770832,"firstEmbedForAccount":false,"firstShareForAccount":false,"stats":{"loadCount":403,"playCount":369,"uniqueLoadCount":267,"uniquePlayCount":250,"averageEngagement":0.186758},"trackingTransmitInterval":79,"playerPreference":"auto","integrations":{"marketo":true},"embed_options":{"volumeControl":"true","fullscreenButton":"true","controlsVisibleOnLoad":"true","playerColor":"7b796a","bpbTime":"false","googleAnalytics":true,"videoQuality":"","vulcan":"false","version":"v2","playButton":"true","smallPlayButton":"true","playbar":"true","branding":"false","plugin":{"socialbar-v1":{"buttons":"twitter-linkedIn-facebook","showTweetCount":"false","tweetText":"Learning lots from the thought leadership at @optimizely! Check out: {video_name}","height":"25"}},"autoPlay":"false","endVideoBehavior":"default"}}, {});
Essentially, it looks like the Wistia iFrame has some JavaScript that runs the function Wistia.iframeInit() and points it to an obsfucated .bin URL such as: https://embed-ssl.wistia.com/deliveries/[Unique40BitToken].bin.
If I parse this out and then wget that (or open it in a browser) - it does seem to work, i.e.:
$ wget https://embed-ssl.wistia.com/deliveries/d402b0a53106ca07fac1fa6a403b280640546102.bin
--2016-06-26 14:29:08-- https://embed-ssl.wistia.com/deliveries/d402b0a53106ca07fac1fa6a403b280640546102.bin
Resolving embed-ssl.wistia.com (embed-ssl.wistia.com)... 72.21.81.253
Connecting to embed-ssl.wistia.com (embed-ssl.wistia.com)|72.21.81.253|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 284937429 (272M) [video/mp4]
Saving to: âd402b0a53106ca07fac1fa6a403b280640546102.binâ
100%[======================================================================================================================================================================>] 284,937,429 45.7MB/s in 6.9s
2016-06-26 14:29:15 (39.2 MB/s) - âd402b0a53106ca07fac1fa6a403b280640546102.binâ saved [284937429/284937429]
The resulting file is 272M which is about right.
So, what would be the easiest way to make this into an elegant one-liner? Parsing the parameters of Wistia.iframeInit() (which appears to be JSON) with sed or awk is most likely the best (or maybe even jsawk) appears to be the best approach.
