3

What would be the best way to come up with a shell script that grabs one or many Wistia videos?

There's plenty of proof-of-concepts out there for downloading YouTube videos via a shell script (i.e.: http://computing.dcu.ie/~humphrys/Notes/UNIX/lab.youtube.html) - however, given that Wistia uses some obfuscation - it seems a little more complicated.

For arguments sake, let's imagine that I wanted to 'wget' all of the videos (in, .mp4 or .flv format) from https://www.optimizely.com/opticon/sessions/ to a *nix server.

We can determine each of the video links (iFrames) to be:

  1. https://fast.wistia.net/embed/iframe/xcorh9bx2t?popover=true
  2. https://fast.wistia.net/embed/iframe/s0yu60meos?popover=true
  3. https://fast.wistia.net/embed/iframe/iz1icfwrgn?popover=true
  4. ...

As a starting point, I've started with:

$ wget https://fast.wistia.net/embed/iframe/xcorh9bx2t

Then, in xcorh9bx2t, we have some interesting HTML which can probably be parsed.

Example:

$ cat xcorh9bx2t | grep .flv
    Wistia.iframeInit({"assets":[{"type":"original","slug":"original","display_name":"Original file","width":1920,"height":1080,"ext":"mp4","size":3099425793,"bitrate":10139,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/a1706057d64d7d4c1a39f28539a2475f8bd28123.bin","created_at":1435770832},{"type":"iphone_video","slug":"mp4_h264_932k","display_name":"360p","container":"mp4","codec":"h264","width":640,"height":360,"ext":"mp4","size":284937429,"bitrate":932,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/d402b0a53106ca07fac1fa6a403b280640546102.bin","created_at":1435770832,"opt_vbitrate":800},{"type":"flash_video","slug":"flv_h264_819k","display_name":"360p","container":"flv","codec":"h264","width":640,"height":360,"ext":"flv","size":250541036,"bitrate":819,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/33d11cd62475bc8e823f4091e802ee1c48407d02.bin","created_at":1435770832,"opt_vbitrate":700},{"type":"flash_video","slug":"flv_h264_330k","display_name":"224p","container":"flv","codec":"h264","width":400,"height":224,"ext":"flv","size":101132686,"bitrate":330,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/35e479cea895d39f6ffa62a1b30f5417b3bbdbe7.bin","created_at":1435770832,"opt_vbitrate":200},{"type":"mp4_video","slug":"mp4_h264_328k","display_name":"224p","container":"mp4","codec":"h264","width":400,"height":224,"ext":"mp4","size":100518328,"bitrate":328,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/f381cae06fbf0812def4dd25b02d75cf5a755862.bin","created_at":1435770832,"opt_vbitrate":200},{"type":"md_flash_video","slug":"flv_h264_1308k","display_name":"540p","container":"flv","codec":"h264","width":960,"height":540,"ext":"flv","size":400097566,"bitrate":1308,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/e7bb3744f468b6957a66e77646264ba7df192da8.bin","created_at":1435770832,"opt_vbitrate":1200},{"type":"md_mp4_video","slug":"mp4_h264_1306k","display_name":"540p","container":"mp4","codec":"h264","width":960,"height":540,"ext":"mp4","size":399475890,"bitrate":1306,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/96c7c3df12bca82781bb91f828c5a73584d41986.bin","created_at":1435770832,"opt_vbitrate":1200},{"type":"hd_flash_video","slug":"flv_h264_2579k","display_name":"720p","container":"flv","codec":"h264","width":1280,"height":720,"ext":"flv","size":788608424,"bitrate":2579,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/4bdc9ccad52d20fa8ef0758723f462cc0975b058.bin","created_at":1435770832,"opt_vbitrate":2500},{"type":"hd_mp4_video","slug":"mp4_h264_2577k","display_name":"720p","container":"mp4","codec":"h264","width":1280,"height":720,"ext":"mp4","size":787978856,"bitrate":2577,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/c6d2bb8dbe4efcc83bbc83360640b35fff88cf09.bin","created_at":1435770832,"opt_vbitrate":2500},{"type":"hd_flash_video","slug":"flv_h264_3802k","display_name":"1080p","container":"flv","codec":"h264","width":1920,"height":1080,"ext":"flv","size":1162495511,"bitrate":3802,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/842bd773588d0ceedec3bd1b947d203735b7bed2.bin","created_at":1435770832,"opt_vbitrate":3750},{"type":"hd_mp4_video","slug":"mp4_h264_3800k","display_name":"1080p","container":"mp4","codec":"h264","width":1920,"height":1080,"ext":"mp4","size":1161857141,"bitrate":3800,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/6f3a6043d431a58b907503997d7fd276c584cb4b.bin","created_at":1435770832,"opt_vbitrate":3750},{"type":"mp4_video","slug":"mp4_h264_311k","display_name":"360p","container":"mp4","codec":"h264","width":640,"height":360,"ext":"mp4","size":95330863,"bitrate":311,"public":false,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/2eaee4ad203f0a5ad223c2db5a9b5c6506683df1.bin","created_at":1435770832,"opt_vbitrate":600},{"type":"still_image","slug":"still_image_1280x720","display_name":"Image","width":1280,"height":720,"ext":"jpg","size":92160,"bitrate":0,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/be55b81db186b83574c23db7c893e12b5651cc80.bin","created_at":1436925915}],"unnamed_assets":[{"type":"original","slug":"original","display_name":"Original file","width":1920,"height":1080,"ext":"mp4","size":3099425793,"bitrate":10139,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/a1706057d64d7d4c1a39f28539a2475f8bd28123.bin","created_at":1435770832},{"type":"iphone_video","slug":"mp4_h264_932k","display_name":"360p","container":"mp4","codec":"h264","width":640,"height":360,"ext":"mp4","size":284937429,"bitrate":932,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/d402b0a53106ca07fac1fa6a403b280640546102.bin","created_at":1435770832,"opt_vbitrate":800},{"type":"flash_video","slug":"flv_h264_819k","display_name":"360p","container":"flv","codec":"h264","width":640,"height":360,"ext":"flv","size":250541036,"bitrate":819,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/33d11cd62475bc8e823f4091e802ee1c48407d02.bin","created_at":1435770832,"opt_vbitrate":700},{"type":"flash_video","slug":"flv_h264_330k","display_name":"224p","container":"flv","codec":"h264","width":400,"height":224,"ext":"flv","size":101132686,"bitrate":330,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/35e479cea895d39f6ffa62a1b30f5417b3bbdbe7.bin","created_at":1435770832,"opt_vbitrate":200},{"type":"mp4_video","slug":"mp4_h264_328k","display_name":"224p","container":"mp4","codec":"h264","width":400,"height":224,"ext":"mp4","size":100518328,"bitrate":328,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/f381cae06fbf0812def4dd25b02d75cf5a755862.bin","created_at":1435770832,"opt_vbitrate":200},{"type":"md_flash_video","slug":"flv_h264_1308k","display_name":"540p","container" "flv","codec":"h264","width":960,"height":540,"ext":"flv","size":400097566,"bitrate":1308,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/e7bb3744f468b6957a66e77646264ba7df192da8.bin","created_at":1435770832,"opt_vbitrate":1200},{"type":"md_mp4_video","slug":"mp4_h264_1306k","display_name":"540p","container":"mp4","codec":"h264","width":960,"height":540,"ext":"mp4","size":399475890,"bitrate":1306,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/96c7c3df12bca82781bb91f828c5a73584d41986.bin","created_at":1435770832,"opt_vbitrate":1200},{"type":"hd_flash_video","slug":"flv_h264_2579k","display_name":"720p","container":"flv","codec":"h264","width":1280,"height":720,"ext":"flv","size":788608424,"bitrate":2579,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/4bdc9ccad52d20fa8ef0758723f462cc0975b058.bin","created_at":1435770832,"opt_vbitrate":2500},{"type":"hd_mp4_video","slug":"mp4_h264_2577k","display_name":"720p","container":"mp4","codec":"h264","width":1280,"height":720,"ext":"mp4","size":787978856,"bitrate":2577,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/c6d2bb8dbe4efcc83bbc83360640b35fff88cf09.bin","created_at":1435770832,"opt_vbitrate":2500},{"type":"hd_flash_video","slug":"flv_h264_3802k","display_name":"1080p","container":"flv","codec":"h264","width":1920,"height":1080,"ext":"flv","size":1162495511,"bitrate":3802,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/842bd773588d0ceedec3bd1b947d203735b7bed2.bin","created_at":1435770832,"opt_vbitrate":3750},{"type":"hd_mp4_video","slug":"mp4_h264_3800k","display_name":"1080p","container":"mp4","codec":"h264","width":1920,"height":1080,"ext":"mp4","size":1161857141,"bitrate":3800,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/6f3a6043d431a58b907503997d7fd276c584cb4b.bin","created_at":1435770832,"opt_vbitrate":3750},{"type":"mp4_video","slug":"mp4_h264_311k","display_name":"360p","container":"mp4","codec":"h264","width":640,"height":360,"ext":"mp4","size":95330863,"bitrate":311,"public":false,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/2eaee4ad203f0a5ad223c2db5a9b5c6506683df1.bin","created_at":1435770832,"opt_vbitrate":600},{"type":"still_image","slug":"still_image_1280x720","display_name":"Image","width":1280,"height":720,"ext":"jpg","size":92160,"bitrate":0,"public":true,"status":2,"progress":1.0,"url":"https://embed-ssl.wistia.com/deliveries/be55b81db186b83574c23db7c893e12b5651cc80.bin","created_at":1436925915}],"distilleryUrl":"https://distillery-main.wistia.com/x","accountKey":"wistia-production_58149","mediaKey":"wistia-production_14783299","type":"Video","progress":1.0,"status":2,"name":"Opticon 2015 Keynote Address","duration":2388.35,"hashedId":"xcorh9bx2t","branding":false,"seoDescription":"an Opticon Session Videos (Public) video from Optimizely","preloadPreference":null,"flashPlayerUrl":"https://embed-ssl.wistia.com/flash/embed_player_v2.0.swf?2015-02-27","showAbout":true,"createdAt":1435770832,"firstEmbedForAccount":false,"firstShareForAccount":false,"stats":{"loadCount":403,"playCount":369,"uniqueLoadCount":267,"uniquePlayCount":250,"averageEngagement":0.186758},"trackingTransmitInterval":79,"playerPreference":"auto","integrations":{"marketo":true},"embed_options":{"volumeControl":"true","fullscreenButton":"true","controlsVisibleOnLoad":"true","playerColor":"7b796a","bpbTime":"false","googleAnalytics":true,"videoQuality":"","vulcan":"false","version":"v2","playButton":"true","smallPlayButton":"true","playbar":"true","branding":"false","plugin":{"socialbar-v1":{"buttons":"twitter-linkedIn-facebook","showTweetCount":"false","tweetText":"Learning lots from the thought leadership at @optimizely! Check out: {video_name}","height":"25"}},"autoPlay":"false","endVideoBehavior":"default"}}, {});

Essentially, it looks like the Wistia iFrame has some JavaScript that runs the function Wistia.iframeInit() and points it to an obsfucated .bin URL such as: https://embed-ssl.wistia.com/deliveries/[Unique40BitToken].bin.

If I parse this out and then wget that (or open it in a browser) - it does seem to work, i.e.:

$ wget https://embed-ssl.wistia.com/deliveries/d402b0a53106ca07fac1fa6a403b280640546102.bin
--2016-06-26 14:29:08--  https://embed-ssl.wistia.com/deliveries/d402b0a53106ca07fac1fa6a403b280640546102.bin
Resolving embed-ssl.wistia.com (embed-ssl.wistia.com)... 72.21.81.253
Connecting to embed-ssl.wistia.com (embed-ssl.wistia.com)|72.21.81.253|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 284937429 (272M) [video/mp4]
Saving to: âd402b0a53106ca07fac1fa6a403b280640546102.binâ

100%[======================================================================================================================================================================>] 284,937,429 45.7MB/s   in 6.9s

2016-06-26 14:29:15 (39.2 MB/s) - âd402b0a53106ca07fac1fa6a403b280640546102.binâ saved [284937429/284937429]

The resulting file is 272M which is about right.

So, what would be the easiest way to make this into an elegant one-liner? Parsing the parameters of Wistia.iframeInit() (which appears to be JSON) with sed or awk is most likely the best (or maybe even jsawk) appears to be the best approach.

2

YouTube-dl supports this:

$ youtube-dl fast.wistia.net/embed/iframe/xcorh9bx2t
WARNING: The url doesn't specify the protocol, trying with http
[Wistia] xcorh9bx2t: Downloading JSON metadata
[download] Destination: Opticon 2015 Keynote Address-xcorh9bx2t.mp4

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.