I have downloaded a number of input files from the NASDAQ website (for example ftp://emi.nasdaq.com/ITCH/01302018.NASDAQ_ITCH50.gz) After decompressing (I double click on a Mac) I can't see the file contents using head or tail or cat. I only see gibberish characters. I have tried downloading each file three times or so. Are there any settings in the terminal that I should change (it's my first time doing this on this machine).

Also I tried checking their MD5 with what is provided on the website. It appears they don't match. What can be done in this case? Again, I downloaded them multiple times to account for the possibility of bits being dropped, etc.

I assume popular website like that will not upload corrupt data files, right?

Could you guide me through what I need to do to see the contents of the files? I have never faced this problem before; I was always able to inspect a file by doing head or cat.

The files are in this folder: ftp://emi.nasdaq.com/ITCH/

  • 1
    just to be sure, you compared the md5sum of the initial, compressed download file? – Jeff Schaller Feb 23 at 1:51
  • No! I uncompressed the file first! – Tapal Goosal Feb 23 at 1:55
  • Then, assuming the MD5s match (they should or they would know immediately, right?), what can I do? – Tapal Goosal Feb 23 at 1:55
  • 1
    I noticed that the md5sum filename contained gz, so I assume they mean for you to compare to the .gz file. – Jeff Schaller Feb 23 at 1:56

Your file contains a NASDAQ TotalView-ITCH Data Feed, as described in Data News #2008 - 91.

The format used by the file is not a pure text format but a binary encoded format.

This binary format is described in detail in the PDF document NASDAQ TotalView-ITCH 4.0. The document NASDAQ TotalView-ITCH 3.1 describes a text-based encoding format.

This is just as a way of allowing you to get further with these files. There is probably software out there that will read these highly domain specific data files. The data provider (NASDAQ) may possibly reference this in their documentation.

The MD5 checksums should be checked against the original, compressed files. Not the uncompressed files (or even re-compressed files).

In general, when a data provider gives you a MD5 checksum for a data file, it will be the checksum of the file that they give to you, in the format that they give it to you.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.