3

Every day 10GB text file is downloaded, the file is ~200 million lines and ~1% of the lines are changed the next day. I want to keep daily files as backup, but I'm trying to save disk space by using the CPU.

EDIT Currently the best way I found is to keep diff files and rebuild them with patch (how @Simon suggested), for example on 01 Jan download the big file and then for a whole month keep doing only diff diff 01jan.txt 02jan.txt > 02jan.diff; rm 02jan.txt and so on for every day of the month.

Is there better way to do this?

1

This is exactly the job version control software like Git, Bazaar or Subversion is doing (plus a little bit extra). So your workflow could be as follows:

  • At the beginning of a month create a new repository.
  • At every day copy new files into the repository and commit the changes
  • Optionally, remove last month's repository.

I the files do not change much from month to month, just work with one repository for every mounth.

1

As you mentioned the diff program, I may safely assume that you are talking about text files...

As you mention 10GB in 200 million lines, this seems to be a single file with an average line length of 50 which looks OK.

  • In such a situation, a version control system is the right way to go.

You need to find the right version control system for your problem. Let me thus assume your information:

  • One new file version every day and 1% of the content changes from day to day.

Given that git does not maintain file deltas but rather stores gzip -4 compressed full files, I expect that after aprox. 2-3 weeks, git will consume more disk space than you expect. So git is not the right tool for your case.

There are other version control systems that use differences for their history handling method.

  • RCS stores reversed diffs and could be a solution, but RCS is slow for files > 256KBytes and RCS takes more time if you do not need the latest version but something older.

  • SCCS is based on diffs, but the storage format is the so called weave format that effectively stores all versions at the same time and allows you to retrieve any version at the same fixed time.

SCCS will create an initial 10GB history file and this history file will grow by 1% for every new version in your case, so I expect the history file to use aprox. 36.5GB after a year. For GIT, I expect a disk space requirement of 100-400GB after a year.

SCCS is OpenSource and can be retrieved from:

http://sccs.sourceforge.net

SCCS is maintained since 1972 (43 years) and thus can be seen as mature ;-) and BTW: I know no faster version control system.

  • Thanks, I see you wrote the program, that is great! I'll try it. – nacholibre Sep 14 '15 at 13:41
  • 1
    The programs have been initially written by Marc J. Rochkind and the frontend program sccs was initially written by Eric Allman. Since I convinced Sun Microsystems to make the software OpenSource in 2006, I am maintaining it. – schily Sep 14 '15 at 13:47
1

Check out 'patch', back in the day this was the same problems we had downloading kernel source code on an almost daily basis. This can be used to get the original file updated or wound back to previous versions using the diff files.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.