As you mentioned the
diff program, I may safely assume that you are talking about text files...
As you mention 10GB in 200 million lines, this seems to be a single file with an average line length of 50 which looks OK.
- In such a situation, a version control system is the right way to go.
You need to find the right version control system for your problem. Let me thus assume your information:
- One new file version every day and 1% of the content changes from day to day.
git does not maintain file deltas but rather stores gzip -4 compressed full files, I expect that after aprox. 2-3 weeks, git will consume more disk space than you expect. So git is not the right tool for your case.
There are other version control systems that use differences for their history handling method.
RCS stores reversed diffs and could be a solution, but RCS is slow for files > 256KBytes and RCS takes more time if you do not need the latest version but something older.
SCCS is based on diffs, but the storage format is the so called
weave format that effectively stores all versions at the same time and allows you to retrieve any version at the same fixed time.
SCCS will create an initial 10GB history file and this history file will grow by 1% for every new version in your case, so I expect the history file to use aprox. 36.5GB after a year. For GIT, I expect a disk space requirement of 100-400GB after a year.
SCCS is OpenSource and can be retrieved from:
SCCS is maintained since 1972 (43 years) and thus can be seen as mature ;-)
and BTW: I know no faster version control system.