Hot answers tagged performance
Check out 'patch', back in the day this was the same problems we had downloading kernel source code on an almost daily basis. This can be used to get the original file updated or wound back to previous versions using the diff files.
As you mentioned the diff program, I may safely assume that you are talking about text files... As you mention 10GB in 200 million lines, this seems to be a single file with an average line length of 50 which looks OK. In such a situation, a version control system is the right way to go. You need to find the right version control system for your ...
This is exactly the job version control software like Git, Bazaar or Subversion is doing (plus a little bit extra). So your workflow could be as follows: At the beginning of a month create a new repository. At every day copy new files into the repository and commit the changes Optionally, remove last month's repository. I the files do not change much ...
Only top voted, non community-wiki answers of a minimum length are eligible