0

I would like to figure out how to manage my storage, which has been quite large. The idea would be to see unnecessary files, and archive/delete.

At the moment, I am simply doing `du -h max-depth 1 and piping the outputs into text files.

However, I would like more information in order to figure out how to archive/delete my current storage.

---What is the best way I could get the file type and size for all files, recursively? This would be a quick way to check for intermediate files which are useless

---I was thinking about outputting all file metadata (i.e. data user, file type, file size, timestamp) into one large tsv file, so I could subset this in Python/R based on size and file type. Is this possible? How would we go about getting this information?

---Are there methods to look at storage versus time?

---Are there any other diagnostics/fancy visualizations which could help me figure out how to organize/archive/delete the data currently stored?

1

This is too long for a comment, but not good enough for an answer. Sorry about that.

---What is the best way I could get the file type and size for all files, recursively? This would be a quick way to check for intermediate files which are useless

Not sure about the "best" part, but first hit on Google searching for "advanced disk usage linux" leads to this.

---I was thinking about outputting all file metadata (i.e. data user, file type, file size, timestamp) into one large tsv file, so I could subset this in Python/R based on size and file type. Is this possible? How would we go about getting this information?

With Python, sure: just use os.walk() to recurse the directory and print out all relevant information about each file. I'd probably use SQLite as an output format instead of TSV though.

---Are there methods to look at storage versus time?

You can run du -s periodically and feed the results to a RRD database. Then you can make nice graphs out of it.

---Are there any other diagnostics/fancy visualizations which could help me figure out how to organize/archive/delete the data currently stored?

There should be, but I'm skeptical about how useful they are for your particular situation.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.