0

I've tried to figure out how to deal with an issue processing a large file list.

I have a list of 2000+ files. When I try to put them in a loop using ls or wc to get the file size for each, after n files, ls/wc fails. It seems if I put a pause for 2 seconds in place every 3-4 files, ls/wc works.

To compound the issue, I'm also using rsh to run this on a remote server so I can compare file sizes on server1 to server2. However, I do have the issue locally when not using rsh as well, but it gets through more of the file list before starting to fail.

server="xy1"
for CompareList in `cat compare_jnj.txt`
do
     rsh $server wc -c /u2/web/$CompareList.java |awk '{print " ",$1," ",$2}'|tr -s " " >> output.out
     rsh $server wc -c /u2/web/$CompareList.class |awk '{print " ",$1," ",$2}'|tr -s " " >> output.out
done

My file list is a static list of file locations/names without the file extension. I've wiped out a lot of what I do in it for safety reasons, but this shows the loop/commands I'm using:

That's after I edited it from using ls -l to get the file name/size to export to a result set

5
  • 1
    Could you show your loop sample code?
    – Tagwint
    Oct 25 '17 at 16:16
  • If you do a ls -l * and you have a trillion files, they would be expanded by the * and that is more bytes than your command line can handle. So, show us your code as Tagwint writes.
    – hschou
    Oct 25 '17 at 18:22
  • I edited my original to add a subset of what I'm doing in my script since I couldn't figure out how to get it to format in a comment.
    – Bryan L
    Oct 26 '17 at 17:58
  • Using wc -c to get the size of a file is a very roundabout way to do it. I would (again) suggest stat or du.
    – DopeGhoti
    Oct 26 '17 at 18:44
  • wc -c was just something I was trying vs using ls to see if I got different results. I didn't. Both have a similar issue.
    – Bryan L
    Oct 27 '17 at 19:48
3

It sounds like your script is parsing the output of ls (or wc) just to get a list of file names. Don't do this; it's a Very Bad Idea.

To get a simple pipe-delimited list of files' names and their sizes in bytes, you can for example use:

stat -c '%n|%s' /path/to/directory/*

Or, to parse a directory tree:

find /path/to/directory -type f -exec stat -c '%n|%s' "{}" \;
1
  • I know the file list I want to get the size for. It's a static text file with the path/names of the files I'm trying to compare. I added a bit to my original question showing the syntax of what I'm running.
    – Bryan L
    Oct 26 '17 at 18:04
0

This is the command I usually run against a directory tree to list the files and their sizes:

find <directory>/* -type f -print0 | xargs -0 -n1 du -h

You can also sort by filesize (I find this useful for my job):

find <directory>/* -type f -print0 | xargs -0 -n1 du -h | sort -n -r
1
  • I know the file list I want to get the size for. It's a static text file with the path/names of the files I'm trying to compare. I added a bit to my original question showing the syntax of what I'm running.
    – Bryan L
    Oct 26 '17 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.