0

I've tried to figure out how to deal with an issue processing a large file list.

I have a list of 2000+ files. When I try to put them in a loop using ls or wc to get the file size for each, after n files, ls/wc fails. It seems if I put a pause for 2 seconds in place every 3-4 files, ls/wc works.

To compound the issue, I'm also using rsh to run this on a remote server so I can compare file sizes on server1 to server2. However, I do have the issue locally when not using rsh as well, but it gets through more of the file list before starting to fail.

server="xy1"
for CompareList in `cat compare_jnj.txt`
do
     rsh $server wc -c /u2/web/$CompareList.java |awk '{print " ",$1," ",$2}'|tr -s " " >> output.out
     rsh $server wc -c /u2/web/$CompareList.class |awk '{print " ",$1," ",$2}'|tr -s " " >> output.out
done

My file list is a static list of file locations/names without the file extension. I've wiped out a lot of what I do in it for safety reasons, but this shows the loop/commands I'm using:

That's after I edited it from using ls -l to get the file name/size to export to a result set

5
  • 1
    Could you show your loop sample code?
    – Tagwint
    Oct 25, 2017 at 16:16
  • If you do a ls -l * and you have a trillion files, they would be expanded by the * and that is more bytes than your command line can handle. So, show us your code as Tagwint writes.
    – hschou
    Oct 25, 2017 at 18:22
  • I edited my original to add a subset of what I'm doing in my script since I couldn't figure out how to get it to format in a comment.
    – Bryan L
    Oct 26, 2017 at 17:58
  • Using wc -c to get the size of a file is a very roundabout way to do it. I would (again) suggest stat or du.
    – DopeGhoti
    Oct 26, 2017 at 18:44
  • wc -c was just something I was trying vs using ls to see if I got different results. I didn't. Both have a similar issue.
    – Bryan L
    Oct 27, 2017 at 19:48

2 Answers 2

3

It sounds like your script is parsing the output of ls (or wc) just to get a list of file names. Don't do this; it's a Very Bad Idea.

To get a simple pipe-delimited list of files' names and their sizes in bytes, you can for example use:

stat -c '%n|%s' /path/to/directory/*

Or, to parse a directory tree:

find /path/to/directory -type f -exec stat -c '%n|%s' "{}" \;
1
  • I know the file list I want to get the size for. It's a static text file with the path/names of the files I'm trying to compare. I added a bit to my original question showing the syntax of what I'm running.
    – Bryan L
    Oct 26, 2017 at 18:04
0

This is the command I usually run against a directory tree to list the files and their sizes:

find <directory>/* -type f -print0 | xargs -0 -n1 du -h

You can also sort by filesize (I find this useful for my job):

find <directory>/* -type f -print0 | xargs -0 -n1 du -h | sort -n -r
1
  • I know the file list I want to get the size for. It's a static text file with the path/names of the files I'm trying to compare. I added a bit to my original question showing the syntax of what I'm running.
    – Bryan L
    Oct 26, 2017 at 18:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .