New answers tagged large-files
Old-style: tail -n +14600 filename | head -n 100 and less-memory-save variant: head -n 14700 filename | tail -n +14600
using awk awk 'NR>=14600&&NR<=147000' filename
Using sed sed -n 14600,14700p filename > newfile Where: p : Print out the pattern space (to the standard output). This command is usually only used in conjunction with the -n command-line option. n : If auto-print is not disabled, print the pattern space, then, regardless, replace the pattern space with the next line of input. If there is no more ...
You can use sed to get those results: sed -n '14600,14700p;14700q' largefile
Some ideas: 1.- Instead of calling cut repeatedly on each line, take advantage of read. The list of variables cut on ' ' are: projectName 1 filepath 2 numbers 3 lang 9 cloneID 10 cloneSubID 11 minToken 12 stride 13 similarity 14 That could be done directly by read as this: while read -r projectName filepath numbers a a a a a lang cloneID cloneSubID ...
One problem here is that you do: while : loop do : processing echo "$results" >>output done <input This will result in minutely increasing execution time per iteration simply because the output is repeatedly *open()*ed at a slightly larger offset than it was the last time. I say minutely because there is virtually no difference in how ...
Large files can be slightly slower to work with than small files — and I don't mean just because there's more data. If file B is 1000 times the size of file A, then it may take 1001 or 1002 times as long to process in its entirety. Reopening the output file (and seeking to the end) on each iteration is a slight performance drain. Try changing your second ...
Have you tried putting the file in /dev/shm, which is a ram-resided filesystem. It will boost your access speed both for read n write from/to file. Finally you may copy the file from shm to the permanent disk partition.
Top 50 recent answers are included