0

I have a shell script for working with text conversions, but it is cumbersome to use and relies on manual judgment and execution. It works as follows.

Step 1: download raw data text.

wget http://example.com/raw.txt

Step 2: Manually compare the contents of the last downloaded file for differences.

The content format of the text is relatively simple, one string per line. If the content has increased, you need to run script processing.

# Last downloaded file content:
wtKpjv
uJlQm1
iS86aE
Hk6ycS
hAYj7k

# Now downloaded file content:
wtKpjv
uJlQm1
RiU8iM
iS86aE
Hk6ycS
qyDNaZ
hAYj7k

# Compare the increased content:
RiU8iM
qyDNaZ

Step 3: If there is a difference in the comparison results, you need to run script processing.

./text_processing.sh raw.txt > new.text

The above is my current usage steps. How do I set up an automated task to complete this process?

I want to check it once a day. If the text content is updated, I need to run the script immediately. If there are no changes, the script is run every 3 days.

I would really appreciate if anyone can help me..

  • will the downloaded file name is the same every time as raw.text and do u have the definite interval between each check? – msp9011 Oct 17 '19 at 9:01
  • @msp9011 The download file name is fixed, so I will manually rename the file after running the script for later comparison. Checking once a day is ideal. – Matthew Oct 17 '19 at 9:08
  • how about the new.txt ? do u want to overwrite the content every time? – msp9011 Oct 17 '19 at 9:12
  • @msp9011 yes, the data will be rewritten to the new.txt file after running the script. – Matthew Oct 17 '19 at 9:18
1

Your question is a bit vague in what you require.

If you only want to check for any differences, use cmp -s oldfile newfile. That will exit with a true status if the files are the same:

if cmp -s oldfile newfile
then echo files are the same
else echo files are different
     ./text_processing.sh ...
fi

If you want to check that the file size has increased, then you can use wc -c (for character count), wc -l (for line count), or stat --format=%s which directly uses the metadata about the file to give the total size in bytes.

if [ $(wc -c oldfile) -lt $(wc -c newfile) ]
then echo oldfile is smaller
     ./text_processing.sh ...
fi

$( ... ) runs the enclosed command and substitutes its output into the command line. [ ... -lt ... ] tests whether the first argument is less than the second argument.

If you specifically want to check that a line has been added, then the best strategy is probably to first sort both files, and then use comm to filter out common lines:

sort -o oldfile.sorted oldfile
sort -o newfile.sorted newfile
if [ $(comm -13 oldfile.sorted newfile.sorted | wc -l) -gt 0 ]
then echo 'line(s) only found in newfile'
     ./text_processing.sh ...
fi

comm -13 matches lines from both files. Normally every line is output, with an indent to indiciate whether the line only occurs in the first file, or only in the second file, or in both. With -13 the lines from only the first file and the common lines are suppressed, so only those lines occuring in the second file is output. That is piped into wc -l which counts the lines, and that is tested to be greater than 0.

Note that changed lines will be represented by one line only in the first file, and another line only in the second file.

|improve this answer|||||
  • Your answer is very helpful to me, thank! Linux command tools are really many, cool! – Matthew Oct 17 '19 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.