5

I have log files which are named in the following manner:

localhost_log_file.2017-09-01.txt
localhost_log_file.2017-09-02.txt
....

localhost_log_file.2017-10-30.txt

In other words, each log file has the following form:

localhost_log_file.YYYY-MM-DD.txt

I want to cat all the log files taken between the dates of 2017-09-03 and 2017-10-08, i.e. every log file starting from localhost_log_file.2017-09-03.txt through localhost_log_file.2017-10-08.txt.

Currently what I do is produce three intermediate files by separately executing each of the following commands:

for((i=3;i<=9;i++)) do cat localhost_log_file.2017-09-0$i.txt >> log1.csv ; done;

for((i=10;i<=30;i++)) do cat localhost_log_file.2017-09-$i.txt >> log2.csv ; done;

for((i=1;i<=8;i++)) do cat localhost_log_file.2017-10-0$i.txt >> log3.csv ; done;

Then I combine the intermediate files as follows:

cat log1.csv log2.csv log3.csv> totallog.csv

Is there a better way to do this?

13

You can nest brace expansion.

Short and sweet:

cat localhost_log_file.2017-{09-{03..30},10-{01..08}}.txt > totallog.csv

Note that some systems such as macOS use an older version of Bash where this doesn't work, as the leading zeros are stripped from brace expansion integer sequences. For Linux this works fine.

1
  • Small comment: Using brace expansion work really for this specific task (and is very concise!), but using the date function generalizes better and is much more flexible, e.g. it will automatically handle varying month lengths, leap years, etc.
    – igal
    Nov 9 '17 at 18:20
8

How about using the date utility to iterate through the range of dates you're interested in? Here is what that might look like for your example:

# Set the date counter to the start date
d=2017-09-03

# Iterate until we reach the end date (i.e. the date after the last date we want)
while [ "$d" != 2017-10-09 ]; do

    # cat each file
    cat "localhost_log_file.${d}.txt";

    # Increment the date counter
    d="$(date -I -d "$d + 1 day")";

done

See this for more information:

Alternatively, you can pass the results of the loop to the cat command instead of invoking cat in the body of the loop.

Here is what that could look like using command-substitution:

d=2017-09-03
cat $(while [ "$d" != 2017-10-09 ]; do
    echo "localhost_log_file.${d}.txt";
    d="$(date -I -d "$d + 1 day")";
done)

And here is the same thing using a pipe and xargs:

d=2017-09-03
while [ "$d" != 2017-10-09 ]; do
    echo "localhost_log_file.${d}.txt";
    d="$(date -I -d "$d + 1 day")";
done | xargs cat
0
3

use date:

for i in {0..35}; do
     cat localhost_log_file.$(date +%F -d "2017-09-03 + $i day").txt
done > totallog.csv
1
  • Similar, but using seq instead: seq 10 | xargs -I{} date +%F -d "2017-09-03 + {} day" | xargs -I{} cat "localhost_log_file.{}.txt"
    – wvxvw
    Nov 9 '17 at 14:09
1

Better is subjective, but you can use brace expansion for both

cat localhost_log_file.2017-09-{03..09}.txt > log1.csv

and

cat log{1..3} > totallog.csv

Assuming you wanted to keep the log?.csv files

4
  • Try echo {03..09} and see what you get. The leading zeros are not preserved.
    – Wildcard
    Nov 9 '17 at 5:02
  • 1
    Already checked on my machine, worked 100% in my bash shell with cat log{02..10}.csv and so does your echo......
    – bu5hman
    Nov 9 '17 at 5:04
  • Aha, right you are. Now I've undeleted my answer and added the caveat about bash versions.
    – Wildcard
    Nov 9 '17 at 5:07
  • I like your solution but I only assumed that the OP split the month of 09 because he wanted to keep the data separate. Other than that brace expansion is more elegant.
    – bu5hman
    Nov 9 '17 at 5:08
0

Use this command:

ls | sort | while read i; do  if ([ "$i" \> "localhost_log_file.2017-09-03.txt" ] || [ "$i" == "localhost_log_file.2017-09-03.txt" ]) && ([ "$i" \< "localhost_log_file.2017-10-08.txt" ] || [ "$i" == "localhost_log_file.2017-10-08.txt" ]); then cat $i >> totallog.csv ; fi ; done

You have to run it like this

ls | sort | while read i hit enter
do hit enter
if ([ "$i" \> "localhost_log_file.2017-09-03.txt" ] || [ "$i" == "localhost_log_file.2017-09-03.txt" ]) && ([ "$i" \< "localhost_log_file.2017-10-08.txt" ] || [ "$i" == "localhost_log_file.2017-10-08.txt" ]) hit enter then hit enter
cat $i >> totallog.csv hit enter
fi hit enter
done hit enter

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.