0

My OS: RHEL7.X

I've a csv file with 3 columns, like so -

#######Category: Fruit
Berries are fruit,04415bef-8c82-4672-848a-9ac80035040a,"Blah/Blah  And/More/Blah"
Apple is a fruit,073c6bf3-74d2-4c20-b8a7-82625cffe256,"Blah1.0/Blah  And/More/Blah2.0"
Orange is a fruit,083dddd7-2df4-422a-a07b-00419ccf98fd,"Blah2.0/Blah  And/More/Blah2.1"
---
---
#######Category: Vegetable
Sprouts are vegetable,84e0a0d8-0c0e-448b-9ad6-21178dac8b86,"Blah/Blah  And/lot More/Blah"
Ginger is a vegetable,abb5fee2-a588-45a7-a8c0-9cd87d60a6d2,"Blah1.0/Blah  And/whole lot more/Blah2.0"
onions is a vegetable,baeee94a-6447-4c80-bddf-313e0fc144a7,"Blah2.0/Blah  And/Lot More/Blah2.1"
---
---
#######Category: Mixed
Tomatoes can be called a mix,fe31c693-cdf4-4171-9c80-7a297d4bdb96,"Blah/Blah And/More/Blah"
cucumber is a joke,d3540c7f-fea5-4e64-87df-c18fdb0b7ff3,"Blah1.0/Blah  And/More/Blah2.0"
Spinach is LOL,da816135-9852-4067-8780-4e504dc8084b,"Blah2.0/Blah  And/Even More/Blah2.1"
---
---

• Above data is from one file

• A new csv is generated everyday with additional entries under each header

• Header example: #######Category: Fruit

  • "---" in example above, represents more entries. There could be 1000s of entries

• The header (that starts with "#") doesn't change from day to day.

• New data scraped and dumped into a file everyday. Let's say today's data is dumped into file: /path/to/Today.txt

• 24 hours old data was dumped into a file: /path/to/Yesterday.txt

• Second column of this csv has unique string without spaces. Let's call it uid

• Not important but just to mention - There can be more headers and thousands of entries under each header.

Goal-1: To find new entries in file Today.txt that doesn't exist in Yesterday.txt using uid (second column)

This bash code does that:

TodayFile=/path/to/Today.txt
YstdayFile=/path/to/Yesterday.txt

sed -e '/^\s*$/ d' -e '/^#/ d' $TodayFile | awk -F , '{print $2}' | sed '/^$/d' | while read uid; do

        if [[ -z $(grep $uid $YstdayFile) ]];then
            grep $uid $TodayFile >> NewEntries.txt
        fi
done

And when we open the newly generated NewEntries.txt, we get new entries that doesn't exist in /path/to/Yesterday.txt

cat -n NewEntries.txt

1 new entry
2 new entry
3 new entry
---
---

But this output is not good enough

The Goal is: To find new entries And keep the headers (under which they belong) intact and in Order

Simulated Final output ought to look like:

#######Category: Fruit
new entry
new entry
---
---
#######Category: Vegetable
new entry
new entry
---
---
#######Category: Mixed
new entry
new entry
---
---

New entry, if any, should appear under the relevant headers

How can I achieve that using shell/bash script...Any suggestions?

And is there an easier solution than running a loop?

1 Answer 1

2

Assumptions:

  • all of Yesterday.txt is included in Today.txt
  • first column does not contain any embedded commas

In this particular case, since you're already using awk, we can write the entire process in a single awk script. This reduces the complextiy of the code and should provide a noticeable improvement in performance (especially for largish files).

Setup (one new line appended on the end of each of 3 categories):

$ cat Yesterday.txt
#######Category: Fruit
Berries are fruit,04415bef-8c82-4672-848a-9ac80035040a,"Blah/Blah  And/More/Blah"
Apple is a fruit,073c6bf3-74d2-4c20-b8a7-82625cffe256,"Blah1.0/Blah  And/More/Blah2.0"
#######Category: Vegetable
Sprouts are vegetable,84e0a0d8-0c0e-448b-9ad6-21178dac8b86,"Blah/Blah  And/lot More/Blah"
Ginger is a vegetable,abb5fee2-a588-45a7-a8c0-9cd87d60a6d2,"Blah1.0/Blah  And/whole lot more/Blah2.0"
#######Category: Mixed
Tomatoes can be called a mix,fe31c693-cdf4-4171-9c80-7a297d4bdb96,"Blah/Blah And/More/Blah"
cucumber is a joke,d3540c7f-fea5-4e64-87df-c18fdb0b7ff3,"Blah1.0/Blah  And/More/Blah2.0"

$ cat Today.txt
#######Category: Fruit
Berries are fruit,04415bef-8c82-4672-848a-9ac80035040a,"Blah/Blah  And/More/Blah"
Apple is a fruit,073c6bf3-74d2-4c20-b8a7-82625cffe256,"Blah1.0/Blah  And/More/Blah2.0"
Orange is a fruit,083dddd7-2df4-422a-a07b-00419ccf98fd,"Blah2.0/Blah  And/More/Blah2.1"
#######Category: Vegetable
Sprouts are vegetable,84e0a0d8-0c0e-448b-9ad6-21178dac8b86,"Blah/Blah  And/lot More/Blah"
Ginger is a vegetable,abb5fee2-a588-45a7-a8c0-9cd87d60a6d2,"Blah1.0/Blah  And/whole lot more/Blah2.0"
onions is a vegetable,baeee94a-6447-4c80-bddf-313e0fc144a7,"Blah2.0/Blah  And/Lot More/Blah2.1"
#######Category: Mixed
Tomatoes can be called a mix,fe31c693-cdf4-4171-9c80-7a297d4bdb96,"Blah/Blah And/More/Blah"
cucumber is a joke,d3540c7f-fea5-4e64-87df-c18fdb0b7ff3,"Blah1.0/Blah  And/More/Blah2.0"
Spinach is LOL,da816135-9852-4067-8780-4e504dc8084b,"Blah2.0/Blah  And/Even More/Blah2.1"

One awk idea:

awk -F',' '
FNR==NR { seen[$2]; next }              # 1st file: save 2nd column (uid) as index in array seen[]
/^#####/ || !($2 in seen)               # 2nd file: if line starts with "#####" or 2nd field is not an index in array seen[], then print current line to stdout
' Yesterday.txt Today.txt

This generates:

#######Category: Fruit
Orange is a fruit,083dddd7-2df4-422a-a07b-00419ccf98fd,"Blah2.0/Blah  And/More/Blah2.1"
#######Category: Vegetable
onions is a vegetable,baeee94a-6447-4c80-bddf-313e0fc144a7,"Blah2.0/Blah  And/Lot More/Blah2.1"
#######Category: Mixed
Spinach is LOL,da816135-9852-4067-8780-4e504dc8084b,"Blah2.0/Blah  And/Even More/Blah2.1"
1
  • 1
    So far, works good. I didn't see any corruption and I ran this on real files with 1000s of entries. As for entries missing in Today's file but existed in yesterday, I simply flip the files in your solution and that works too. Thank you @markp-fuso Oct 12, 2023 at 22:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .