Shell Programming and Scripting

Hi

The raw data comes from a website scrape and contains 12000 products name but no brands names, e.g.

Filippo Berio Mild & Light Olive Oil (500ml)

What I've done initially is to the get the scrape program to take the first word of each product to use as the brand name. This is good for about 90% of brands but of course only works for those which are only a single word.

I then use a simple sed script to pick out the incorrect brands e.g.

sed -i "s/^7/7 Up/g" brands.txt
sed -i "s/^Ainsley/Ainsley Harriott/g" brands.txt
sed -i "s/^Air/Air Wick/g" brands.txt
sed -i "s/^Alfa/Alfa One/g" brands.txt
sed -i "s/^All/All About Shine/g" brands.txt
sed -i "s/^Alta/Alta Italia/g" brands.txt
sed -i "s/^Ambi/Ambi Pur/g" brands.txt
sed -i "s/^Angel/Angel Delight/g" brands.txt

However this again only fixes %70 of brands as some brands have the same first word, e.g.

John West
John Frieda

I can pull out a list of all the brands if needed. I've attached just the product list file for now but will produce and attach the brand file later today. What makes this a little bit more complicated is that this list will be updated weekly so new brands are constantly being added.

I can get my brand list to a point where it will only take me 10 - 15 minutes of manual editing so it's not the end of the world :-)

Many thanks for you kind help on this little problem...

The best way of doing this would be to find a way to extract only the brand name from the web site.
Otherwise, you have to build a list that will contains all the "multi word" brand name so that we can then setup a script to parse it with a pseudo code like :

for all brand in multi word band name list
set the : separator at the right place
for all other
set the : separator just after the first word

If you give more clue about the way you initially generate the initial file, maybe it can help to directly separate the brand & product fields at generation step.

The file is initially generated using a piece of web scraping software, I've just had a look at the web pages I do the scrape from, it looks like I can produce a scrape of all the brand names.

so if you can generate the list of the brand name only, and then the other file containing everything, you can then setup the separator at the right place.
Can you upload the file containing only the brand name ?

I'm afraid automatically generating the brand list is harder than I initially thought. I will still continue to look at this. The support forum for the website for the scrape software is currently down and I need to ask a question there. Many thanks for your help :-)

some line appear more than once in your file :

Here you can see at which line number the duplicate occure :

Code:

$ sort tst | uniq -d | while read a
> do
> cat -n tst | grep "$a"
> done
  6776  Bloo Acticlean Cistern Blocks Citrus (2)
  6777  Bloo Acticlean Cistern Blocks Citrus (2)
  7704  Tesco Premium Supermeat Variety Pack (6x400g)
  7981  Tesco Premium Supermeat Variety Pack (6x400g)
  7711  Winalot Classics in Jelly Variety Pack (6x400g)
  7712  Winalot Classics in Jelly Variety Pack (6x400g)

---------- Post updated at 03:24 PM ---------- Previous update was at 03:21 PM ----------

Code:

$ sort tst | uniq -d | while read a
> do
> grep -n "$a" tst
> done
6776:Bloo Acticlean Cistern Blocks Citrus (2)
6777:Bloo Acticlean Cistern Blocks Citrus (2)
7704:Tesco Premium Supermeat Variety Pack (6x400g)
7981:Tesco Premium Supermeat Variety Pack (6x400g)
7711:Winalot Classics in Jelly Variety Pack (6x400g)
7712:Winalot Classics in Jelly Variety Pack (6x400g)

Wow didn't realise that, the website that I scraped the data from also has the duplicates! They are a big company with lots of developers you'd think they would have noticed :-)