You have been my saviour again Frank and the first script worked for me!!
I am yet to test the script part due to some meetings coming up ahead. Just can't express words when people provide so much help so quickly!!
I will post complete information on this thread shortly for all other newbies like me
Cheers,
Andy
---------- Post updated 02-10-11 at 11:23 AM ---------- Previous update was 02-09-11 at 06:27 PM ----------
Ok, following is what worked for me after MUCH required help from Franklin!!
I used the same script provided by Franklin to get my URLs filtered -
PHP Code:
sed -n 's!.*service=\(http://[^/]*/[^/]*/\).*!\1!p' file
However, I still haven't tested the script part provided by Franklin yet and I will post the output later.
Another problem I faced while trying to filter output received after using Franklin's script was I had few URLs with BIG strings with special characters and had to use following to get rid of them. (To get rid of &, ? and ' ' basically AND have them sorted)
PHP Code:
cat old.file | awk -F \& '{print $1}' | awk -F \? '{print $1}' |awk -F ' ' '{print $1}'| sort -u > output.txt
Thanks guys for all your support and now I have another issue cropping up which I will post in the next post since otherwise this post will be way too big.
Cheers,
Andy
---------- Post updated at 11:28 AM ---------- Previous update was at 11:23 AM ----------
One more problem I have is while trying to remove duplicate lines, I need to treat lower and upper cases in URLs carefully as below -
For following duplicate lines, I need to have only two URLs since currently they are all being treated as UNIQUE URLs.
(note: Separate IPs don't matter since I am only concerned with lower and upper case letters)
PHP Code:
56.555.72.69/crm_ababcdves/
81.745.42.59/CRM_Ababcdves/
38.475.62.19/squitv3/
92.625.42.89/Squitv3/
37.288.30.12/cview/
63.598.30.89/Cview/
85.048.30.52/CView/
So final output should be -
PHP Code:
56.555.72.69/crm_ababcdves/
38.475.62.19/squitv3/
37.288.30.12/cview/
Now if someone can help me with this, that would be really great since I am a newbie on these things so far though getting better since past few days.
Cheers,
Andy