Red Hat

No problem, you're welcome.

I need some more help people and would be posting here shortly

Thanks in advance,
Andy

---------- Post updated at 12:59 PM ---------- Previous update was at 12:58 PM ----------

Ok, I must admit I have another small problem I need to tweak.
Initially I wanted only the URL portion of a line from a log file however, what I am after now is I need a little longer URL something like below.

Code:

10.155.65.5 - - [20/Jan/2011:07:41:58 +0100] "POST /cas/login?post=true&service=http://test.domain.ca:8000/psp/ppf88prd/?cmd=login%26languageCd=CAN%26userid=VP1%26pwd=TEST123 HTTP/1.1" 200 888

I need http://test.domain.ca:8000/psp/ instead of just http://test.domain.ca like below.

Can someone please help if how I do achieve that. I did try playing with a earlier script I had been given by Frank as below though unfortunately it didn't work

Code:

sed -n 's!.*service=\(http://[^:]*\):.*!\1!p' logfile > newfile

Also one more help would be to be aware of a script which would need to find a similar result from around 20 files instead of just 1 file as of now. I am doing it manually for now though need some help in this too.
The file format is something like below -

Code:

abc-accent.2011-01-20.log
abc-accent.2011-02-21.log
abc-accent.2011-03-22.log
abc-accent.2011-04-23.log
abc-accent.2011-05-24.log

Again, thanks once again for all help offered already!!!
Cheers,
Andy

---------- Post updated at 01:08 PM ---------- Previous update was at 12:59 PM ----------

Ok, I just found another script that may require some tweaking which I will do later though someone please confirm if it will suit the purpose. (Again I would be testing it in some time on a test box to be sure)
It basically searches and replaces strings while searching a certain parrtern of files in a directory we specify.

Code:

#!/bin/bash
OLD="xyz"
NEW="abc"
DPATH="/home/you/foo/*.txt"
BPATH="/home/you/bakup/foo"
TFILE="/tmp/out.tmp.$$"
[ ! -d $BPATH ] && mkdir -p $BPATH || :
for f in $DPATH
do
  if [ -f $f -a -r $f ]; then
    /bin/cp -f $f $BPATH
   sed "s/$OLD/$NEW/g" "$f" > $TFILE && mv $TFILE "$f"
  else
   echo "Error: Cannot read $f"
  fi
done
/bin/rm $TFILE

Any suggestions as always are quiet welcome.

Thanks,
Andy

Try this one:

Code:

sed -n 's!.*service=\(http://[^/]*/[^/]*/\).*!\1!p' file

You can use something like this to loop through the abs-accent log files:

Code:

ls abc-accent*.log |
while read file
do
  sed -n 's!.*service=\(http://[^/]*/[^/]*/\).*!\1!p' "$file" > URL_file
done

You have been my saviour again Frank and the first script worked for me!!

I am yet to test the script part due to some meetings coming up ahead. Just can't express words when people provide so much help so quickly!!

I will post complete information on this thread shortly for all other newbies like me

Cheers,
Andy

---------- Post updated 02-10-11 at 11:23 AM ---------- Previous update was 02-09-11 at 06:27 PM ----------

Ok, following is what worked for me after MUCH required help from Franklin!!

I used the same script provided by Franklin to get my URLs filtered -

However, I still haven't tested the script part provided by Franklin yet and I will post the output later.

Another problem I faced while trying to filter output received after using Franklin's script was I had few URLs with BIG strings with special characters and had to use following to get rid of them. (To get rid of &, ? and ' ' basically AND have them sorted)

Thanks guys for all your support and now I have another issue cropping up which I will post in the next post since otherwise this post will be way too big.

Cheers,
Andy

---------- Post updated at 11:28 AM ---------- Previous update was at 11:23 AM ----------

One more problem I have is while trying to remove duplicate lines, I need to treat lower and upper cases in URLs carefully as below -

For following duplicate lines, I need to have only two URLs since currently they are all being treated as UNIQUE URLs.
(note: Separate IPs don't matter since I am only concerned with lower and upper case letters)

So final output should be -

Now if someone can help me with this, that would be really great since I am a newbie on these things so far though getting better since past few days.

Cheers,
Andy

Is this what you're looking for?

Code:

$ cat file
56.555.72.69/crm_ababcdves/&kllk?jkjk op
81.745.42.59/CRM_Ababcdves/&kllk?jkjk op
38.475.62.19/squitv3/
92.625.42.89/Squitv3/&kllk?jkjk op
37.288.30.12/cview/
63.598.30.89/Cview/&kllk?jkjk op
85.048.30.52/CView/
$
$ awk -F"/" '{
  gsub("[ &,?].*",x)
  s=toupper($2)
}
s in a {next}
{a[s]}1' file
56.555.72.69/crm_ababcdves/
38.475.62.19/squitv3/
37.288.30.12/cview/
$

Not sure how do I run that?
It looks like a small .sh script to me.
So please tell me whether I should save it as a sh file with all duplicate entries in it to ge the final output?

Regards,
Andy

Code:

awk -F"/" '{
  gsub("[ &,?].*",x)
  s=toupper($2)
}
s in a {next}
{a[s]}1' file

This is a command to get rid of the characters &, ? and ' ' and to remove the duplicates.

The example above shows an example of the content of a file with those characters, the awk command and the final output.

You could use the command in a script or on the command line.