Sample Log file
IP.address Date&TimeStamp GET/POST URL ETC
123.45.67.89 MMDDYYYYHHMM GET myURL
http://ABC.com
123.45.67.90 MMDDYYYYHHMM GET myURL
http://XYZ.com
I have a very huge web server log file (about 1.3GB) that contains entries like the one above. I need to get the last entries of all the different IPs that has myURL in it? Is there a quick way of looping? My idea was
# Get all the Unique IP addresses and then proceed to check each
cat weblog | awk '{print $1} > ip.list
for i in `cat ip.list`
do
cat weblog | grep $i | grep myURL > lastpages.lis
done
each day has around 3000+ unique IP entries and a day's log is about 48MB. with this process, it takes around 30 mins to process a days worth of data. is there a faster way to do this?