faster way to loop? Post: 302072957

Sponsored Content

Top Forums UNIX for Advanced & Expert Users faster way to loop? Post 302072957 by tads98 on Tuesday 9th of May 2006 06:46:15 AM

05-09-2006

Registered User

faster way to loop?

Sample Log file

IP.address Date&TimeStamp GET/POST URL ETC
123.45.67.89 MMDDYYYYHHMM GET myURL http://ABC.com
123.45.67.90 MMDDYYYYHHMM GET myURL http://XYZ.com

I have a very huge web server log file (about 1.3GB) that contains entries like the one above. I need to get the last entries of all the different IPs that has myURL in it? Is there a quick way of looping? My idea was

# Get all the Unique IP addresses and then proceed to check each
cat weblog | awk '{print $1} > ip.list

for i in `cat ip.list`
do
cat weblog | grep $i | grep myURL > lastpages.lis
done

each day has around 3000+ unique IP entries and a day's log is about 48MB. with this process, it takes around 30 mins to process a days worth of data. is there a faster way to do this?

tads98

View Public Profile for tads98

Find all posts by tads98

9 More Discussions You Might Find Interesting

1. IP Networking

Mandrake should be faster.

For some reason 8.1 Mandrake Linux seems much slower than Windows 2000 with my cable modem. DSL reports test says they conferable speed with Windows2 though. This is consistant slow with both of my boxes, at the same time. Linux used to be faster, but not with Mandrake. Any way to fix this?

2. Shell Programming and Scripting

Faster then cp ?

Hi , I need to copy every day about 35GB of files from one file system to another. Im using the cp command and its toke me about 25 min. I also tried to use dd command but its toke much more. Is there better option ? Regards.

3. UNIX for Dummies Questions & Answers

How to grep faster ?

Hi I have to grep for 2000 strings in a file one after the other.Say the file name is Snxx.out which has these strings. I have to search for all the strings in the file Snxx.out one after the other. What is the fastest way to do it ?? Note:The current grep process is taking lot of time per...

4. UNIX for Dummies Questions & Answers

Which command will be faster? y?

i)wc -c/etc/passwd|awk'{print $1}' ii)ls -al/etc/passwd|awk'{print $5}'

5. UNIX for Dummies Questions & Answers

Why is RAID0 faster?

I have read anecdotes about people installing RAID0 (RAID - Wikipedia, the free encyclopedia) on some of their machines because it gives a performance boost. Because bandwidth on the motherboard is limited, can someone explain exactly why it should be faster?

6. Shell Programming and Scripting

Make script faster

Hi all, In bash scripting, I use to read files: cat $file | while read line; do ... doneHowever, it's a very slow way to read file line by line. E.g. In a file that has 3 columns, and less than 400 rows, like this: I run next script: cat $line | while read line; do ## Reads each...

7. Shell Programming and Scripting

need to process for loop faster

I have the following code running against a file. The file can have upwards of 10000 lines. problem is, the for loop takes a while to go through all those lines. is there a faster way to go about it? for line in `grep -P "${MONTH} ${DAY}," file | ${AWK} -F" " '{print $4}' | awk -F":"...

8. UNIX for Dummies Questions & Answers

Which system is faster?

i'm trying to decide if to move operations from one of these hosts to the other. but i cant decide which one of them is the most powerful. each host has 8 cpus. HOSTA processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Intel(R) Xeon(R) CPU ...

9. Shell Programming and Scripting

How to make faster loop in multiple directories?

Hello, I am under Ubuntu 18.04 Bionic. I have one shell script run.sh (which is out of my topic) to run files under multiple directories and one file to control all processes running under those directories (control.sh). I set a cronjob task to check each of them with two minutes of intervals....

LEARN ABOUT NETBSD

urlphy

URL(4)							   BSD Kernel Interfaces Manual 						    URL(4)

NAME

     url -- Realtek RTL8150L USB Ethernet driver

SYNOPSIS

     url*    at uhub?
     urlphy* at mii?

DESCRIPTION

     The url driver provides support for USB Ethernet adapters based on the Realtek RTL8150L USB-ether bridge chip.

     The url driver supports the following adapters:

	   CompUSA USBKR100
	   GreenHouse GH-USB100B
	   Melco Inc. LUA-KTX
	   Sitecom LN013

     For more information on configuring this device, see ifconfig(8).

DIAGNOSTICS

     url%d: watchdog timeout  A packet was queued for transmission and a transmit command was issued, however the device failed to acknowledge the
     transmission before a timeout expired.

     url%d: no memory for rx list  The driver failed to allocate an mbuf for the receiver ring.

SEE ALSO

     arp(4), mii(4), netintro(4), usb(4), ifconfig(8)

HISTORY

     The url device driver first appeared in NetBSD 1.6.

AUTHORS

     The url driver was written by Shingo WATANABE <nabe@nabechan.org>.

BSD
								September 18, 2006							       BSD