Sponsored Content
Top Forums Shell Programming and Scripting Get only domain from url file bind Post 302960959 by Don Cragun on Sunday 22nd of November 2015 02:00:00 PM
Old 11-22-2015
Back in post #14 I showed you a way to get rid of lots of extra processes you are invoking for each line you read. In post #16 you said my suggestion improved performance of your script by a factor of 5. But, in your latest script in post #24 in this thread, you have thrown away that improvement and added several more duplicated invocations of geoiplookup piped through various combinations of awk, cut, awk again, and tr. Why? If you would have taken that earlier advice and applied it to the geoiplookup invocations as well, you would easily get an improvement by at least a factor of 10.

Your script in post #24 has two distinct parts. The 1st 49 lines process the stuff we have been talking about in this thread. The last line processes a file that is not mentioned in the rest of the script. Why not move that command to the start of your script and run it asynchronously (and add a wait at the end of your script if your script should not be allowed to terminate before that command completes)?

How many cores can you use on your system? If you want to run thirty copies of your script simultaneously, you will probably be fighting against yourself if you don't have at least thirty cores at your disposal.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

2. Shell Programming and Scripting

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

3. UNIX for Dummies Questions & Answers

ReDirecting a URL to another URL - Linux

Hello, I need to redirect an existing URL, how can i do that? There's a current web address to a GUI that I have to redirect to another webaddress. Does anyone know how to do this? This is on Unix boxes Linux. example: https://m45.testing.address.net/host.php make it so the... (3 Replies)
Discussion started by: SkySmart
3 Replies

4. Windows & DOS: Issues & Discussions

How to: Linux BOX in Windows Domain (w/out joining the domain)

Dear Expert, i have linux box that is running in the windows domain, BUT did not being a member of the domain. as I am not the System Administrator so I have no control on the server in the network, such as modify dns entry , add the linux box in AD and domain record and so on that relevant. ... (2 Replies)
Discussion started by: regmaster
2 Replies

5. Web Development

Regex to rewrite URL to another URL based on HTTP_HOST?

I am trying to find a way to test some code, but I need to rewrite a specific URL only from a specific HTTP_HOST The call goes out to http://SUB.DOMAIN.COM/showAssignment/7bde10b45efdd7a97629ef2fe01f7303/jsmodule/Nevow.Athena The ID in the middle is always random due to the cookie. I... (5 Replies)
Discussion started by: EXT3FSCK
5 Replies

6. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Here is what I have so far: find . -name "*php*" -or -name "*htm*" | xargs grep -i iframe | awk -F'"' '/<iframe*/{gsub(/.\*iframe>/,"\"");print $2}' Here is an example content of a PHP or HTM(HTML) file: <iframe src="http://ADDRESS_1/?click=5BBB08\" width=1 height=1... (18 Replies)
Discussion started by: striker4o
18 Replies

7. Shell Programming and Scripting

Hit multiple URL from a text file and store result in other test file

Hi, I have a problem where i have to hit multiple URL that are stored in a text file (input.txt) and save their output in different text file (output.txt) somewhat like : cat input.txt http://192.168.21.20:8080/PPUPS/international?NUmber=917875446856... (3 Replies)
Discussion started by: mukulverma2408
3 Replies

8. Shell Programming and Scripting

Reading URL using Mechanize and dump all the contents of the URL to a file

Hello, Am very new to perl , please help me here !! I need help in reading a URL from command line using PERL:: Mechanize and needs all the contents from the URL to get into a file. below is the script which i have written so far , #!/usr/bin/perl use LWP::UserAgent; use... (2 Replies)
Discussion started by: scott_cog
2 Replies

9. UNIX for Dummies Questions & Answers

Putting the colon infront of the URL domain

I have a file like this: http://hello.com www.examplecom computer Company I wanted to keep dot (.) infront of com. to make the file like this http://hello.com www.example.com computer Company I applied this expression sed -r 's/com/.com/g'but what I get is: http://hello.com ... (4 Replies)
Discussion started by: csim_mohan
4 Replies

10. UNIX for Dummies Questions & Answers

Extracting URL with domain

I have a file like this: http://article.wn.com/view/2010/11/26/IV_drug_policy_feels_HIV_patients_Red_Cross/ http://aidsjournal.com/,www.cfpa.org.cn/page1/page2 , www.youtube.com http://seattletimes.nwsource.com/html/jerrybrewer/2013517803_brewer25.html... (1 Reply)
Discussion started by: csim_mohan
1 Replies
SLEEP(1)						    BSD General Commands Manual 						  SLEEP(1)

NAME
sleep -- suspend execution for an interval of time SYNOPSIS
sleep seconds DESCRIPTION
The sleep command suspends execution for a minimum of seconds. If the sleep command receives a signal, it takes the standard action. IMPLEMENTATION NOTES
The SIGALRM signal is not handled specially by this implementation. The sleep command will accept and honor a non-integer number of specified seconds (with a '.' character as a decimal point). This is a non- portable extension, and its use will nearly guarantee that a shell script will not execute properly on another system. EXIT STATUS
The sleep utility exits 0 on success, and >0 if an error occurs. EXAMPLES
To schedule the execution of a command for x number seconds later (with csh(1)): (sleep 1800; sh command_file >& errors)& This incantation would wait a half hour before running the script command_file. (See the at(1) utility.) To reiteratively run a command (with the csh(1)): while (1) if (! -r zzz.rawdata) then sleep 300 else foreach i (`ls *.rawdata`) sleep 70 awk -f collapse_data $i >> results end break endif end The scenario for a script such as this might be: a program currently running is taking longer than expected to process a series of files, and it would be nice to have another program start processing the files created by the first program as soon as it is finished (when zzz.rawdata is created). The script checks every five minutes for the file zzz.rawdata, when the file is found, then another portion processing is done courteously by sleeping for 70 seconds in between each awk job. SEE ALSO
nanosleep(2), sleep(3) STANDARDS
The sleep command is expected to be IEEE Std 1003.2 (``POSIX.2'') compatible. HISTORY
A sleep command appeared in Version 4 AT&T UNIX. BSD
April 18, 1994 BSD
All times are GMT -4. The time now is 06:27 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy