Querying a website at random bunches simultaneously Post: 302750857

Sponsored Content

Top Forums Shell Programming and Scripting Querying a website at random bunches simultaneously Post 302750857 by jim mcnamara on Wednesday 2nd of January 2013 04:17:46 PM

01-02-2013

Registered User

Why random?

Let's assume your site can handle 10 at once (you decide on a number, 10 is just a choice)
You want parallelism, this does exactly that.
Use this as a template:

Code:

cnt=1
for ((i=0; i < ${#array[*]}; i++))
do
	  (curl -s "http://www.website.com/api?stock=${array[i]}"|sed 's/</\n</g' |
                  sed '/data=/!d; s/ data=/=/g; s/\/>/; /g; s/</GF_/g'  ~/stocks/${array[i]}.txt
	     echo "${array[i]},$(date +%Y-%m-%d), $GF_open, $GF_high, $GF_low, $GF_last, $GF_volume" >> somefile) &

              cnt=$(( $cnt + 1 ))

          [ $(( $cnt % 10 )) -eq 0 ]  && wait

done
wait   # this one is important

I altered the code to stop writing to the terminal, because the & places the child process in the background. The loop waits until all children are done - but it lets 10 of them fire off, then waits in bunches of 10.

This is the most efficient way to create parallel code for what you are doing.

jim mcnamara

View Public Profile for jim mcnamara

Find all posts by jim mcnamara

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Connecting to DB2 database on Mainframe and Querying DB

I am trying to develop a script that can connect to a DB2 database mainframe and Query the database and display the results. I've been researching, but I have yet to find a definitive solution where I can enter in the Mainframe DB2 address db2:// and query the database and return the results. If...

2. Shell Programming and Scripting

Querying database from unix

Hi, I have a shell script to query the database to get the statistics of tables like sum,max,min,etc of all numeric columns,max and min of date columns and length of varchar columns for each and every table. There are nearly 1600 tables. My script queries few tables like about 100 tables and...

3. HP-UX

Querying HP VA array

Hello, all... I've got an older HP-UX 11i machine hooked up to an HP VA array. I've never worked with VA arrays, so I was wondering if anyone could point me in the direction of some documentation on querying and manipulating VAs (HP VAs for Dummies or something). I'm used to EMC arrays, so...

4. AIX

Servers still querying old DNS server?

Hello, I've created new DNS servers and changed all of the clients /etc/resolv.conf to point to them, but when I check the old DNS logs, I see that the clients are still querying it. Does anybody know why? thanks,

5. Solaris

commands hanging when querying hardware

Hi all Got another strange one. If I try to enquire about the hardware, the command hangs implying Ive got a hardware issue. So, if I execute :- iostat -en sysdef - ( stops at the devices part ) format cfgadm -al Anything that searches the devices, then the command hangs. ...

6. Red Hat

Error in querying file information

Hi, We have one application running where there is Linux Machine. For some some process I found the following message came, error 5.8/116455-02/116455-02.zip Error in querying file information: Permission denied :confused: If anybody has any idea for this please let me...

7. Ubuntu

expect script for random password and random commands

Hi I am new to expect. Please if any one can help on my issue its really appreciable. here is my issue: I want expect script for random passwords and random commands generation. please can anyone help me? Many Thanks in advance

8. Shell Programming and Scripting

Generic script for Querying a DB

I have a few databases that i need to get some basic output from... so I'll be returning 1 or 2 records from a variety of different databases and the input queries will always differ. What im looking for is a generic script that I can use across of the databases. Theory being i have: ...

9. Shell Programming and Scripting

Need to generate a file with random data. /dev/[u]random doesn't exist.

Need to use dd to generate a large file from a sample file of random data. This is because I don't have /dev/urandom. I create a named pipe then: dd if=mynamed.fifo do=myfile.fifo bs=1024 count=1024 but when I cat a file to the fifo that's 1024 random bytes: cat randomfile.txt >...

10. Shell Programming and Scripting

Querying v$session table

Hi All, I'm using below code entry_data_control=`sqlplus ${sysuser}/${syspwd} << data_control whenever sqlerror exit sql.sqlcode; whenever oserror exit failure; INSERT INTO LSHADMIN.STUDY_LEVEL_TABLE(SESSION_SID,SESSION_SERIAL_NO,FILE_NAME) ...

LEARN ABOUT MOJAVE

curlinfo_protocol

CURLINFO_PROTOCOL(3)					     curl_easy_getinfo options					      CURLINFO_PROTOCOL(3)

NAME

       CURLINFO_PROTOCOL - get the protocol used in the connection

SYNOPSIS

       #include <curl/curl.h>

       CURLcode curl_easy_getinfo(CURL *handle, CURLINFO_PROTOCOL, long *p);

DESCRIPTION

       Pass  a	pointer to a long to receive the version used in the last http connection.  The returned value will be one of the CURLPROTO_* val-
       ues.

PROTOCOLS

       All

EXAMPLE

       CURL *curl = curl_easy_init();
       if(curl) {
	 CURLcode res;
	 curl_easy_setopt(curl, CURLOPT_URL, "http://example.com");
	 res = curl_easy_perform(curl);
	 if(res == CURLE_OK) {
	   long protocol;
	   curl_easy_getinfo(curl, CURLINFO_PROTOCOL, &protocol);
	 }
	 curl_easy_cleanup(curl);
       }

AVAILABILITY

       Added in 7.52.0

RETURN VALUE

       Returns CURLE_OK if the option is supported, and CURLE_UNKNOWN_OPTION if not.

SEE ALSO

       CURLINFO_RESPONSE_CODE(3), curl_easy_getinfo(3), curl_easy_setopt(3),

libcurl 7.54.0							 November 23, 2016					      CURLINFO_PROTOCOL(3)

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Connecting to DB2 database on Mainframe and Querying DB

Discussion started by: developncode

2. Shell Programming and Scripting

Querying database from unix

Discussion started by: ragavhere

3. HP-UX

Querying HP VA array

Discussion started by: kknigga

4. AIX

Servers still querying old DNS server?

Discussion started by: ctcuser