Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Querying a website at random bunches simultaneously
Top Forums Shell Programming and Scripting Querying a website at random bunches simultaneously Post 302750857 by jim mcnamara on Wednesday 2nd of January 2013 04:17:46 PM
Old 01-02-2013
Why random?

Let's assume your site can handle 10 at once (you decide on a number, 10 is just a choice)
You want parallelism, this does exactly that.
Use this as a template:

Code:
cnt=1
for ((i=0; i < ${#array[*]}; i++))
do
	  (curl -s "http://www.website.com/api?stock=${array[i]}"|sed 's/</\n</g' |
                  sed '/data=/!d; s/ data=/=/g; s/\/>/; /g; s/</GF_/g'  ~/stocks/${array[i]}.txt
	     echo "${array[i]},$(date +%Y-%m-%d), $GF_open, $GF_high, $GF_low, $GF_last, $GF_volume" >> somefile) &

              cnt=$(( $cnt + 1 ))

          [ $(( $cnt % 10 )) -eq 0 ]  && wait

done
wait   # this one is important

I altered the code to stop writing to the terminal, because the & places the child process in the background. The loop waits until all children are done - but it lets 10 of them fire off, then waits in bunches of 10.

This is the most efficient way to create parallel code for what you are doing.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Connecting to DB2 database on Mainframe and Querying DB

I am trying to develop a script that can connect to a DB2 database mainframe and Query the database and display the results. I've been researching, but I have yet to find a definitive solution where I can enter in the Mainframe DB2 address db2:// and query the database and return the results. If... (2 Replies)
Discussion started by: developncode
2 Replies

2. Shell Programming and Scripting

Querying database from unix

Hi, I have a shell script to query the database to get the statistics of tables like sum,max,min,etc of all numeric columns,max and min of date columns and length of varchar columns for each and every table. There are nearly 1600 tables. My script queries few tables like about 100 tables and... (3 Replies)
Discussion started by: ragavhere
3 Replies

3. HP-UX

Querying HP VA array

Hello, all... I've got an older HP-UX 11i machine hooked up to an HP VA array. I've never worked with VA arrays, so I was wondering if anyone could point me in the direction of some documentation on querying and manipulating VAs (HP VAs for Dummies or something). I'm used to EMC arrays, so... (2 Replies)
Discussion started by: kknigga
2 Replies

4. AIX

Servers still querying old DNS server?

Hello, I've created new DNS servers and changed all of the clients /etc/resolv.conf to point to them, but when I check the old DNS logs, I see that the clients are still querying it. Does anybody know why? thanks, (2 Replies)
Discussion started by: ctcuser
2 Replies

5. Solaris

commands hanging when querying hardware

Hi all Got another strange one. If I try to enquire about the hardware, the command hangs implying Ive got a hardware issue. So, if I execute :- iostat -en sysdef - ( stops at the devices part ) format cfgadm -al Anything that searches the devices, then the command hangs. ... (4 Replies)
Discussion started by: sbk1972
4 Replies

6. Red Hat

Error in querying file information

Hi, We have one application running where there is Linux Machine. For some some process I found the following message came, error 5.8/116455-02/116455-02.zip Error in querying file information: Permission denied :confused: If anybody has any idea for this please let me... (2 Replies)
Discussion started by: sptr
2 Replies

7. Ubuntu

expect script for random password and random commands

Hi I am new to expect. Please if any one can help on my issue its really appreciable. here is my issue: I want expect script for random passwords and random commands generation. please can anyone help me? Many Thanks in advance (0 Replies)
Discussion started by: vanid
0 Replies

8. Shell Programming and Scripting

Generic script for Querying a DB

I have a few databases that i need to get some basic output from... so I'll be returning 1 or 2 records from a variety of different databases and the input queries will always differ. What im looking for is a generic script that I can use across of the databases. Theory being i have: ... (2 Replies)
Discussion started by: atelford
2 Replies

9. Shell Programming and Scripting

Need to generate a file with random data. /dev/[u]random doesn't exist.

Need to use dd to generate a large file from a sample file of random data. This is because I don't have /dev/urandom. I create a named pipe then: dd if=mynamed.fifo do=myfile.fifo bs=1024 count=1024 but when I cat a file to the fifo that's 1024 random bytes: cat randomfile.txt >... (7 Replies)
Discussion started by: Devyn
7 Replies

10. Shell Programming and Scripting

Querying v$session table

Hi All, I'm using below code entry_data_control=`sqlplus ${sysuser}/${syspwd} << data_control whenever sqlerror exit sql.sqlcode; whenever oserror exit failure; INSERT INTO LSHADMIN.STUDY_LEVEL_TABLE(SESSION_SID,SESSION_SERIAL_NO,FILE_NAME) ... (3 Replies)
Discussion started by: Pratiksha Mehra
3 Replies

LEARN ABOUT DEBIAN

finance::quote::cdnfundlibrary

Finance::Quote::Cdnfundlibrary(3pm)			User Contributed Perl Documentation		       Finance::Quote::Cdnfundlibrary(3pm)

NAME

       Finance::Quote::Cdnfundlibrary  - Obtain mutual fund prices from www.fundlibrary.com

SYNOPSIS

	   use Finance::Quote;

	   $q = Finance::Quote->new;

	   %stockinfo = $q->fetch("canadamutual","fundlib-code"); # Can
       failover to other methods
	   %stockinfo = $q->fetch("fundlibrary","fundlib-code"); # Use this
       module only.

	   # NOTE: currently no failover methods exist for canadamutual

DESCRIPTION

       This module obtains information about Canadian Mutual Fund prices from www.fundlibrary.com.  The information source "canadamutual" can be
       used if the source of prices is irrelevant, and "fundlibrary" if you specifically want to use www.fundlibrary.com.

FUNDLIB-CODE
       In Canada a mutual fund does not have a unique global symbol identifier.

       This module uses an id that represents the mutual fund on an id used by www.fundlibrary.com.  There is no easy way of fetching the id
       except to jump onto the fundlibrary website, look up the fund and view the url for clues to its id number.

LABELS RETURNED

       Information available from fundlibrary may include the following labels:

       exchange method link source name currency yield last nav price.	The link label will be a url location for a one page snapshot that
       fundlibrary provides on the fund.

SEE ALSO

       Fundlibrary website - http://www.fundlibrary.com/

       Finance::Quote

perl v5.14.2							    2012-05-06				       Finance::Quote::Cdnfundlibrary(3pm)
All times are GMT -4. The time now is 02:05 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy