Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

parallel excution for 2000 files.

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting parallel excution for 2000 files.
# 1  
Old 08-02-2010
parallel excution for 2000 files.
Code:
Hi,

I have a function abc(). i am calling the function 9 times. it is working fine and completed the script execution 10 hours.


input files:
CUSTOMER_INFO_1111_12345.csv
CUSTOMER_INFO_1222_12345.csv
CUSTOMER_INFO_1322_12345.csv
CUSTOMER_INFO_1333_12345.csv
CUSTOMER_INFO_1151_12345.csv
CUSTOMER_INFO_2221_12345.csv
CUSTOMER_INFO_3333_12345.csv
CUSTOMER_INFO_4444_12345.csv

like that i have 2000 files are present inthe source directory.

Code:
abc()
{
#Calculations are added in the function
for file in `ls |egrep '(CUSTOMER_INFO_.*_'$Flag'.*csv$)'`
do
tail -1 $file1| awk '/^"9"/'  > new_$file
done
}

Flag=1
abc $p1 $p2 $p3 $p4  $Flag &
Flag=2
abc $p1 $p2 $p3 $p4  $Flag &
Flag=3
abc $p1 $p2 $p3 $p4  $Flag &
Flag=4
abc $p1 $p2 $p3 $p4  $Flag &
Flag=5
abc $p1 $p2 $p3 $p4  $Flag &
Flag=6
abc $p1 $p2 $p3 $p4  $Flag &
Flag=7
abc $p1 $p2 $p3 $p4  $Flag &
Flag=8
abc $p1 $p2 $p3 $p4  $Flag &
Flag=9
abc $p1 $p2 $p3 $p4  $Flag &

wait

echo "script completed"


Code:
Currently execution based on flag and at a time 9 files are executed. i want to run the script for all the 2000 files at the same time. 
when i run the scipt all the 2000 files get start for the excution. all the files are executed in parallel.
i am unable to execute files at a time.

# 2  
Old 08-02-2010
Since these are triggered from the script paralley, Check the ulimit for sub background processes in your system. If ulimit value is less than 2000, I dont think so its possibkle for you to execute.
# 3  
Old 08-02-2010
Code:
ulimit -a

time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        4194304
memory(kbytes)       unlimited
coredump(blocks)     10485755
nofiles(descriptors) unlimited


as i can see that is unlimted in file(blocks) that means i can run unlimited files.

# 4  
Old 08-02-2010
Quote:
tail -1 $file1| awk '/^"9"/' > new_$file
Slightly off topic.
The routine generates $file not $file1 .

Quote:
for file in `ls |egrep '(CUSTOMER_INFO_.*_'$Flag'.*csv$)'`
do
This construct is likely to give trouble with 2000 files because the command line will become very long. The syntax shown is not valid in my "egrep" so it is difficult to suggest an alternative without knowing what selection of files was intended.


On topic: The kernel parameter for the maximum number of concurrent processes by a single user will be relevant. This value is not displayed by the "ulimit" command and is Operating System specific.
# 5  
Old 08-02-2010
PHP Code:
touch the files 

CUSTOMER_INFO_12345_1111.csv
CUSTOMER_INFO_12345_2222.csv
CUSTOMER_INFO_12345_2234.csv
CUSTOMER_INFO_12345_3456.csv
CUSTOMER_INFO_12345_5553.csv
CUSTOMER_INFO_12345_6661.csv
CUSTOMER_INFO_12345_7752.csv

then the below command.

ls |egrep '(CUSTOMER_INFO_.*_1.*csv$)'
CUSTOMER_INFO_12345_1111.csv
ls |egrep '(CUSTOMER_INFO_.*_3.*csv$)'
CUSTOMER_INFO_12345_3456.csv 
HTML Code:
i am able to fetch the files. When you see the below command i am fetching two files.

Code:
ls |egrep '(CUSTOMER_INFO_.*_2.*csv$)'

PHP Code:
CUSTOMER_INFO_12345_2222.csv
CUSTOMER_INFO_12345_2234.csv


the function should run parallely for the both files. Currently it is running for CUSTOMER_INFO_12345_2222.csv then followed by CUSTOMER_INFO_12345_2234.csv.

for the above command extracting for file names which is starts with 2. 
Any help grealy appriciated.
Last edited by onesuri; 08-03-2010 at 02:58 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to paste multiple files in parallel?

Hi all, I am trying to paste thousands of files together into a matrix. Each file has only 1 column and all the files have the same number of rows (~27k rows). I tried paste * > output as well as some other for loops but the output only contains the columns from the 1st and last files. The... (8 Replies)
Discussion started by: notimenocall
8 Replies

2. Shell Programming and Scripting

Comparing list of files in parallel

Hi everyone. I have a list of files like: file001 file002 file003 . . . . file385 file386 file387 There are more files than above, but I hope you understand what I'm trying to do here. Is there a way I can create a loop to compare: file001 with file385 file002 with file386 (9 Replies)
Discussion started by: craigsky
9 Replies

3. Shell Programming and Scripting

Customized command excution

Hi, The below commands will be inside a KSH script and SOLARIS machine. Command-1 //Executes in 10s,5s,..5mins,etc TIMER = Look for Command-1 execution status - IF finished in 25secs move to next command in the script IF NOT kill above command and move to next command. Command-2 Any... (9 Replies)
Discussion started by: penqueen
9 Replies

4. Shell Programming and Scripting

Need to read two files in parallel

Following is the requirement In FileA I have the content as follows. 1,2,3 111,222,333 1000,2000,3000 In FileB I have the content as follows. 4,5,6 444,555,666 4000,5000,6000 I need to read FileA and FileB parallely and create the FileC as follows. 1,2,3,4,5,6... (1 Reply)
Discussion started by: kmanivan82
1 Replies

5. Programming

Expext Script excution problem..

Hey Guys , iam trying to match to string in TCL scripts but it not working . Its only working true condition to print yes.. When condtion fail then even it prints YES Any suggestion this ?? if {} { puts"yes" ... (3 Replies)
Discussion started by: hackerdilli
3 Replies

6. Linux

clarification for cmd excution

Hi Code: nohup sar -u 10 $COUNT | awk 'NR>2 {$2=$4=$7="";}1' | tr -s ' ' '\t' 2>/dev/null >sar.out & i have execute above line from shell script .. but sar.out is created with ZERO bite size.. Help on that what is the problem here (1 Reply)
Discussion started by: Mani_apr08
1 Replies

7. Shell Programming and Scripting

Unable to transfer files from Windows 2000 Server to Sun Solaris..

Dear Friends, I need to transfer few files from a Windows 2000 server to Sun Solaris system, connected in the same network. This copy should be done as a batch job without asking for password to be entered every time. How to make this possible ??? At present I am using cygwin in my laptop... (4 Replies)
Discussion started by: ks_reddy
4 Replies

8. UNIX for Dummies Questions & Answers

Copy files in Parallel

Is there any out of box Command in Unix or is it possible through shell scripting to copy all the files in a directory in Parallel. Example. I am doing a COLD database backup. I have the data/dbx directory which has around 1000 data files. I was thinking is there a way to spawn the copy... (10 Replies)
Discussion started by: simonsimon
10 Replies

9. UNIX for Advanced & Expert Users

To find the time of excution of a particular unix command

Hi experts, It would be grateful if you can guide me on the following, 1.How to find the time of excution of a particular unix command. 2. How to find the terminal (IP Address) from where that particular unix command was excuted. 3. How to find the user who executed that particular unix... (1 Reply)
Discussion started by: apsprabhu
1 Replies

10. UNIX for Dummies Questions & Answers

xterm closing automatically after command excution

Hi, I have a commands in a file called commands.file, I am using this file with the following commnad xterm -e "commands.file" After executing this file in the new xterm, it is closing automatically. I want to use that xterm after that. Please help me regarding this? :) ... (4 Replies)
Discussion started by: chaitubek
4 Replies
Login or Register to Ask a Question