Login or Register to Ask a Question and Join Our Community


UNIX for Advanced & Expert Users

How to process multiple files parallely

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users How to process multiple files parallely
# 1  
Old 10-18-2012
How to process multiple files parallely
Hi,

I have a requirement to process multiple files in a directory parallely.Consider the below scenario:

In a directory there are three files file1,file2 and file3.When I use for loop each file will be executed in sequence but I want to process parallely.

Any Help would be appreciated.

Thanks,
Liyakath Ali.
# 2  
Old 10-18-2012
Do you simply want to background each process and wait until they are all completed, WARNING untested off the top of my head, on my first cup of coffee code follows
Code:
for index in $(seq 1 10); do
  pid_1=file_1&
  pid_2=file_2& 
  pid_3=file_3&
  while kill -0 $pid_1 -o  kill -0 $pid_2 -o  kill -0 $pid_3 ; do
    sleep 1;
  done
done

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Xargs to call python executable to process multiple bam files

I am running the below loop that to process the 3 bam files (which isn't always the case). A .py executable is then called using | xargs sh to further process. If I just run it with echo the output is fine and expected, however when | xargs sh is added I get the error. I tried adding | xargs... (4 Replies)
Discussion started by: cmccabe
4 Replies

2. Shell Programming and Scripting

Process multiple large files with awk

Hi there, I'm camor and I'm trying to process huge files with bash scripting and awk. I've got a dataset folder with 10 files (16 millions of row each one - 600MB), and I've got a sorted file with all keys inside. For example: a sample_1 200 a.b sample_2 10 a sample_3 10 a sample_1 10 a... (4 Replies)
Discussion started by: camor
4 Replies

3. Shell Programming and Scripting

Run multiple procedures from shell script parallely

Hi, I need to write a Shell Script wherein i will connect to a DB2 Database and run multiple DB procedures. I know how to do in a way where procedures will be called one after the other, like when first procedure finishes, second will be executed. But what i want is to run them at the same time... (11 Replies)
Discussion started by: Neelkanth
11 Replies

4. UNIX for Dummies Questions & Answers

Writing a loop to process multiple input files by a shell script

I have multiple input files that I want to manipulate using a shell script. The files are called 250.1 through 250.1000 but I only want the script to manipulate 250.300 through 250.1000. Before I was using the following script to manipulate the text files: for i in 250.*; do || awk... (4 Replies)
Discussion started by: evelibertine
4 Replies

5. Shell Programming and Scripting

How to process multiple input files using Shell scripting

Hi, I would like to write a for loop that does the following: I have a file called X.txt and other files called 1.txt,2.txt, .....,1000.txt. I want to substitute the 6th column of the file X.txt with 1.txt and store the output as X.1. Then I want to do the same with X.txt and 2.txt and store... (0 Replies)
Discussion started by: evelibertine
0 Replies

6. Shell Programming and Scripting

Multiple Threads/Tasks to run parallely using the shell script

Scenario: I have two PCs (named as A & B) which would send some traps to my third PC (named as C). In PC C, I have to write a shell script such that it should accept the datas from both the PC-A & B parallely. So my question is, is it possible to have two different child threads/tasks... (2 Replies)
Discussion started by: nthiruvenkatam
2 Replies

7. Shell Programming and Scripting

executing 2 or more files inside another file parallely

hi All, i have 3 files file1 file2 file3 these 3 files are shell (KSH) scripts. i want to place all these 3 shell scripts in another file file99 and execute these files simultaneoulsy inside the file99 these 3 files should be present and executed simultaneously when i excute file99 (nohup... (1 Reply)
Discussion started by: dareman123
1 Replies

8. Programming

Process multiple pcap files at once

Hi all, I'm writing a program using libpcap, and I have multiple pcap files in a folder that I want to capture. I currently have handle = pcap_open_offline("/data/traffic/pcap1.pcap", errbuf"); which works fine since pcap_open_offline() takes in a filename. However, I want to process... (0 Replies)
Discussion started by: lancer6238
0 Replies

9. UNIX for Dummies Questions & Answers

How do I Process Multiple Files

Hi, First post here. In Mac OSX terminal I need to run a program against multiple files in a directory and append the output to tab separated variable file. I currently type the following to process just one file MBP:/Users/dc1743/Desktop/SFT root# ./myprogram myfile1.plist >>... (1 Reply)
Discussion started by: dc1743
1 Replies

10. Shell Programming and Scripting

How to process multiple files in Korn Shell

How do I make the below ksh to process all of the files inside a user specified directory? Currently it can only process one file at a time. #!/bin/ksh tr -s '\11 ' ' ' < $1 > temp0 sed -e 's/,//g' temp0 > temp1 cut -d' ' -f1,4,5 temp1 > final_output rm temp0 temp1 (3 Replies)
Discussion started by: stevefox
3 Replies
Login or Register to Ask a Question