Need some help to replace bash script with parallel to speed up job on multiple files (400files.list is the file contains the absolute path to those 400 files). The bash script is to run the same program over the files repetitively.
My bash_script.sh is:
Parallel fits into this job. From gnu.org parallel manual I read:
Quote:
Assuming that file contains a list of shell commands, one per line,
will evaluate the commands using the shell (since no explicit command is supplied as an argument), in blocks of ten shell jobs at a time.
which is related to my previous post. I was thinking something like
but I have two issues here:
there will be ~400 .sh files, which seems not correct obviously;
my script contains multiple lines of a single job with other variables embedded such as I_DIR, O_DIR, GENOME_DIR and LOG_DIR.
I'm quite lost in my mind how to replace the bash script with parallel to speed up the job.
Thanks in advance!
---------- Post updated at 06:32 PM ---------- Previous update was at 03:48 PM ----------
Just tried one way myself, but not sure it can be optimized:
Then change the permission of the script to be executable. Run as:
Thanks for any suggestion on parallel and bash script.
Last edited by rbatte1; 08-25-2016 at 08:43 AM..
Reason: Converted to formatted number-list
Hi,
Iam having the scripts as follows. i jus want to run those in parallel.
main aim is to minimise the time for overall execution of the script.
now out.txt is having 1 lac records.
script1(split.sh)
split -1000 out.txt splitout
ls -A splitout* > filelist.txt
cat filelist.txt... (6 Replies)
Hi All,
I have 3 shell scripts, Script1,Script2 and Script3. Now I want to run Script1 and Script2 in parallel and Script3 should depend on successful completion of both Script1 and Script2.
Could you please suggest an approach of acheiving this...
Thanks in advance (2 Replies)
Hi,
I have to write a program to compute the checksums of files
./script.sh
I wrote the program using bash and it took me forever since I am a beginner but it works very well.
I'm getting so close to the deadline and I realised today that actually I have to use normal Bourne shell... (3 Replies)
I have one parent directory and within that parent directory there are several other sub-directories and within those sub-directories there are several other "large number" of sub-directories.
All the sub directories have a shell script in them with a common file name execute_command.sh I want... (4 Replies)
Hi
I want to execute few scripts in Parallel. There is a Master Script (MS.ksh) which will call internally all the scripts we need to run in Parallel. Say there are three set of scripts :
ABC_1.ksh --> ABC_2.ksh --> ABC_3.ksh (execute ABC_2 when ABC_1 is successful ; Execute ABC_3 only when... (6 Replies)
Hi
Say I am interested in processing a big data set over shell, and each process individually takes a long time, but many such processes can be pipe-lined, is there a way to do this automatically or efficiently in shell?
For example consider pinging a list addresses upto 5 times each. Of... (5 Replies)
Hi all - I am totally stuck here :wall
I have been asked to write a shell script that does a few little things and then reads from a config file and kicks off an instance of another script, say scriptB.ksh for each line in the config file. These should all be run in parallel. This is all fine but... (2 Replies)
i have script A and script B, both scripts have to run in parallel, my requirement is script A create table temp1, post creating it will run fr 4 hrs , script B has to start 0nly after creation of table temp1 ( which is done by script A) , again script B will run for 5 hrs
if i run sequencially... (7 Replies)
Hi all,
I am struggling to build a utility which can do the parallel processing. I achieved same in Linux using xargs -P but same is not working on AIX.
I am building file copy utility where I will have all required info in a file (like souce file info and target location details), now i need... (2 Replies)
Hello all,
I tried to parralise my treatments but after a while 'ps -ef' display all child process <defunct> (zombie)
Parent bash script to process all files (>100000) in directory:
for filename in /Data/*.txt; do
./child_pprocess.sh $filename &
done
exit(0)I understand that the... (1 Reply)