|
|
Sponsored Content
Top Forums
Shell Programming and Scripting
Parallelize bash commands/jobs
Post 302974614 by yifangt on Wednesday 1st of June 2016 11:34:09 AM
06-01-2016
Quote:
You have a basic misunderstanding here. The number of cores is mostly irrelevant here. ......What is far more of importance, is the fact, that not cpu cores matters most, but I/O capacity does.
Quote:
Furthermore an issue of I/O is the matter of fact, that especially hard drives are not faster if you spawn multiple I/O stressing tasks. ......If you spawn e. g. 12 parallel I/O stressing tasks, the head of the hard disk has to do a lot of seeking: Continuously jumping between the different locations of all 12 different processes data files. That's VERY costly regarding performance and greatly INCREASES the runtime of your script. ...... Your testcase script does not create any suprising nor relevant result because "sleep" is not creating any I/O which is what your real script does.
Quote:
The question arising is: What do you want to achieve by concatenating .gz files?
Quote:
It seems to be the best, if you go back another couple of steps and tell us what you are trying to achieve in this whole process. What do you want to do and what's your current plan to achieve your goal?
2) The reason for 1) is hundreds file (~10GB average size) are spread in different places but belong to single project. They must be combined to give simpler organization like usual biological experiment design.
3)
Quote:
... that you can get a lot faster towards your goal if you start riding it.
Thank you again!
| Moderator's Comments: | ||
|
Last edited by RudiC; 06-01-2016 at 01:06 PM.. Reason: Changed icode tags to quote tags
|
|
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
i need to execute 5 jobs at a time in background and need to get the exit status of all the jobs i wrote small script below , i'm not sure this is right way to do it.any ideas please help.
$cat run_job.ksh
#!/usr/bin/ksh
####################################
typeset -u SCHEMA_NAME=$1
... (1 Reply)
Discussion started by: GrepMe
1 Replies
2. Shell Programming and Scripting
I want to log into a remote server transfer over a new config and then backup the existing config, replace with the new config.
I am not sure if I can do this with BASH scripting.
I have set up password less login by adding my public key to authorized_keys file, it works.
I am a little... (1 Reply)
Discussion started by: bash_in_my_head
1 Replies
3. Shell Programming and Scripting
Hi All,
I am trying to run this script. I have a small problem:
each "./goada.sh" command when done produces three files (file1, file2, file3) then they are moved to their respective directory as can be seem from this script snippet here.
The script goada.sh sends some commands for some... (1 Reply)
Discussion started by: faizlo
1 Replies
4. Shell Programming and Scripting
Status quo is, within a web application, which is coded completely in php (not by me, I dont know php), I have to fill out several fields, and execute it manually by clicking the "go" button in my browser, several times a day.
Thats because:
The script itself pulls data (textfiles) from a... (3 Replies)
Discussion started by: lowmaster
3 Replies
5. Shell Programming and Scripting
I need to find a smarter way to process about 60,000 files in a single directory.
Every night a script runs on each file generating a output on another directory; this used to take 5 hours, but as the data grows it is taking 7 hours.
The files are of different sizes, but there are 16 cores... (10 Replies)
Discussion started by: vhope07
10 Replies
6. Shell Programming and Scripting
Hello,
I am running GNU bash, version 3.2.39(1)-release (x86_64-pc-linux-gnu). I have a specific question pertaining to waiting on jobs run in sub-shells, based on the max number of parallel processes I want to allow, and then wait... (1 Reply)
Discussion started by: srao
1 Replies
7. Shell Programming and Scripting
Dear all,
I'm a newbie in programming and I would like to know if it is possible to parallelize the script:
for l in {1..1000}
do
cut -f$l quase2 |tr "\n" "," |sed 's/$/\
/g' |sed '/^$/d' >a_$l.t
done
I tried:
for l in {1..1000}
do
cut -f$l quase2 |tr "\n" "," |sed 's/$/\
/g' |sed... (7 Replies)
Discussion started by: valente
7 Replies
8. Shell Programming and Scripting
Hello,
the bulk of my work is run by scripts. An example is as such:
#!/bin/bash
awk '{print first line}' Input.in > Intermediate.ter
awk '{print second line}' Input.in > Intermediate_2.ter
command Intermediate.ter Intermediate_2.ter > Output.out
It works the way I want it to, but it's not... (1 Reply)
Discussion started by: Leo_Boon
1 Replies
9. Shell Programming and Scripting
I have multiple jobs and each job dependent on other job.
Each Job generates a log and If job completed successfully log file end's with JOB ENDED SUCCESSFULLY message and if it failed then it will end with JOB ENDED with FAILURE.
I need an help how to start.
Attaching the JOB dependency... (3 Replies)
Discussion started by: santoshkumarkal
3 Replies
10. Shell Programming and Scripting
How to run several bash commands put in bash command line without needing and requiring a script file.
Because I'm actually a windows guy and new here so for illustration is sort of :
$ bash "echo ${PATH} & echo have a nice day!"
will do output, for example:... (4 Replies)
Discussion started by: abdulbadii
4 Replies