Quote:
You have a basic misunderstanding here. The number of cores is mostly irrelevant here. ......What is far more of importance, is the fact, that not cpu cores matters most, but I/O capacity does.
Thank you for point this out as I was not aware of this originally.
Quote:
Furthermore an issue of I/O is the matter of fact, that especially hard drives are not faster if you spawn multiple I/O stressing tasks. ......If you spawn e. g. 12 parallel I/O stressing tasks, the head of the hard disk has to do a lot of seeking: Continuously jumping between the different locations of all 12 different processes data files. That's VERY costly regarding performance and greatly INCREASES the runtime of your script. ...... Your testcase script does not create any suprising nor relevant result because "sleep" is not creating any I/O which is what your real script does.
I think I understand the points more now with this explanation.
Quote:
The question arising is: What do you want to achieve by concatenating .gz files?
Hard to explain in short here. Those files are genomic data of different collections for each same material to get the gene expression abundance by "sequence" (string in computer term, (count number, compare to
"grep "ATCG" file.rice | wc") ) at different time. They are concatenated to have summed abundance in total.
Quote:
It seems to be the best, if you go back another couple of steps and tell us what you are trying to achieve in this whole process. What do you want to do and what's your current plan to achieve your goal?
1) I was looking for the correct syntax to pipe jobs from bash-script with parallel, as I was thinking of parallel to speed up the processes, which seems not quite correct.
2) The reason for 1) is hundreds file (~10GB average size) are spread in different places but belong to single project. They must be combined to give simpler organization like usual biological experiment design.
3)
Quote:
... that you can get a lot faster towards your goal if you start riding it.
Of course I do want to ride the bicycle instead of carrying it, and my ultimate goal is to learn how to ride the bicycle, or to drive a car, as you guys are driving race cars here!
Thank you again!
Moderator's Comments:
|
|
Please use quote tags, not icode tags, when quoting other posts!
|
|