You have a basic misunderstanding here. The number of cores is mostly irrelevant here. The remaining relevance of the number of cores is the huge amount of processes you want (not) to spawn here. Here indeed the number of cores does help. And yes spawning some 100 processes at once puts a considerable load on the server.

What is far more of importance, is the fact, that not cpu cores matters most, but I/O capacity does. And if you spawn multiple excessive I/O hungry tasks, that will hurt server performance much more(It may even kill the server) than the use of cpu-power, because cpu power is what this machine have more than 50 times of what is needed.

Furthermore an issue of I/O is the matter of fact, that especially hard drives are not faster if you spawn multiple I/O stressing tasks. On the contrary: Parallelizing this slows things further down. If you let one process get the full I/O power the hard disk(array), it may read/write continously with maximum speed(You get maximum Throughput if you either only read or write sequentially in one process from/to one disk(array). "copy" or "cat" reads and writes simultaneously). If you spawn e. g. 12 parallel I/O stressing tasks, the head of the hard disk has to do a lot of seeking: Continuosly jumping between the different locations of all 12 different processes data files. That's VERY costly regarding performance and greatly INCREASES the runtime of your script.

----

Your testcase script does not create any suprising nor relevant result because "sleep" is not creating any I/O which is what your real script does.

---

Thanks for posting the real names in your script, because this sheds a little more light on your case.

The question arising is: What do you want to achieve by concatenating .gz files? In my experience, you render the data in the resulting files unusable, because no tool known to me can separate such files into single .gz-archives again.

---

It seems to be the best, if you go back another couple of steps and tell us what you are trying to achieve in this whole process. What do you want to do and what's your current plan to achieve your goal?

I have a hunch that the current method is more like carrying a bicycle and that you can get a lot faster towards your goal if you start riding it.