That's roughly what I meant with back some steps, but the situation is far away from a clear picture, what you try to do.

I understand for sure that you have plenty of GB of strings/sequences.

Regarding this command

Code:

cat EMS01a_Early_Rice_*_R1.fq.gz EMS01b_Early_Rice_*_R1.fq.gz > ../Early_Rice_1_R1.fq.gz

I'm sorry to say, but this command and all alikes are nonsense. What you do here is - as I said - concatenate compressed files which will not be possible to be recovered.

Maybe the following is what you try to achieve(And this needs cpu-power!):

Code:

zcat EMS01a_Early_Rice_*_R1.fq.gz EMS01b_Early_Rice_*_R1.fq.gz | gzip -9 > ../Early_Rice_1_R1.fq.gz

I guess you like to combine different sequence sets for different projects. If the sequence sets are statically - if they do not change(strict requirement), you maybe able to just use the data-set you need dynamically assigned to a project.

Example for dynamic assignment of static data to different projects

The following data/sequence-sets are available(directory structure):

Code:

data_sets
|---EMS01
|   EMS01a.txt
|   EMS01b.txt
|   EMS01c.txt
\---EMS02
    EMS02a.txt
    EMS02b.txt
    EMS02c.txt

..and maybe you have a project structure too, which is using specific files of the data set(The arrows mean the files are symbolic links to the real data files):

Code:

projects
|---project_01_data
|   a -> ../../data_sets/EMS01a.txt
|   b -> ../../data_sets/EMS01c.txt
|   c -> ../../data_sets/EMS02c.txt
\---project_02_data
    a -> ../../data_sets/EMS01a.txt
    b -> ../../data_sets/EMS02b.txt
    c -> ../../data_sets/EMS02c.txt

So if you want the data for project_01 you may just create the data on standard output(which should be fed into your processing - whatever that is) on the fly with this command:

Code:

cat projects/project_01_data/*  | research_program

The basic idea is: Don't copy data around if not really needed. Have the parts once where needed and keep them forever as they are.

--

To further reduce I/O you can compress the Data files. If you have xz available: use that! It can compress far better than gzip - it needs more cpu-power too, but if compressing speed matters that can be parallelized quite good in your scenario. And you don't even have to permanently decompress your files. As a replacement for cat there's zcat(gzip) and xzcat(xz) to decompress to stdout when you need it.

Compressing can save a lot I/O!