Hi all,

I have a folder hierarchy with many gz files in them. I would like to recursively decompress them, but keep the original files. I would also like to move all the decompressed files (these are very large HDF5 files with .CP12 extension) to another data folder.

Currently I am using four steps to achieve this:

1. Make a copy of the source directory hierarchy:

Code:

cp -R old_archive/ new_archive/

2. Inside new_archive, gunzip recursively:

Code:

find . -name "*.gz" -exec gunzip {} \;

3. Move all decompressed files from new_archive hierarchy to a data folder:

Code:

find . -name "*.CP12" | xargs -I {} mv -iv {} ~/data/

4. Remove new_archive (empty hierarchy)

Code:

rm -rf new_archive/

This works. old_archive contains gz files and data contains the decompressed versions. But is time consuming.

My question is: how can I perform this recursive extraction efficiently? I would like to avoid step 1, since it takes a very long time (terabyte size datasets).

I need to prevent gunzip's default behavior (removing the original gz file). Since, grep -d or gunzip has a "-c" option to extract to stdout, how I can recursively extract and put in data?

If I write a simple shell script for this, would running gunzip -c hdf-file.gz> hdf-file recursively be more efficient that doing cp and then gunzip? Note that the decompressed files can be very big (gigabyte size each) and so I also want to prevent sized related errors during the pipe process. Could someone comment on this. Thanks in advance!

---------- Post updated at 06:21 PM ---------- Previous update was at 06:03 PM ----------

Here is another script version for the same:

Code:

#!/bin/bash

shopt -s extglob

for dir in *
do
  if [ -d $dir ]
  then
    echo "--- Entering directory $dir ---"
        for file in "${dir}"/*.gz
         do
            fname=`basename "$file" .gz`
            echo "Now processing $fname ..."
            gunzip -cv "$fname.gz" > "$fname"
            mv -iv "$fname" ~/data
         done
  fi
done

Is there a better way?