Shell Programming and Scripting

what does the below script doing, the first path with wild card clinical_event* has all .gz compressed files multiple and the final path has 1 single clinical_event.txt file

almost 350GB in size.

Code:

hadoop fs -text /user/hive/warehouse/stage.db/CLINICAL_EVENT/CLINICAL_EVENT* | hadoop fs -put - $BASE_FOLDER/$base_name/final/clinical_event.txt

CLINICAL_EVENT* i have all .gz file 20 files and in the final path i only have 1 file .txt with almost 350GB file clinical_event.txt

is it uncompressing all .gz files almost 20 numbers from first path and consolidating into one big uncompressed .txt file with same name clinical_event.txt


Thanks a lot for the helpful info.

Moderator's Comments:

Please use CODE tags for full line and multi-line sample input, output, and code segments and ICODE tags for partial line sample input, output, and code segments.

First, note that clinical_event* and CLINICAL_EVENT_* match non-overlapping sets of files. (Case matters.)

To what strings do $BASE_FOLDER and $base_name expand?

What is the output (in CODE tags) from the command:

Code:

ls -l /user/hive/warehouse/stage.db/CLINICAL_EVENT/CLINICAL_EVENT* $BASE_FOLDER/$base_name/final/clinical_event.txt

Sorry, i didn't realize i put the variables.

here are the exact full paths:

Code:

hadoop fs -text /user/hive/warehouse/stage.db/CLINICAL_EVENT/CLINICAL_EVENT* | hadoop fs -put - /user/hive/warehouse/stage.db/Clinical_event/final/clinical_event.txt

doe the above script line picking all .gz files which are almost 20 numbers and consolidating into one uncompressed txt file to final path?

the only file i see is clinical_event.txt with size almost 350GB.

Moderator's Comments:

PLEASE do not use ICODE tags when displaying full-line and multi-line code segments; use CODE tags instead.

And, as requested before, what output do you get from the command:

Code:

ls -l /user/hive/warehouse/stage.db/CLINICAL_EVENT/CLINICAL_EVENT* /user/hive/warehouse/stage.db/Clinical_event/final/clinical_event.txt

hadoop is a command line utility with many subcommands.
the fs is one of those subcommands.
The subcommand fs accepts many options, one of those options is -text, which requires a source, one of the sources can be multiple gzip files and they get outputted into text format.
Another option of the fs subcomand is -put which requires two arguments, a local source and a destination.

Let's use that to know what's going on:
There are two instances of hadoop fs working together joined by a pipe (|), where the output of the first one, becomes the first argument of the second one.
In essence, it converts all those .gzip files (assuming that CLINICAL_EVENT*, refers to only gzip files, like you said) to text format and pushes it to standard output where the second picks it up, via standard input, (that's the - highlighted in red) and places it into /user/hive/warehouse/stage.db/Clinical_event/final/clinical_event.txt, which is a single file.

Legend:
Blue: subcommand with option
Red: stdin
Purple: pipe

Code:

hadoop fs -text /user/hive/warehouse/stage.db/CLINICAL_EVENT/CLINICAL_EVENT* | hadoop fs -put - /user/hive/warehouse/stage.db/Clinical_event/final/clinical_event.txt