Shell Programming and Scripting

Hi All ,
I am very new to unix script.I am aware of unix commands but never put together in unix script level.If any one can suggest me technical guidance in the below scenario that will highly beneficial.

Data have been already migrated from mainframe to Hadoop file system(HDFS).HDFS server is hosted on a unix box.HDFS file is just .txt file.We are currently downloading HDFS to local unix file system by HDFS command and then adding sequence number to the local unix file.We just want to automate this process for any file.

Currently we are using below steps :
We need to automate the above process through unix script for any user provided input file .If anyone can provide me technical approach/block of code to automate the above scenario that will be real help.Thanks in advance !

---------- Post updated at 08:24 AM ---------- Previous update was at 03:52 AM ----------

Hi All ,
I have written the below script which is working for a single file.I have hardcoded the hdfs path in the script itself.And the script is creating pipe dilimited sequence generated o/p file in the local directory.We just need to modify this script for any user provided input file.Can anyone suggest me how to automate this script for any user provided input file which should not be hardcoded in the script.Any help in this regard will be highly appreciated.Thanks.

Code:

#! /bin/bash
#Downloading HDFS file to Local Unix & Reformatting

hdfs dfs -copyToLocal /user/target/file.txt .

awk '{printf "%06d|",NR} 1' file.txt >output.txt

What exactly would they be inputting?

We need to provide directory name & file name.Suppose a file called input.txt is located at hdfs path /user/target directory .We can pass the total filepath like /user/target/input.txt or
<input_directory> <sourcefile> as two different parameter.

Code:

case "$#" in
2)
        hdfs dfs -copyToLocal "$1"/"$2" .
        FILE="$2"
        ;;
1)
        hdfs dfs -copyToLocal "$1" .
        OLDIFS="$IFS"
        # Split $1="a/b/filename" into $1="a", $2="b", $3="filename"
        IFS="/"
                set -- $1
        IFS="$OLDIFS"

        # Get rid of "a", "b"
        shift "$(( $# - 1 ))

        FILE="$1"
        ;;
*)
        echo "Usage:  $0 path file"
        echo "Alternate usage:  $0 path/file"
        exit 1
        ;;
esac

awk '{printf "%06d|",NR} 1' "$FILE" >output.txt

Hi Corona688 ,

Thanks a lot for your reply.As Im new to unix script level ,I am not clear the block of code you have provided.So please bear with me and requesting you to kindly provide the explanation .
You have written the codeblock for which case
if
case1:/user/target/input.txt as a single parameter
case 2 : <input_directory> <sourcefile> as two different parameter.

If you kindly explain your block of code ,that will be really helpful for me.Thanks !

$# is a special variable meaning "number of arguments". It gets fed into a 'case' statement to run different code for different values.

When there's 2 arguments, it does

Code:

        hdfs dfs -copyToLocal "$1"/"$2" .
        FILE="$2"

When there's 1 argument, it does

Code:

        hdfs dfs -copyToLocal "$1" .
        OLDIFS="$IFS"
        # Split $1="a/b/filename" into $1="a", $2="b", $3="filename"
        IFS="/"
                set -- $1
        IFS="$OLDIFS"

        # Get rid of "a", "b"
        shift "$(( $# - 1 ))

        FILE="$1"
        ;;

IFS is a special variable used by the shell to control splitting, and set sets the $1 $2 ... arguments to what you tell it to, used together it splits a "/path/to/file" string into "path", "to", "file".

The shift gets rid of the first $# - 1 arguments to leave the last one.

Then it runs awk '{printf "%06d|",NR} 1' "$FILE" >output.txt and exits.

P.S. I might change that awk into

Code:

awk '{printf "%06d|",NR} 1 ; END { printf "\n" }' "$FILE" >output.txt

...to add a newline to the end of the file. Text files that don't end in a newline can confuse certain programs.