Sponsored Content
Top Forums Shell Programming and Scripting Downloading hdfs file to local UNIX through UNIX script Post 302942016 by Corona688 on Thursday 23rd of April 2015 02:13:57 PM
Old 04-23-2015
$# is a special variable meaning "number of arguments". It gets fed into a 'case' statement to run different code for different values.

When there's 2 arguments, it does
Code:
        hdfs dfs -copyToLocal "$1"/"$2" .
        FILE="$2"

When there's 1 argument, it does

Code:
        hdfs dfs -copyToLocal "$1" .
        OLDIFS="$IFS"
        # Split $1="a/b/filename" into $1="a", $2="b", $3="filename"
        IFS="/"
                set -- $1
        IFS="$OLDIFS"

        # Get rid of "a", "b"
        shift "$(( $# - 1 ))

        FILE="$1"
        ;;

IFS is a special variable used by the shell to control splitting, and set sets the $1 $2 ... arguments to what you tell it to, used together it splits a "/path/to/file" string into "path", "to", "file".

The shift gets rid of the first $# - 1 arguments to leave the last one.

Then it runs awk '{printf "%06d|",NR} 1' "$FILE" >output.txt and exits.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Downloading Unix..

I am not sure, yet i want to learn.. therefor .. I want to put some Unix on my machine.. Unix 03, nut i have heard its not free and not open source.. please advise me on what to do.. (4 Replies)
Discussion started by: binary_w0lf
4 Replies

2. Shell Programming and Scripting

How i ftp a unix file to my local window

Hello Sir/ Madam, i m new user in unix shell scripting.Please guide me to crack this problem. Thanking you. (1 Reply)
Discussion started by: Nirmal
1 Replies

3. UNIX for Dummies Questions & Answers

Unix command used for downloading file from net

Hi, Which command I can use to download a file from website? I tried with wget and lwp-dowload but no gain. Can any one suggest me the good way to dowload? (2 Replies)
Discussion started by: siba.s.nayak
2 Replies

4. Shell Programming and Scripting

how to FTP a file from the local folder to unix server

Hi All, please help me to write a shell that ftp a file which is in the local (C:\) drive to a Unix server. Where as i know the IP for the Unix server. i could do this process by using ftp command. pls help me to write as Shell script. Thanks in advance for all of your answers.:b::b: (3 Replies)
Discussion started by: little_wonder
3 Replies

5. UNIX for Dummies Questions & Answers

How to transfer file from Local PC to Unix Directory without FTP!!!

Dear Friends, How to transfer files from my local PC to Unix directory without using FTP. Scenario: Transfer/Upload a file from PC to unix using web browser without using FTP technologies. I heard something like sendunix and sendpc used to transfer files from unix to Desktop and... (1 Reply)
Discussion started by: kk_c2il2
1 Replies

6. Shell Programming and Scripting

How to transfer file from Local PC to UNIX Directory without FTP?

Dear All, i am trying to get the file from windows location to unix location without using FTP and neither thru entering the user id and password. I have one unix form which is running on web application and user is entering the location and file name there now i know the file name and path. So i... (8 Replies)
Discussion started by: ripudaman.singh
8 Replies

7. Shell Programming and Scripting

Downloading file from mainframe to UNIX

Hi All , I need a help regarding file ftp ing from mainframe to unix.Our source file is mainframe file.I need to download the mainframe file to local unix server through unix script.If anyone can help me how we can do it through unix script ,it will be really helpful.Thanks. (7 Replies)
Discussion started by: STCET22
7 Replies

8. Shell Programming and Scripting

Need help how to copy few records from hdfs to UNIX

Hi All , I am facing one issue here...I have a huge file in hadoop file system.Some disk space issues are thr ,thatswhy I want to copy 1st 100 records from hdfs to local unix.I tried below command but it is no working .Its giving error like cat: Unable to write to output stream.if any one can... (2 Replies)
Discussion started by: STCET22
2 Replies

9. UNIX for Advanced & Expert Users

UNIX and HDFS - file systems on same partition.

I am learning Hadoop. As a part of that, Hdfs - Hadoop distributed file system has commands similar to unix where we can create,copy,move files from unix/linux file system to HDFS. My question is 1) how two file systems (unix and hdfs) can coexist on thr same partition.. 2)What if block used... (0 Replies)
Discussion started by: Narendra Eliset
0 Replies

10. UNIX for Beginners Questions & Answers

UNIX and HDFS - file systems on same partition.

I am learning Hadoop. As a part of that, Hdfs - Hadoop distributed file system has commands similar to unix where we can create,copy,move files from unix/linux file system to HDFS. My question is 1) how two file systems (unix and hdfs) can coexist on thr same partition.. 2)What if block... (1 Reply)
Discussion started by: Narendra Eliset
1 Replies
read(1) 							   User Commands							   read(1)

NAME
read - read a line from standard input SYNOPSIS
/usr/bin/read [-r] var... sh read name... csh set variable = $< ksh read [ -prsu [n]] [ name ? prompt] [name...] DESCRIPTION
/usr/bin/read The read utility will read a single line from standard input. By default, unless the -r option is specified, backslash () acts as an escape character. If standard input is a terminal device and the invoking shell is interactive, read will prompt for a continuation line when: o The shell reads an input line ending with a backslash, unless the -r option is specified. o A here-document is not terminated after a NEWLINE character is entered. The line will be split into fields as in the shell. The first field will be assigned to the first variable var, the second field to the second variable var, and so forth. If there are fewer var operands specified than there are fields, the leftover fields and their interven- ing separators will be assigned to the last var. If there are fewer fields than vars, the remaining vars will be set to empty strings. The setting of variables specified by the var operands will affect the current shell execution environment. If it is called in a subshell or separate utility execution environment, such as one of the following: (read foo) nohup read ... find . -exec read ... ; it will not affect the shell variables in the caller's environment. The standard input must be a text file. sh One line is read from the standard input and, using the internal field separator, IFS (normally space or tab), to delimit word boundaries, the first word is assigned to the first name, the second word to the second name, and so on, with leftover words assigned to the last name. Lines can be continued using ewline. Characters other than NEWLINE can be quoted by preceding them with a backslash. These backslashes are removed before words are assigned to names, and no interpretation is done on the character that follows the backslash. The return code is 0, unless an end-of-file is encountered. csh The notation: set variable = $< loads one line of standard input as the value for variable. (See csh(1)). ksh The shell input mechanism. One line is read and is broken up into fields using the characters in IFS as separators. The escape character, (), is used to remove any special meaning for the next character and for line continuation. In raw mode, -r, the character is not treated specially. The first field is assigned to the first name, the second field to the second name, and so on, with leftover fields assigned to the last name. The -p option causes the input line to be taken from the input pipe of a process spawned by the shell using |&. If the -s flag is present, the input will be saved as a command in the history file. The flag -u can be used to specify a one digit file descriptor unit n to read from. The file descriptor can be opened with the exec special command. The default value of n is 0. If name is omitted, REPLY is used as the default name. The exit status is 0 unless the input file is not open for reading or an end-of-file is encoun- tered. An end-of-file with the -p option causes cleanup for this process so that another can be spawned. If the first argument contains a ?, the remainder of this word is used as a prompt on standard error when the shell is interactive. The exit status is 0 unless an end-of- file is encountered. OPTIONS
The following option is supported: -r Does not treat a backslash character in any special way. Considers each backslash to be part of the input line. OPERANDS
The following operand is supported: var The name of an existing or non-existing shell variable. EXAMPLES
Example 1: An example of the read command The following example for /usr/bin/read prints a file with the first field of each line moved to the end of the line: example% while read -r xx yy do printf "%s %s " "$yy" "$xx" done < input_file ENVIRONMENT VARIABLES
See environ(5) for descriptions of the following environment variables that affect the execution of read: LANG, LC_ALL, LC_CTYPE, LC_MES- SAGES, and NLSPATH. IFS Determines the internal field separators used to delimit fields. PS2 Provides the prompt string that an interactive shell will write to standard error when a line ending with a backslash is read and the -r option was not specified, or if a here-document is not terminated after a newline character is entered. EXIT STATUS
The following exit values are returned: 0 Successful completion. >0 End-of-file was detected or an error occurred. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ |Interface Stability |Standard | +-----------------------------+-----------------------------+ SEE ALSO
csh(1), ksh(1), line(1), set(1), sh(1), attributes(5), environ(5), standards(5) SunOS 5.10 28 Mar 1995 read(1)
All times are GMT -4. The time now is 08:26 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy