Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Downloading hdfs file to local UNIX through UNIX script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Downloading hdfs file to local UNIX through UNIX script
# 8  
Old 04-27-2015
Hi Corona688 ,

Thanks for your explanation.We need to modify this script .We used the block of code for one file in a directory and corresponding reformatting for one file.Could you kindly suggest me how to automate this for multiple files in a directory .Suppose in target directory (/user/target) 10 files are there ,and all the them are .txt file ,then how to automate this for all 10 files and then it will create 10 reformatted file(sequence no added) for all files in a different name.If you kindly advice me how to automate the above scenario ,it will be really benefecial .Thanks.
Code:
#! /bin/bash
#Downloading HDFS file to Local Unix & Reformatting


hdfs dfs -copyToLocal "$1"/"$2" .

FILE="$2"

awk '{printf "%06d,",NR} 1 ; END { printf "\n" }' "$FILE" >output.txt

# 9  
Old 04-30-2015
Sorry for missing this.

How about this:

Code:
#! /bin/bash
#Downloading HDFS file to Local Unix & Reformatting

N=0

DIR="$1" ; shift

while [ "$#" -gt 0 ]
do
    hdfs dfs -copyToLocal "$DIR"/"$1" .

    FILE="$1"

    awk '{printf "%06d,",NR} 1 ; END { printf "\n" }' "$FILE" >output${N}.txt
    let N=N+1
    shift
done

Run it like ./script.sh folder file1 file2 file3 file4 ...

Otherwise explain in detail what you really need.
This User Gave Thanks to Corona688 For This Post:
STCET22
# 10  
Old 05-07-2015
Hi Corona688 ,
Thanks a lot for your reply.When I am running the below script(./script.sh /user/target file1 file2 ) for multiple files , although the files are present in the directory ,I am getting error like no such file or directory.Also it would be highly beneficial for me if you kindly explain this script.Thanks !
# 11  
Old 05-11-2015
Hi Corona688 ,
Whenever I am running the script provided by you(./script.sh /user/target file1 file2 ) for multiple files ,I am getting error like below ,although file1,file2 is present in the /user/target directory.

Code:
-bash: ./Script.sh: /bin/bash^M: bad interpreter: No such file or directory

Could you kindly look into this.It would be really helpful for me if you kindly help me in this regard.If possible ,cud you pls explain the block of code you provided for multiple files.Thanks !
Last edited by STCET22; 05-11-2015 at 09:34 AM.. Reason: typo
# 12  
Old 05-11-2015
UNIX and Linux system tools use the <newline> character as the line terminator; not the Windows <carriage-return><newline> character pair. So, you should get this error no matter how many file operands you pass to this script. Remove the <carriage-return> characters from your script and it will probably work as expected. However, in theory, there should be no space before the interpreter name in the first line of the script, so I would change the first line in the script to:
Code:
#!/bin/bash

in case your operating system is picky about this issue.
This User Gave Thanks to Don Cragun For This Post:
STCET22
# 13  
Old 05-12-2015
Hi Don/Corona688 ,
Whenever I am using the below script for single file ,its working fine for me.
Code:
#! /bin/bash
#Downloading HDFS file to Local Unix & Reformatting

hdfs dfs -copyToLocal "$1"/"$2" .

FILE="$2"

awk '{printf "%06d,",NR} 1 ; END { printf "\n" }' "$FILE" >output.txt

But whenever I m running the below script for multiple files ,its throwing error.
Code:
#! /bin/bash
#Downloading HDFS file to Local Unix & Reformatting

N=0

DIR="$1" ; shift

while [ "$#" -gt 0 ]
do
    hdfs dfs -copyToLocal "$DIR"/"$1" .

    FILE="$1"

    awk '{printf "%06d,",NR} 1 ; END { printf "\n" }' "$FILE" >output${N}.txt
    let N=N+1
    shift
done

Error is like below :

Code:
./Script1.sh: /bin/bash^M: bad interpreter: No such file or directory

It would be really beneficial for me if you kindly help me in this regard.Thanks !
# 14  
Old 05-12-2015
Look at message #12 in this thread. I told you exactly what to do. You didn't do what I told you to do or you wouldn't be getting the error saying that it can't find /bin/bash<carriage-return> (AKA /bin/bash/^M).

Get rid of all of the <carriage-return> characters in your failing script and get rid of the space in the 1st line in that script. Then your script will at least have a chance to start running.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

UNIX and HDFS - file systems on same partition.

I am learning Hadoop. As a part of that, Hdfs - Hadoop distributed file system has commands similar to unix where we can create,copy,move files from unix/linux file system to HDFS. My question is 1) how two file systems (unix and hdfs) can coexist on thr same partition.. 2)What if block... (1 Reply)
Discussion started by: Narendra Eliset
1 Replies

2. UNIX for Advanced & Expert Users

UNIX and HDFS - file systems on same partition.

I am learning Hadoop. As a part of that, Hdfs - Hadoop distributed file system has commands similar to unix where we can create,copy,move files from unix/linux file system to HDFS. My question is 1) how two file systems (unix and hdfs) can coexist on thr same partition.. 2)What if block used... (0 Replies)
Discussion started by: Narendra Eliset
0 Replies

3. Shell Programming and Scripting

Need help how to copy few records from hdfs to UNIX

Hi All , I am facing one issue here...I have a huge file in hadoop file system.Some disk space issues are thr ,thatswhy I want to copy 1st 100 records from hdfs to local unix.I tried below command but it is no working .Its giving error like cat: Unable to write to output stream.if any one can... (2 Replies)
Discussion started by: STCET22
2 Replies

4. Shell Programming and Scripting

Downloading file from mainframe to UNIX

Hi All , I need a help regarding file ftp ing from mainframe to unix.Our source file is mainframe file.I need to download the mainframe file to local unix server through unix script.If anyone can help me how we can do it through unix script ,it will be really helpful.Thanks. (7 Replies)
Discussion started by: STCET22
7 Replies

5. Shell Programming and Scripting

How to transfer file from Local PC to UNIX Directory without FTP?

Dear All, i am trying to get the file from windows location to unix location without using FTP and neither thru entering the user id and password. I have one unix form which is running on web application and user is entering the location and file name there now i know the file name and path. So i... (8 Replies)
Discussion started by: ripudaman.singh
8 Replies

6. UNIX for Dummies Questions & Answers

How to transfer file from Local PC to Unix Directory without FTP!!!

Dear Friends, How to transfer files from my local PC to Unix directory without using FTP. Scenario: Transfer/Upload a file from PC to unix using web browser without using FTP technologies. I heard something like sendunix and sendpc used to transfer files from unix to Desktop and... (1 Reply)
Discussion started by: kk_c2il2
1 Replies

7. Shell Programming and Scripting

how to FTP a file from the local folder to unix server

Hi All, please help me to write a shell that ftp a file which is in the local (C:\) drive to a Unix server. Where as i know the IP for the Unix server. i could do this process by using ftp command. pls help me to write as Shell script. Thanks in advance for all of your answers.:b::b: (3 Replies)
Discussion started by: little_wonder
3 Replies

8. UNIX for Dummies Questions & Answers

Unix command used for downloading file from net

Hi, Which command I can use to download a file from website? I tried with wget and lwp-dowload but no gain. Can any one suggest me the good way to dowload? (2 Replies)
Discussion started by: siba.s.nayak
2 Replies

9. Shell Programming and Scripting

How i ftp a unix file to my local window

Hello Sir/ Madam, i m new user in unix shell scripting.Please guide me to crack this problem. Thanking you. (1 Reply)
Discussion started by: Nirmal
1 Replies

10. UNIX for Dummies Questions & Answers

Downloading Unix..

I am not sure, yet i want to learn.. therefor .. I want to put some Unix on my machine.. Unix 03, nut i have heard its not free and not open source.. please advise me on what to do.. (4 Replies)
Discussion started by: binary_w0lf
4 Replies
Login or Register to Ask a Question