Sponsored Content
Top Forums Shell Programming and Scripting Downloading hdfs file to local UNIX through UNIX script Post 302941949 by STCET22 on Thursday 23rd of April 2015 09:24:57 AM
Old 04-23-2015
Downloading hdfs file to local UNIX through UNIX script

Hi All ,
I am very new to unix script.I am aware of unix commands but never put together in unix script level.If any one can suggest me technical guidance in the below scenario that will highly beneficial.

Data have been already migrated from mainframe to Hadoop file system(HDFS).HDFS server is hosted on a unix box.HDFS file is just .txt file.We are currently downloading HDFS to local unix file system by HDFS command and then adding sequence number to the local unix file.We just want to automate this process for any file.

Currently we are using below steps :
  1. Downlading HDFS file to local unix system by below command :
    Code:
    Hdfs dfs -copyToLocal (HDFS file path) (Local directory path)

  2. Adding sequnce generated No :
    Code:
    awk '{printf "%06d,",NR} 1' File.txt >File_Output.txt


We need to automate the above process through unix script for any user provided input file .If anyone can provide me technical approach/block of code to automate the above scenario that will be real help.Thanks in advance !

---------- Post updated at 08:24 AM ---------- Previous update was at 03:52 AM ----------

Hi All ,
I have written the below script which is working for a single file.I have hardcoded the hdfs path in the script itself.And the script is creating pipe dilimited sequence generated o/p file in the local directory.We just need to modify this script for any user provided input file.Can anyone suggest me how to automate this script for any user provided input file which should not be hardcoded in the script.Any help in this regard will be highly appreciated.Thanks.

Code:
#! /bin/bash
#Downloading HDFS file to Local Unix & Reformatting

hdfs dfs -copyToLocal /user/target/file.txt .

awk '{printf "%06d|",NR} 1' file.txt >output.txt


Last edited by rbatte1; 04-30-2015 at 01:33 PM.. Reason: Added CODE and LIST=1 tags
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Downloading Unix..

I am not sure, yet i want to learn.. therefor .. I want to put some Unix on my machine.. Unix 03, nut i have heard its not free and not open source.. please advise me on what to do.. (4 Replies)
Discussion started by: binary_w0lf
4 Replies

2. Shell Programming and Scripting

How i ftp a unix file to my local window

Hello Sir/ Madam, i m new user in unix shell scripting.Please guide me to crack this problem. Thanking you. (1 Reply)
Discussion started by: Nirmal
1 Replies

3. UNIX for Dummies Questions & Answers

Unix command used for downloading file from net

Hi, Which command I can use to download a file from website? I tried with wget and lwp-dowload but no gain. Can any one suggest me the good way to dowload? (2 Replies)
Discussion started by: siba.s.nayak
2 Replies

4. Shell Programming and Scripting

how to FTP a file from the local folder to unix server

Hi All, please help me to write a shell that ftp a file which is in the local (C:\) drive to a Unix server. Where as i know the IP for the Unix server. i could do this process by using ftp command. pls help me to write as Shell script. Thanks in advance for all of your answers.:b::b: (3 Replies)
Discussion started by: little_wonder
3 Replies

5. UNIX for Dummies Questions & Answers

How to transfer file from Local PC to Unix Directory without FTP!!!

Dear Friends, How to transfer files from my local PC to Unix directory without using FTP. Scenario: Transfer/Upload a file from PC to unix using web browser without using FTP technologies. I heard something like sendunix and sendpc used to transfer files from unix to Desktop and... (1 Reply)
Discussion started by: kk_c2il2
1 Replies

6. Shell Programming and Scripting

How to transfer file from Local PC to UNIX Directory without FTP?

Dear All, i am trying to get the file from windows location to unix location without using FTP and neither thru entering the user id and password. I have one unix form which is running on web application and user is entering the location and file name there now i know the file name and path. So i... (8 Replies)
Discussion started by: ripudaman.singh
8 Replies

7. Shell Programming and Scripting

Downloading file from mainframe to UNIX

Hi All , I need a help regarding file ftp ing from mainframe to unix.Our source file is mainframe file.I need to download the mainframe file to local unix server through unix script.If anyone can help me how we can do it through unix script ,it will be really helpful.Thanks. (7 Replies)
Discussion started by: STCET22
7 Replies

8. Shell Programming and Scripting

Need help how to copy few records from hdfs to UNIX

Hi All , I am facing one issue here...I have a huge file in hadoop file system.Some disk space issues are thr ,thatswhy I want to copy 1st 100 records from hdfs to local unix.I tried below command but it is no working .Its giving error like cat: Unable to write to output stream.if any one can... (2 Replies)
Discussion started by: STCET22
2 Replies

9. UNIX for Advanced & Expert Users

UNIX and HDFS - file systems on same partition.

I am learning Hadoop. As a part of that, Hdfs - Hadoop distributed file system has commands similar to unix where we can create,copy,move files from unix/linux file system to HDFS. My question is 1) how two file systems (unix and hdfs) can coexist on thr same partition.. 2)What if block used... (0 Replies)
Discussion started by: Narendra Eliset
0 Replies

10. UNIX for Beginners Questions & Answers

UNIX and HDFS - file systems on same partition.

I am learning Hadoop. As a part of that, Hdfs - Hadoop distributed file system has commands similar to unix where we can create,copy,move files from unix/linux file system to HDFS. My question is 1) how two file systems (unix and hdfs) can coexist on thr same partition.. 2)What if block... (1 Reply)
Discussion started by: Narendra Eliset
1 Replies
Test::Script(3) 					User Contributed Perl Documentation					   Test::Script(3)

NAME
Test::Script - Basic cross-platform tests for scripts DESCRIPTION
The intent of this module is to provide a series of basic tests for 80% of the testing you will need to do for scripts in the script (or bin as is also commonly used) paths of your Perl distribution. Further, it aims to provide this functionality with perfect platform-compatibility, and in a way that is as unobtrusive as possible. That is, if the program works on a platform, then Test::Script should always work on that platform as well. Anything less than 100% is considered unacceptable. In doing so, it is hoped that Test::Script can become a module that you can safely make a dependency of all your modules, without risking that your module won't on some platform because of the dependency. Where a clash exists between wanting more functionality and maintaining platform safety, this module will err on the side of platform safety. FUNCTIONS
script_compiles script_compiles( 'script/foo.pl', 'Main script compiles' ); The "script_compiles" test calls the script with "perl -c script.pl", and checks that it returns without error. The path it should be passed is a relative unix-format script name. This will be localised when running "perl -c" and if the test fails the local name used will be shown in the diagnostic output. Note also that the test will be run with the same perl interpreter that is running the test script (and not with the default system perl). This will also be shown in the diagnostic output on failure. script_runs script_runs( 'script/foo.pl', 'Main script runs' ); The "script_runs" test executes the script with "perl script.pl" and checks that it returns success. The path it should be passed is a relative unix-format script name. This will be localised when running "perl -c" and if the test fails the local name used will be shown in the diagnostic output. The test will be run with the same perl interpreter that is running the test script (and not with the default system perl). This will also be shown in the diagnostic output on failure. SUPPORT
All bugs should be filed via the bug tracker at <http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Test-Script> For other issues, or commercial enhancement and support, contact the author. AUTHOR
Adam Kennedy <adamk@cpan.org> SEE ALSO
prove, <http://ali.as/> COPYRIGHT
Copyright 2006 - 2009 Adam Kennedy. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. The full text of the license can be found in the LICENSE file included with this module. perl v5.18.2 2009-11-23 Test::Script(3)
All times are GMT -4. The time now is 09:20 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy