To get latest hdfs file system


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
To get latest hdfs file system

Hi All,

I am having below hdfs file system

Code:
/user/home/dte=2019_01_30/part_1
/user/home/dte=2019_01_30/part_2
/user/home/dte=2019_01_31/part_1

I need to take the latest month hdfs folder while passing date as parameter.
For eg . if i pass as Feb month i.e. 20190201(YYYYMMDD), then i need the output as 2019_01_31 by reading the above hdfs direcotories. I also need to check if there are any data then get the dte as 2019_01_31. For eg /user/home/dte=2019_01_31/part_1 should be more than 0 KB file

My try where i am struck

Code:
final_cnt =`hadoop fs -cat /user/home/dte=2019_01_31/part_1 | wc -l`
if [ $final_cnt >1 ]
then
finat_dte = `hdfs dfs -ls -R /user/home/dte=2019_01_31/ | grep  "^-" | tr -s " " | cut -d' ' -f6-8 | `

Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Read CSV file and delete hdfs, hive and hbase tables
shivamayam
I have a CSV file with hdfs directories, hive tables and hbase tables. 1. first column - hdfs directories 2. second column - hive tables 3. third column - hbase tables I have to check the csv file and look for the first column and delete the hdfs directory from the hdfs path, now...... Shell Programming and Scripting
2
Shell Programming and Scripting
UNIX and HDFS - file systems on same partition.
Narendra Eliset
I am learning Hadoop. As a part of that, Hdfs - Hadoop distributed file system has commands similar to unix where we can create,copy,move files from unix/linux file system to HDFS. My question is 1) how two file systems (unix and hdfs) can coexist on thr same partition.. 2)What if block...... UNIX for Beginners Questions & Answers
1
UNIX for Beginners Questions & Answers
UNIX and HDFS - file systems on same partition.
Narendra Eliset
I am learning Hadoop. As a part of that, Hdfs - Hadoop distributed file system has commands similar to unix where we can create,copy,move files from unix/linux file system to HDFS. My question is 1) how two file systems (unix and hdfs) can coexist on thr same partition.. 2)What if block used...... UNIX for Advanced & Expert Users
0
UNIX for Advanced & Expert Users
Downloading hdfs file to local UNIX through UNIX script
STCET22
Hi All , I am very new to unix script.I am aware of unix commands but never put together in unix script level.If any one can suggest me technical guidance in the below scenario that will highly beneficial. Data have been already migrated from mainframe to Hadoop file system(HDFS).HDFS server...... Shell Programming and Scripting
15
Shell Programming and Scripting
RSYNC syntax for pushing file with latest system date
tfort73
OK, I am a little new to AIX 5.3 and also to scripting. I have a shell script that I wrote and am having difficulty pushing specific files by the system date. Here is my script: #!/usr/bin/sh RSYNC=/usr/local/bin/rsync SSH=/usr/local/bin/ssh KEY=<path> somekey.key RUSER=mike...... Shell Programming and Scripting
4
Shell Programming and Scripting
Ubuntu

Ubuntu is a free, open-source Linux distribution based on Debian Linux; and Ubuntu is named after the African philosophy of ubuntu, which Canonical translates as "humanity to others" or "I am what I am because of who we all are". Ubuntu is developed by Canonical and the community under a "meritocratic" governance model. Canonical provides security updates and support for each Ubuntu release, starting from the release date and until the release reaches its designated end-of-life (EOL) date. Canonical generates revenue through the sale of premium services associated with Ubuntu. Ubuntu is generally released to the public in three editions: Desktop, Server, and Core (for IoT devices and robots). Ubuntu is a popular operating system for cloud computing and supports OpenStack. Ubuntu is released every six months with long-term support (LTS) releases every two years.
Please enjoy and help our forum community by "showing your work" and posting your code, even when your code is not working as expected. To help others help you, please wrap your code blocks, sample input, sample output, error messages, and other data in CODE tags and wrap your short commands and short data objects in ICODE tags. We were all beginners in the beginning. If you have any questions about how to register or how to post, please contact us in Live Chat. Thank you and enjoy this "forever free" technical support community for UNIX, Linux and computer information technology in general.