Today (Saturday) We will make some minor tuning adjustments to MySQL.

You may experience 2 up to 10 seconds "glitch time" when we restart MySQL. We expect to make these adjustments around 1AM Eastern Daylight Saving Time (EDT) US.


To get latest hdfs file system


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
To get latest hdfs file system

Hi All,

I am having below hdfs file system

Code:
/user/home/dte=2019_01_30/part_1
/user/home/dte=2019_01_30/part_2
/user/home/dte=2019_01_31/part_1

I need to take the latest month hdfs folder while passing date as parameter.
For eg . if i pass as Feb month i.e. 20190201(YYYYMMDD), then i need the output as 2019_01_31 by reading the above hdfs direcotories. I also need to check if there are any data then get the dte as 2019_01_31. For eg /user/home/dte=2019_01_31/part_1 should be more than 0 KB file

My try where i am struck

Code:
final_cnt =`hadoop fs -cat /user/home/dte=2019_01_31/part_1 | wc -l`
if [ $final_cnt >1 ]
then
finat_dte = `hdfs dfs -ls -R /user/home/dte=2019_01_31/ | grep  "^-" | tr -s " " | cut -d' ' -f6-8 | `

Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Read CSV file and delete hdfs, hive and hbase tables
shivamayam
I have a CSV file with hdfs directories, hive tables and hbase tables. 1. first column - hdfs directories 2. second column - hive tables 3. third column - hbase tables I have to check the csv file and look for the first column and delete the hdfs directory from the hdfs path, now...... Shell Programming and Scripting
2
Shell Programming and Scripting
UNIX and HDFS - file systems on same partition.
Narendra Eliset
I am learning Hadoop. As a part of that, Hdfs - Hadoop distributed file system has commands similar to unix where we can create,copy,move files from unix/linux file system to HDFS. My question is 1) how two file systems (unix and hdfs) can coexist on thr same partition.. 2)What if block...... UNIX for Beginners Questions & Answers
1
UNIX for Beginners Questions & Answers
UNIX and HDFS - file systems on same partition.
Narendra Eliset
I am learning Hadoop. As a part of that, Hdfs - Hadoop distributed file system has commands similar to unix where we can create,copy,move files from unix/linux file system to HDFS. My question is 1) how two file systems (unix and hdfs) can coexist on thr same partition.. 2)What if block used...... UNIX for Advanced & Expert Users
0
UNIX for Advanced & Expert Users
How to Keep your core System and personal Data safe while updating to latest distro?
freddie50
Hi everyone, Almost everything is in the title! Which partitions do you keep? Which partitions do you reformat, while doing a clean install? Personaly, I never format /var and /home partitions when I update to latest linux distribution. It has been working quite ok up to now, but I was...... Linux
3
Linux
RSYNC syntax for pushing file with latest system date
tfort73
OK, I am a little new to AIX 5.3 and also to scripting. I have a shell script that I wrote and am having difficulty pushing specific files by the system date. Here is my script: #!/usr/bin/sh RSYNC=/usr/local/bin/rsync SSH=/usr/local/bin/ssh KEY=<path> somekey.key RUSER=mike...... Shell Programming and Scripting
4
Shell Programming and Scripting
Module::Build::Bundling(3)				User Contributed Perl Documentation				Module::Build::Bundling(3)

NAME
Module::Build::Bundling - How to bundle Module::Build with a distribution
SYNOPSIS
# Build.PL use inc::latest 'Module::Build'; Module::Build->new( module_name => 'Foo::Bar', license => 'perl', )->create_build_script;
DESCRIPTION
WARNING -- THIS IS AN EXPERIMENTAL FEATURE In order to install a distribution using Module::Build, users must have Module::Build available on their systems. There are two ways to do this. The first way is to include Module::Build in the "configure_requires" metadata field. This field is supported by recent versions CPAN and CPANPLUS and is a standard feature in the Perl core as of Perl 5.10.1. Module::Build now adds itself to "configure_requires" by default. The second way supports older Perls that have not upgraded CPAN or CPANPLUS and involves bundling an entire copy of Module::Build into the distribution's "inc/" directory. This is the same approach used by Module::Install, a modern wrapper around ExtUtils::MakeMaker for Makefile.PL based distributions. The "trick" to making this work for Module::Build is making sure the highest version Module::Build is used, whether this is in "inc/" or already installed on the user's system. This ensures that all necessary features are available as well as any new bug fixes. This is done using the new inc::latest module. A "normal" Build.PL looks like this (with only the minimum required fields): use Module::Build; Module::Build->new( module_name => 'Foo::Bar', license => 'perl', )->create_build_script; A "bundling" Build.PL replaces the initial "use" line with a nearly transparent replacement: use inc::latest 'Module::Build'; Module::Build->new( module_name => 'Foo::Bar', license => 'perl', )->create_build_script; For authors, when "Build dist" is run, Module::Build will be automatically bundled into "inc" according to the rules for inc::latest. For users, inc::latest will load the latest Module::Build, whether installed or bundled in "inc/".
BUNDLING OTHER CONFIGURATION DEPENDENCIES
The same approach works for other configuration dependencies -- modules that must be available for Build.PL to run. All other dependencies can be specified as usual in the Build.PL and CPAN or CPANPLUS will install them after Build.PL finishes. For example, to bundle the Devel::AssertOS::Unix module (which ensures a "Unix-like" operating system), one could do this: use inc::latest 'Devel::AssertOS::Unix'; use inc::latest 'Module::Build'; Module::Build->new( module_name => 'Foo::Bar', license => 'perl', )->create_build_script; The "inc::latest" module creates bundled directories based on the packlist file of an installed distribution. Even though "inc::latest" takes module name arguments, it is better to think of it as bundling and making available entire distributions. When a module is loaded through "inc::latest", it looks in all bundled distributions in "inc/" for a newer module than can be found in the existing @INC array. Thus, the module-name provided should usually be the "top-level" module name of a distribution, though this is not strictly required. For example, Module::Build has a number of heuristics to map module names to packlists, allowing users to do things like this: use inc::latest 'Devel::AssertOS::Unix'; even though Devel::AssertOS::Unix is contained within the Devel-CheckOS distribution. At the current time, packlists are required. Thus, bundling dual-core modules, including Module::Build, may require a 'forced install' over versions in the latest version of perl in order to create the necessary packlist for bundling. This limitation will hopefully be addressed in a future version of Module::Build. WARNING -- How to Manage Dependency Chains Before bundling a distribution you must ensure that all prerequisites are also bundled and load in the correct order. For Module::Build itself, this should not be necessary, but it is necessary for any other distribution. (A future release of Module::Build will hopefully address this deficiency.) For example, if you need "Wibble", but "Wibble" depends on "Wobble", your Build.PL might look like this: use inc::latest 'Wobble'; use inc::latest 'Wibble'; use inc::latest 'Module::Build'; Module::Build->new( module_name => 'Foo::Bar', license => 'perl', )->create_build_script; Authors are strongly suggested to limit the bundling of additional dependencies if at all possible and to carefully test their distribution tarballs on older versions of Perl before uploading to CPAN.
AUTHOR
David Golden <dagolden@cpan.org> Development questions, bug reports, and patches should be sent to the Module-Build mailing list at <module-build@perl.org>. Bug reports are also welcome at <http://rt.cpan.org/NoAuth/Bugs.html?Dist=Module-Build>.
SEE ALSO
perl(1), inc::latest, Module::Build(3), Module::Build::API(3), Module::Build::Cookbook(3), perl v5.16.3 2014-06-10 Module::Build::Bundling(3)

Featured Tech Videos