To get latest hdfs file system


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting To get latest hdfs file system
# 1  
Old 05-15-2019
To get latest hdfs file system

Hi All,

I am having below hdfs file system

Code:
/user/home/dte=2019_01_30/part_1
/user/home/dte=2019_01_30/part_2
/user/home/dte=2019_01_31/part_1

I need to take the latest month hdfs folder while passing date as parameter.
For eg . if i pass as Feb month i.e. 20190201(YYYYMMDD), then i need the output as 2019_01_31 by reading the above hdfs direcotories. I also need to check if there are any data then get the dte as 2019_01_31. For eg /user/home/dte=2019_01_31/part_1 should be more than 0 KB file

My try where i am struck

Code:
final_cnt =`hadoop fs -cat /user/home/dte=2019_01_31/part_1 | wc -l`
if [ $final_cnt >1 ]
then
finat_dte = `hdfs dfs -ls -R /user/home/dte=2019_01_31/ | grep  "^-" | tr -s " " | cut -d' ' -f6-8 | `

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell Script for HDFS Ingestion Using JDBC

Peers, I was in process of building a script that connects to salesforce using jdbc and pull the data using spark and process in hive table. During this process I have encountered a problem where and variable assigned with hadoop command that list files in Azure Data lake is not parsing the... (2 Replies)
Discussion started by: jg355187
2 Replies

2. Shell Programming and Scripting

Read CSV file and delete hdfs, hive and hbase tables

I have a CSV file with hdfs directories, hive tables and hbase tables. 1. first column - hdfs directories 2. second column - hive tables 3. third column - hbase tables I have to check the csv file and look for the first column and delete the hdfs directory from the hdfs path, now... (2 Replies)
Discussion started by: shivamayam
2 Replies

3. UNIX for Beginners Questions & Answers

Help with HDFS Linux permission

Hi, I am unable to change the permissions for a directory in HDFS. from what i understand acl's supersede all other permissions. even if a directory is not owned by me, but there is an acl for me with rwx then i must be able to change the permissions of that directory. Please find the... (8 Replies)
Discussion started by: desind
8 Replies

4. UNIX for Beginners Questions & Answers

UNIX and HDFS - file systems on same partition.

I am learning Hadoop. As a part of that, Hdfs - Hadoop distributed file system has commands similar to unix where we can create,copy,move files from unix/linux file system to HDFS. My question is 1) how two file systems (unix and hdfs) can coexist on thr same partition.. 2)What if block... (1 Reply)
Discussion started by: Narendra Eliset
1 Replies

5. UNIX for Advanced & Expert Users

UNIX and HDFS - file systems on same partition.

I am learning Hadoop. As a part of that, Hdfs - Hadoop distributed file system has commands similar to unix where we can create,copy,move files from unix/linux file system to HDFS. My question is 1) how two file systems (unix and hdfs) can coexist on thr same partition.. 2)What if block used... (0 Replies)
Discussion started by: Narendra Eliset
0 Replies

6. Shell Programming and Scripting

Need help how to copy few records from hdfs to UNIX

Hi All , I am facing one issue here...I have a huge file in hadoop file system.Some disk space issues are thr ,thatswhy I want to copy 1st 100 records from hdfs to local unix.I tried below command but it is no working .Its giving error like cat: Unable to write to output stream.if any one can... (2 Replies)
Discussion started by: STCET22
2 Replies

7. Shell Programming and Scripting

Downloading hdfs file to local UNIX through UNIX script

Hi All , I am very new to unix script.I am aware of unix commands but never put together in unix script level.If any one can suggest me technical guidance in the below scenario that will highly beneficial. Data have been already migrated from mainframe to Hadoop file system(HDFS).HDFS server... (15 Replies)
Discussion started by: STCET22
15 Replies

8. Linux

How to Keep your core System and personal Data safe while updating to latest distro?

Hi everyone, Almost everything is in the title! Which partitions do you keep? Which partitions do you reformat, while doing a clean install? Personaly, I never format /var and /home partitions when I update to latest linux distribution. It has been working quite ok up to now, but I was... (3 Replies)
Discussion started by: freddie50
3 Replies

9. Shell Programming and Scripting

RSYNC syntax for pushing file with latest system date

OK, I am a little new to AIX 5.3 and also to scripting. I have a shell script that I wrote and am having difficulty pushing specific files by the system date. Here is my script: #!/usr/bin/sh RSYNC=/usr/local/bin/rsync SSH=/usr/local/bin/ssh KEY=<path> somekey.key RUSER=mike... (4 Replies)
Discussion started by: tfort73
4 Replies

10. Solaris

Who has "Solaris 10 Advanced System Administration(SA-202-S10)" latest revision doc?

Dear All, I am going to examine SA202 but I got SA-202-S10 revision A.1 so it might be out of date. Who has "Solaris 10 Advanced System Administration(SA-202-S10)" latest revision document? Could you share it ? Thank you in advance (2 Replies)
Discussion started by: unitipon
2 Replies
Login or Register to Ask a Question
secure_sid_scripts(5)						File Formats Manual					     secure_sid_scripts(5)

NAME
secure_sid_scripts - controls whether setuid and setgid bits on scripts are honored VALUES
Failsafe Default Allowed values Recommended values DESCRIPTION
This tunable controls whether and bits on executable scripts have any effect. Honoring on scripts make a system vulnerable to attack by malicious users. The default value for this variable is 1, indicating that bits are to be ignored by the execve(2) system call for higher security. The tunable can be set to 0 for a compatibility with older releases at the expense of security. Hewlett-Packard strongly recommends that you not change the value of this tunable unless there is an urgent need to do so. When a script with bits is executed, the kernel generates the following error message to both the terminal controlling and the system log. (To view the error message, use dmesg(1M) or inspect Who is Expected to Change This Tunable? Administrator. Restrictions on Changing Changes to this tunable take effect for new scripts started after the change. When Should the Value of This Tunable Be Changed? This tunable controls operational modes rather than data structure sizes and limits. The appropriate setting for a system depends on whether you consider security or compatibility to be most important. A value of is compatible with previous releases of HP-UX, but it is also less secure. A value of provides security against race condition attacks exploiting scripts. What Are the Side Effects of Changing the Value This tunable controls only executable scripts (not programs) with bit set. HP-UX does not ship with any such scripts. If the customer wishes to use scripts, third party applications such as or can be used. Alternatively, the shell script can be wrapped in a simple C pro- gram that runs the shell script with appropriate permissions: What Other Tunable Values Should Be Changed at the Same Time? None. WARNINGS
None. All HP-UX kernel tunable parameters are release specific. This parameter may be removed or have its meaning changed in future releases of HP-UX. Installation of optional kernel software, from HP or other vendors, may cause changes to tunable parameter values. After installation, some tunable parameters may no longer be at the default or recommended values. For information about the effects of installation on tun- able values, consult the documentation for the kernel software being installed. For information about optional kernel software that was factory installed on your system, see at FILES
AUTHOR
was developed by HP. SEE ALSO
chmod(1), execve(2), kctune(1M). Tunable Kernel Parameters secure_sid_scripts(5)