Unix/Linux Go Back    


Programming Post questions about C, C++, Java, SQL, and other programming languages here.

To check file size in hadoop

Programming


Tags
solved

Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 12-15-2016
rohit_shinez rohit_shinez is offline
Registered User
 
Join Date: Jul 2013
Last Activity: 9 March 2017, 5:12 AM EST
Posts: 157
Thanks: 53
Thanked 0 Times in 0 Posts
To check file size in hadoop

Hi Guys,

i am writng a script to check if the file size is lessthan limit say 10KB then exit
i need to check in hadoop file system , can someone advise


Code:
Minsize=10
for file in `hadoop fs -ls path/*`
do
Actualsize=$(du -k "$file" | cut -f 1)
if [ $Actualsize -lt $Minsize ]; then
    echo "File generated incorrectly for $file : Filesize - $Actualsize KB "
    echo "Exiting from Script file size found less than 10KB"
  exit 1;
fi
done

Sponsored Links
    #2  
Old Unix and Linux 12-15-2016
rbatte1 rbatte1 is offline Forum Staff  
Root armed
 
Join Date: Jun 2007
Last Activity: 25 May 2017, 7:38 AM EDT
Location: Lancashire, UK
Posts: 3,102
Thanks: 1,301
Thanked 590 Times in 534 Posts
Would this thread be a useful start?

File name and its count


Robin
Sponsored Links
    #3  
Old Unix and Linux 12-15-2016
rohit_shinez rohit_shinez is offline
Registered User
 
Join Date: Jul 2013
Last Activity: 9 March 2017, 5:12 AM EST
Posts: 157
Thanks: 53
Thanked 0 Times in 0 Posts
Sorry that didnt help, i have made some changes like below but getting error


Code:
Minsize=10
for Actualsize in `hadoop fs -du -h /path | cut -d" " -f1`
do
if [ $Actualsize -lt $Minsize ]; then
    echo "File generated incorrectly for $file : Filesize - $Actualsize KB "
    echo "Exiting from Script file size found less than 10KB"
  exit 1;
fi
done

integer expression expected


Last edited by rohit_shinez; 12-15-2016 at 10:00 PM..
    #4  
Old Unix and Linux 12-16-2016
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 24 May 2017, 3:30 PM EDT
Location: Saskatchewan
Posts: 22,176
Thanks: 1,092
Thanked 4,171 Times in 3,860 Posts
Well, what value does $Actualsize actually end up being?
Sponsored Links
    #5  
Old Unix and Linux 12-17-2016
rohit_shinez rohit_shinez is offline
Registered User
 
Join Date: Jul 2013
Last Activity: 9 March 2017, 5:12 AM EST
Posts: 157
Thanks: 53
Thanked 0 Times in 0 Posts
Its providing me decimal value for e.g. 1.5
Sponsored Links
    #6  
Old Unix and Linux 12-17-2016
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 25 May 2017, 2:59 AM EDT
Location: Aachen, Germany
Posts: 10,818
Thanks: 272
Thanked 3,323 Times in 3,060 Posts
As indicated by that error message, you seem to run bash - although you failed to mention that explicitly, by the way. Decimal numbers can't be processed by it.
So - why don't you use a command / tool that IS capable of calculating with floating point numbers?

Last edited by RudiC; 12-17-2016 at 01:29 PM..
Sponsored Links
    #7  
Old Unix and Linux 12-17-2016
rohit_shinez rohit_shinez is offline
Registered User
 
Join Date: Jul 2013
Last Activity: 9 March 2017, 5:12 AM EST
Posts: 157
Thanks: 53
Thanked 0 Times in 0 Posts
Hi,

Thanks for your suggestion, is there a way to round the decimal value and compare
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
check the file size sandy1028 Shell Programming and Scripting 1 11-11-2010 06:16 AM
Check for file size is zero or not. Hangman2 Shell Programming and Scripting 3 04-14-2009 04:08 PM
To check file size Shahul Shell Programming and Scripting 2 09-15-2008 11:53 AM
file size check malaymaru Shell Programming and Scripting 2 10-19-2005 12:26 AM



All times are GMT -4. The time now is 02:04 PM.