To check file size in hadoop


 
Thread Tools Search this Thread
Top Forums Programming To check file size in hadoop
# 1  
Old 12-15-2016
To check file size in hadoop

Hi Guys,

i am writng a script to check if the file size is lessthan limit say 10KB then exit
i need to check in hadoop file system , can someone advise

Code:
Minsize=10
for file in `hadoop fs -ls path/*`
do
Actualsize=$(du -k "$file" | cut -f 1)
if [ $Actualsize -lt $Minsize ]; then
    echo "File generated incorrectly for $file : Filesize - $Actualsize KB "
    echo "Exiting from Script file size found less than 10KB"
  exit 1;
fi
done

# 2  
Old 12-15-2016
Would this thread be a useful start?

File name and its count


Robin
# 3  
Old 12-15-2016
Sorry that didnt help, i have made some changes like below but getting error

Code:
Minsize=10
for Actualsize in `hadoop fs -du -h /path | cut -d" " -f1`
do
if [ $Actualsize -lt $Minsize ]; then
    echo "File generated incorrectly for $file : Filesize - $Actualsize KB "
    echo "Exiting from Script file size found less than 10KB"
  exit 1;
fi
done

integer expression expected


Last edited by rohit_shinez; 12-15-2016 at 11:00 PM..
# 4  
Old 12-16-2016
Well, what value does $Actualsize actually end up being?
# 5  
Old 12-17-2016
Its providing me decimal value for e.g. 1.5
# 6  
Old 12-17-2016
As indicated by that error message, you seem to run bash - although you failed to mention that explicitly, by the way. Decimal numbers can't be processed by it.
So - why don't you use a command / tool that IS capable of calculating with floating point numbers?

Last edited by RudiC; 12-17-2016 at 02:29 PM..
# 7  
Old 12-17-2016
Hi,

Thanks for your suggestion, is there a way to round the decimal value and compare
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

File size check

I am trying to check whether two files are empty or not using below if condition but its checking for only one file if ] Again I tried if && ] Need your assistance (2 Replies)
Discussion started by: Aditya_001
2 Replies

2. Shell Programming and Scripting

Check a file size

I'm doing a script thats check if mylogfile.log is bigger then 5000 but i dont know how to write it. thanks in avance. (6 Replies)
Discussion started by: Froob
6 Replies

3. Shell Programming and Scripting

Check the presence of file >size?

Hi, Following script work fine: #!/bin/bash FILE=$1 if ; then echo Yay else echo Boo fi But I would like to add another condition that if FILE... (3 Replies)
Discussion started by: nrjrasaxena
3 Replies

4. Shell Programming and Scripting

check the file size

if ; then cp /tmp/testfolder/*.* ~/new/logs/ else echo "No files today" exit fi The problem is this doen't work when there is more than 1 file. Please tell me how to take the latest file and check the size of the file in a directory (1 Reply)
Discussion started by: sandy1028
1 Replies

5. Shell Programming and Scripting

Check the file size - help

I want to write a batch job (ksh) with the following requirement we have file feeds coming to our system from other team, if the file size is greater than expected then we dont need to process the file for the day and need to archive the file and send email notification to the manager saying... (5 Replies)
Discussion started by: sithara
5 Replies

6. Shell Programming and Scripting

Check for file size is zero or not.

I have following script on AIX/KSH if ] ; then echo "filename exists and is > 0 bytes" else echo "filename does not exist or is zero length" fi It is not working. What is wrong here??? (3 Replies)
Discussion started by: Hangman2
3 Replies

7. Shell Programming and Scripting

To check file size

Hi All, I am in small problem.. i have one script which transfers some big files to my ftp usign normal command like put .... my problem is how to check whether my file have been transferred successfully on ftp or not... i know only inside ftp we have option like 'size' command which... (2 Replies)
Discussion started by: Shahul
2 Replies

8. Shell Programming and Scripting

file size check

How can I perform size check of any character file(which switch)? For example: I have to perform certain actions if file size is not zero. How can I do that? Is this syntax fine? if test ! -z $filename then fi (2 Replies)
Discussion started by: malaymaru
2 Replies

9. UNIX for Dummies Questions & Answers

Check file size

I need a unix script that will check the size of multiple files in the same directory or from a text file. (6 Replies)
Discussion started by: alnita
6 Replies
Login or Register to Ask a Question