Shell Programming and Scripting

Hi

I have a requirement, where i will receive multiple files in a folder (say: /fol1/fol2/). There will be at least 14 to 16 files. The size of the files will different, some may be 80GB or 90GB, some may be less than 5 GB (and the size of the files are very unpredictable). But the names of the files will be have a particular format like "Table1_Insert.dat" , Table1_Update.dat, Table1_delete.dat, Table2_ins.dat, Table2_upd.dat, Table2_del.dat... like this...

I have to read one file at a time, check the size of the file (in GB), if the file size is greater than 90 GB (file size wont be more than 100GB always), then split the files into 5GB. So if the file size is 90 GB, then it should split the source file into 18 sub files (like TT_table1_ins.dataa, TT_Table1_ins.datab , TT_Table1_ins.datac... etc)

I want my script to take only one input argument - just the file name (with the path).

I know we can do this using split -l command, but i need some help. Can somebody help me with a script. I'm very new to shell scripting. I can understand the commands but cannot write a script...

Thanks

Is this a homework assignment?

No this is not homework... may be i gave too much explanation... thats why it seems like home work... but this is my work...

Why do you need a script? What stops you from using it as a one line command like

Code:

[ $(stat -c"%s" Table1_ins.dat) -gt 90000000000 ] && split -b5000000000 -a1 --verbose Table1_ins.dat TT_table1_ins.dat

This splits by bytes and could split half lines; use -l with an average line length to keep entire lines.

You did not give too much information! (I've never seen anyone make that mistake in these forums!)

If the 1-liner RudiC gave you works, you can put that into a script.

If it doesn't, show us the diagnostic messages it prints and tell us what operating system and shell you're using. (The stat utility and the split --verbose option are not available on all operating systems.)

Some splits (GNU for example) support the --line-bytes=SIZE to keep whole lines eg:

Code:

cd /fol1/fol2
for file in *.dat
do
    if [ $(stat -c"%s" "$file") -gt 90000000000 ]
    then
        split --line-bytes=5000000000 -a1 --verbose "$file" "TT_$file"
    fi
done

Hi
Thanks for the replies. I apologize for the delay.

I tried the following, and It didn't work.
I tried:

Code:

stat -c"%s" FileName.txt

I got:

Code:

ksh: stat:  not found

I tried:

Code:

stat -f FileName.txt

I got:

Code:

ksh: stat:  not found

So here's what I tried. (I tried this in DataStage - on AIX). I created a DataStage code, that will read the and use some simple UNIX commands and do this.

1. Check if the Directory exist, if yes remove all the files inside the directory and the directory.

2. Then check the file size:

Code:

ls -lSr /inputpath/Filename.txt | tail -n1 | awk '{$5=sprintf("%.9f GB", $5/1024^3)} 1' | cut -d' ' -f5 | awk '{printf "%.0f\n", $1}'

This will give me the file size in GB. if the file size is small, then it tells me 0.

(but now I found a new way to get the size: ls -l filename.txt | awk '{print $5}'

3. calculate Split file count (number of files to split): Here I will get the size from above and see how many pieces I can split them. For example, if the file size if 100 GB, then I want to split then into 20 pieces (5GB Each). I will get this number.

4. Get the row count, using wc -l, and then parse to get only the numeric part.

5. The I calculate the rows per split file (I do it this way, because if I split the file just by size then there may be a possibility that the last row may get split). (Formula used: Total Row Count divided by number of files to split. For example, if I have file with 1000 rows, and the size is 100 GB (which means I want to split them into 20 files). So this would give me 1000 divided by 20 equals 50. So this will create 20 files with 50 rows each.

6. The round the result from above to a whole number (higher end).

7. The I use the split command (a simple script) - This will take the file name and the row count for each split file as input argument. This script will first check for correct I/p argument, then creates a directory where I can place the split files, and then uses the split command as

Code:

split -l rowcount file_name /destination/directory/forsplitfile/data_

this will split my actual input files and then store in into the folder "forsplitfile" with prefix "data_"

8. Then I check if all the split file size equals the actual file size, if Yes, then continue, if no, then abort.

9. Then I use my load job to read each split file and load them one by one. once all the files are done, the job completes.

10. If the job fails in between (say split_file_5) then when I re-start my job, it will pick up from where it failed. (split_file_5).

I know this is too much, but can somebody help me put this in a script. I can do this in DataStage with some UNIX commands, but I think what I'm doing wont be a stable solution. So can somebody help me please.

Thanks

---------- Post updated at 03:33 PM ---------- Previous update was at 03:30 PM ----------

Also how do I add checkpoint in the shell script, so when I restart, it can start from where it failed.