Hi
Thanks for the replies. I apologize for the delay.

I tried the following, and It didn't work.
I tried:

Code:

stat -c"%s" FileName.txt

I got:

Code:

ksh: stat:  not found

I tried:

Code:

stat -f FileName.txt

I got:

Code:

ksh: stat:  not found

So here's what I tried. (I tried this in DataStage - on AIX). I created a DataStage code, that will read the and use some simple UNIX commands and do this.

1. Check if the Directory exist, if yes remove all the files inside the directory and the directory.

2. Then check the file size:

Code:

ls -lSr /inputpath/Filename.txt | tail -n1 | awk '{$5=sprintf("%.9f GB", $5/1024^3)} 1' | cut -d' ' -f5 | awk '{printf "%.0f\n", $1}'

This will give me the file size in GB. if the file size is small, then it tells me 0.

(but now I found a new way to get the size: ls -l filename.txt | awk '{print $5}'

3. calculate Split file count (number of files to split): Here I will get the size from above and see how many pieces I can split them. For example, if the file size if 100 GB, then I want to split then into 20 pieces (5GB Each). I will get this number.

4. Get the row count, using wc -l, and then parse to get only the numeric part.

5. The I calculate the rows per split file (I do it this way, because if I split the file just by size then there may be a possibility that the last row may get split). (Formula used: Total Row Count divided by number of files to split. For example, if I have file with 1000 rows, and the size is 100 GB (which means I want to split them into 20 files). So this would give me 1000 divided by 20 equals 50. So this will create 20 files with 50 rows each.

6. The round the result from above to a whole number (higher end).

7. The I use the split command (a simple script) - This will take the file name and the row count for each split file as input argument. This script will first check for correct I/p argument, then creates a directory where I can place the split files, and then uses the split command as

Code:

split -l rowcount file_name /destination/directory/forsplitfile/data_

this will split my actual input files and then store in into the folder "forsplitfile" with prefix "data_"

8. Then I check if all the split file size equals the actual file size, if Yes, then continue, if no, then abort.

9. Then I use my load job to read each split file and load them one by one. once all the files are done, the job completes.

10. If the job fails in between (say split_file_5) then when I re-start my job, it will pick up from where it failed. (split_file_5).

I know this is too much, but can somebody help me put this in a script. I can do this in DataStage with some UNIX commands, but I think what I'm doing wont be a stable solution. So can somebody help me please.

Thanks

---------- Post updated at 03:33 PM ---------- Previous update was at 03:30 PM ----------

Also how do I add checkpoint in the shell script, so when I restart, it can start from where it failed.