UNIX for Advanced & Expert Users

Hello,

I will be processing 47000 zip files each of size 5.5 MB.

Following operation will be performed for each zip file:

1. Unzip the file.

2. Read each text and binary file.

3. Preprocess the files.

4. Store the files in different tables in Oracle.

5. Delete the extracted files.

All of this operation must be completed in 4 hours.

To speed-up this process, is it possible to keep the extracted files in RAM instead of hard-disk?

It is possible to set aside some amount of RAM and treat that as hard-disk on Windows; to resolve this problem can something on similar lines be done in UNIX also?

Thanks,

Vaibhav

Yes, it's called memory mapped files. You'll have to use Pro*C and call mmap.

try man mmap - you can then operate on all of the file in memory with norma data pointers.

edit>
On second though you could run 'parallel' by writing a script to do your work then dropping off 10 or 12 jobs and waiting sort of like this:

Code:

#!/bin/ksh
let cnt=0
while read file
do
     let cnt=$cnt+1
     myscript.sh  "$file" &
     if [[ $cnt -eq 12 ]]; then
        wait
        let cnt=0
     fi
done < list_of_filenames

Oracle is very good at handling a lot of jobs at one time.

Thanks for the information.

Is it possible to simply create another drive in the RAM?

That is this drive will be treated by UNIX OS as a "physical" drive?

Please let me know.

-- Vaibhav

Yes, this is possible. Which UNIX?

Newer versions of Linux will have a /dev/shm/ directory which already does what you want.

The version is UNIX 11.

I need to complete the processing of every zip file in about 300 ms.

The server has 4 dual Intel processors.

Please don't make us guess. UNIX is not a brand name. Is this HP UNIX? here is how to use a RAM drive in HP UNIX, though they call it "unsupported and dangerous"...

Since it has four processors, you could try running four simultaneous jobs. That way, one job would only need to complete in 1200ms to get the required throughput.

If you have enough buffer cache, RAM disk versus normal disk shouldn't matter a huge amount. All changes get cached in RAM anyway.

Sorry about that...

It is HP UNIX.

The server is HP RP3440.

It has 8 GB of RAM.