Sponsored Content
Full Discussion: Direct read / load
Top Forums Shell Programming and Scripting Direct read / load Post 302911986 by ManoharMa on Tuesday 5th of August 2014 04:12:32 AM
Old 08-05-2014
Direct read / load

Guys,

I wrote a script to do the following -
  1. Read a file that is placed into a directory and depending upon the contents, load into one or multiple tables into Oracle.
  2. After the file is loaded into the table, depending upon the exit status of the process I will either delete the file or mark the record entry for that file as incomplete record in the table.

Initially, I was told that there would only be few files in a 15 minute time span and I cron-ed this script to run for every 5 mins and it was fast enough to load it in few minutes. Now, they are sending huge files and on couple occasions, the script ran longer than 10 minutes.

Now, Im running into an issue that I cant seem to grapple with on multiple levels.
  1. First, how do I find which records ran and which one did not ?
  2. How do I identify as to which records got completed and which are pending and which one did not even process ?
  3. how do I implement some checks and balances so that whatever is placed in the directory is loaded into the table.

I am sure a great many of you might have ran into this and want to learn as to what could be the best practice here. I am thinking of implementing a thread based approach but I never wrote thread programming in POSIX.

Please advise

Last edited by rbatte1; 08-05-2014 at 07:23 AM.. Reason: Spelling and converting to numbered LIST tags
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to read a dynamically changing file and load into Oracle?

I have a tab delimited file which has 27 character fields. The file needs to be loaded into an Oracle table. But the challenge is that everytime the file comes it may or may not have values in all 27 fields. Column Definition of the 27 fields: TYPE: Char (1) NAME: Char (30) CUSTOM_VAL: Char... (8 Replies)
Discussion started by: madhunk
8 Replies

2. Shell Programming and Scripting

Need help in wrting Load Script for a Load-Resume type of load.

hi all need your help. I am wrting a script that will load data into the table. then on another load will append the data into the existing table. Regards Ankit (1 Reply)
Discussion started by: ankitgupta
1 Replies

3. Shell Programming and Scripting

shell script to read data from text file and to load it into a table in TOAD

Hi....can you guys help me out in this script?? Below is a portion text file and it contains these: GEF001 000093625 MKL002510 000001 000000 000000 000000 000000 000000 000001 GEF001 000093625 MKL003604 000001 000000 000000 000000 000000 000000 000001 GEF001 000093625 MKL005675 000001... (1 Reply)
Discussion started by: pallavishetty
1 Replies

4. Shell Programming and Scripting

Connect:Direct

Hello all, I have a requirement to transfer files to mainframe usinf NDM connect:direct. So can anybody provide me a sample shell script on how to call a connect;direct script by providing filename as a parameter please? Thanks, Ajay (0 Replies)
Discussion started by: ajaykumar4534
0 Replies

5. Shell Programming and Scripting

How to read file and load data into a script as variable

I need to read a text file that contain columns of data, i need to read 1st column as a function to call, and others are the data i need to get into a ksh script. I am quite new to ksh scripting, i am not very sure how to read each row line by line and the data in each columns of that line, set... (3 Replies)
Discussion started by: gavin_L
3 Replies

6. Shell Programming and Scripting

Manipulating sed Direct Input to Direct Output

Hi guys, been scratching round the forums and my mountain of resources. Maybe I havn't read deep enough My question is not how sed edits a stream and outputs it to a file, rather something like this below: I have a .txt with some text in it :rolleyes: abc:123:xyz 123:abc:987... (7 Replies)
Discussion started by: the0nion
7 Replies

7. UNIX for Dummies Questions & Answers

VPS has load 200, httpd load no activity, netstat nothing

Hello, on my hostserver i see one VPS of mine got load of 200.00 and netstat nothing (not a single blank line on netstat command) after some time, netstat started showing connections, but i see no excessive IP connections. tail -f /var/log/httpd/access_log shows no activity /var/log/messages ;... (1 Reply)
Discussion started by: postcd
1 Replies

8. UNIX for Beginners Questions & Answers

Direct Printing In AIX

Hello. I recently upgraded to a new Power 8 server and running AIX 7.1. I migrated from an IBM P520 and AIX 5.5. My application on the P520 works best if I direct print, it doesn't work well with spooling. My IBM rep set up the new Power 8 server with spooling which is causing an issue.... (4 Replies)
Discussion started by: ldavis1080
4 Replies

9. UNIX for Beginners Questions & Answers

Creating script to read many files and load into database via single control file

Hi, I have many files but with only 2 names , I want to load the data of that file into database through sqlldr with single control file. how can i do that ????? Example: switch_file switch_file billing_file billing_file now these files should be loaded into same database but different... (1 Reply)
Discussion started by: niti_sharma
1 Replies
lib(3pm)						 Perl Programmers Reference Guide						  lib(3pm)

NAME
lib - manipulate @INC at compile time SYNOPSIS
use lib LIST; no lib LIST; DESCRIPTION
This is a small simple module which simplifies the manipulation of @INC at compile time. It is typically used to add extra directories to perl's search path so that later "use" or "require" statements will find modules which are not located on perl's default search path. Adding directories to @INC The parameters to "use lib" are added to the start of the perl search path. Saying use lib LIST; is almost the same as saying BEGIN { unshift(@INC, LIST) } For each directory in LIST (called $dir here) the lib module also checks to see if a directory called $dir/$archname/auto exists. If so the $dir/$archname directory is assumed to be a corresponding architecture specific directory and is added to @INC in front of $dir. lib.pm also checks if directories called $dir/$version and $dir/$version/$archname exist and adds these directories to @INC. The current value of $archname can be found with this command: perl -V:archname The corresponding command to get the current value of $version is: perl -V:version To avoid memory leaks, all trailing duplicate entries in @INC are removed. Deleting directories from @INC You should normally only add directories to @INC. If you need to delete directories from @INC take care to only delete those which you added yourself or which you are certain are not needed by other modules in your script. Other modules may have added directories which they need for correct operation. The "no lib" statement deletes all instances of each named directory from @INC. For each directory in LIST (called $dir here) the lib module also checks to see if a directory called $dir/$archname/auto exists. If so the $dir/$archname directory is assumed to be a corresponding architecture specific directory and is also deleted from @INC. Restoring original @INC When the lib module is first loaded it records the current value of @INC in an array @lib::ORIG_INC. To restore @INC to that value you can say @INC = @lib::ORIG_INC; CAVEATS
In order to keep lib.pm small and simple, it only works with Unix filepaths. This doesn't mean it only works on Unix, but non-Unix users must first translate their file paths to Unix conventions. # VMS users wanting to put [.stuff.moo] into # their @INC would write use lib 'stuff/moo'; NOTES
In the future, this module will likely use File::Spec for determining paths, as it does now for Mac OS (where Unix-style or Mac-style paths work, and Unix-style paths are converted properly to Mac-style paths before being added to @INC). If you try to add a file to @INC as follows: use lib 'this_is_a_file.txt'; "lib" will warn about this. The sole exceptions are files with the ".par" extension which are intended to be used as libraries. SEE ALSO
FindBin - optional module which deals with paths relative to the source file. PAR - optional module which can treat ".par" files as Perl libraries. AUTHOR
Tim Bunce, 2nd June 1995. "lib" is maintained by the perl5-porters. Please direct any questions to the canonical mailing list. Anything that is applicable to the CPAN release can be sent to its maintainer, though. Maintainer: The Perl5-Porters <perl5-porters@perl.org> Maintainer of the CPAN release: Steffen Mueller <smueller@cpan.org> COPYRIGHT AND LICENSE
This package has been part of the perl core since perl 5.001. It has been released separately to CPAN so older installations can benefit from bug fixes. This package has the same copyright and license as the perl core. perl v5.16.3 2014-06-17 lib(3pm)
All times are GMT -4. The time now is 01:24 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy