Sponsored Content
Top Forums Shell Programming and Scripting shellscript to read data from txt file and import to oracle db Post 302583605 by robot_mas on Wednesday 21st of December 2011 12:04:31 AM
Old 12-21-2011
shellscript to read data from txt file and import to oracle db

Hi all,
Help needed urgently.
I am currently writing a shellscript to read data/record from a flat file (.txt) file, and import/upload the data to oracle database. The script is working fine, but it takes too long time (for 18000 records, it takes around 90 mins).

I guess it takes so long time to insert to db is because the login to db each time after reading one line of record from txt file (sqlplus -s username/password <<EOF)

login to db each time before insert statement because i want switch the environment to oracle db, after insert, i will exit, so that it will point me back to the backend server to get the dir.txt file.

Is this the correct way?
Is there any way/command to use so that no need to switch between environment? or meaning to say, no need to login to db each time when i want to insert a row of data.

Hope somebody can help as i am newbie to shellscript. thx a lot in advance!!!

Below is the codes:
Check if $FILE exists or not
Code:
if test ! -f "dir.txt"
then
echo "Error - $FILE not found or mcelog is not configured for 64
bit Linux systems."
exit 1
else
echo "dir.txt file exist"
fi
 
#start for dir.txt
FILENAME='dir.txt'
count=0
cat $FILENAME | while read LINE
do
 
dname=`echo "$LINE" | awk '{ print substr( $0, 0, 44 ) }'|sed "s/\'/\'\'/g"`
dicnum1=`echo "$LINE" | awk '{ print substr( $0, 45, 26 ) }'` 
dposition=`echo "$LINE" | awk '{ print substr( $0, 71, 26 ) }'|sed "s/\'/\'\'/g"`
dcompnm=`echo "$LINE" | awk '{ print substr( $0, 97, 8 ) }'|sed "s/\'/\'\'/g"`
dcpicno=`echo "$LINE" | awk '{ print substr( $0, 105, 14 ) }'`
dcpname=`echo "$LINE" | awk '{ print substr( $0, 119, 46 ) }'|sed "s/\'/\'\'/g"` 
dcprel=`echo "$LINE" | awk '{ print substr( $0, 165, 20 ) }'`
NOW=$(date +"%d/%m/%Y %H:%M:%S")
 
sqlplus -s username/password <<EOF
set scan off;
Insert into T_DIR_TEST (INSERT_DATE,S_NAME,S_NEW_IC,S_POSITION,S_COMPANY,CP_NAME,CP_IC_NO,CP_REL) VALUES (to_date('$NOW','dd/mm/yyyy HH24:MI:SS'),'$dname','$dicnum1','$dposition','$dcompnm','$dcpname','$dcpicno','$dcprel');
Exit
EOF
done


Last edited by Franklin52; 12-21-2011 at 04:15 AM.. Reason: Please use code tags for code and data samples, thank you
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Import data from compressed file

HI I need to import data from a file which is in comressed format but system doesn't have enough space to uncompress file Is there any way so that i can do import from compressed file. (4 Replies)
Discussion started by: ap_gore79
4 Replies

2. Shell Programming and Scripting

txt file to oracle database

hiya, i have a query: i want to read a file which contains: 2005/02/21 16:56:54.301: 111 PS (200, 10) sent <log instrument="FXA.ROSS"... (9 Replies)
Discussion started by: jorhul
9 Replies

3. Shell Programming and Scripting

shellscript.query Oracle table..populate in a text file

Hi Guys, I'm new to this forum as well as to UNIX shell scripting. I'm looking for a shellscript to query an Oracle database table and populate the result set of the query in a text file. Could you someone help me out with a sample code? Thanks, Bhagat (7 Replies)
Discussion started by: bhagat.singh-j
7 Replies

4. Programming

import .txt and split word into array C

Hi, if I want to import .txt file that contain information and the number separate by space how can I split and put into array In C Example of .txt file 3 Aqaba 49789 10000 5200 25.78 6987 148976 12941 15.78 99885 35262 2501 22.98 Thank (3 Replies)
Discussion started by: guidely
3 Replies

5. Shell Programming and Scripting

Want to read data from a file name.txt and search it in another file and then matching...

Hi Frnds... I have an input file name.txt and another file named as source.. name.txt is having only one column and source is having around 25 columns...i need to read from name.txt line by line and search it in source file and then save the result in results file.. I have a rough idea about the... (15 Replies)
Discussion started by: ektubbe
15 Replies

6. Shell Programming and Scripting

How to import a value from txt file

Hello, I want to give a value from a txt file to my variable at my ksh script. I`ve searched on the net and I found the command variable < file.txt but it cannot see the file. The .txt file contains two values in the first and the second line. Is there any way to give the first-line value to... (1 Reply)
Discussion started by: chris_euop
1 Replies

7. UNIX and Linux Applications

How to import and dump file to remote Oracle server?

Hi All, I have a linux centos instance which has a dump file. I need to import the dump file to the oracle server which is located at some remote location. I have installed the oracle client on my machine and I am able to connect to the remote oracle server. Now how to import the dump to the... (3 Replies)
Discussion started by: Palak Sharma
3 Replies

8. Homework & Coursework Questions

Oracle dump file (del format) import into db2

1. The problem statement, all variables and given/known data: are the oracle dump files compatible to direct import into db2? I already tried many times but it always truncated results. anyone can help/ advice or suggest? 2. Relevant commands, code, scripts, algorithms: exp... (3 Replies)
Discussion started by: Sonny_103024
3 Replies

9. Shell Programming and Scripting

How to get the shell script to read the .txt file as an input/data?

i have written my shell script in notepad however i am struggling to pass the data file to be read to the script the data file is of .txt format. My target is to run the shell script from the terminal and pass 3 arguments e.g. polg@DESKTOP-BVPDC5C:~/CS1420/coursework$ bash valsplit.sh input.txt... (11 Replies)
Discussion started by: Gurdza32
11 Replies

10. UNIX for Beginners Questions & Answers

Using bash script : How to Import data from a dsv file into multiple tables in mysql

HI I have a dsv file that looks like: <<BOF>> record_number|id_number|first name|last name|msisdn|network|points|card number|gender 312|9101011234011|Test Junior|Smith|071 123 4321|MTN|73|1241551413214444|M 313|9012023213011|Bob|Smith|27743334321|Vodacom|3|1231233232323244|M... (4 Replies)
Discussion started by: tera
4 Replies
acct(1M)																  acct(1M)

NAME
acct: acctdisk, acctdusg, accton, acctwtmp, closewtmp, utmp2wtmp - overview of accounting and miscellaneous accounting commands SYNOPSIS
file] file] [file] reason DESCRIPTION
Accounting software is structured as a set of tools (consisting of both C programs and shell procedures) that can be used to build account- ing systems. The shell procedures, described in acctsh(1M), are built on top of the C programs. Connect time accounting is handled by various programs that write records into the The programs described in acctcon(1M) convert this file into session and charging records which are then summarized by (see acctmerg(1M)). Process accounting is performed by the HP-UX system kernel. Upon termination of a process, one record per process is written to a file (normally The programs in acctprc(1M) summarize this data for charging purposes; is used to summarize command usage (see acctcms(1M)). Current process data can be examined using (see acctcom(1M)). Process accounting and connect time accounting (or any accounting records in the format described in acct(4)) can be merged and summarized into total accounting records by (see the format in acct(4)). is used to format any or all accounting records (see acctsh(1M)). reads lines that contain user ID, login name, and number of disk blocks, and converts them to total accounting records that can be merged with other accounting records. reads its standard input (usually from and computes disk resource consumption (including indirect blocks) by login. Only files found under login directories (as determined from the password file) are accounted for. All files under a login directory are assumed to belong to that user regardless of actual owner. If is given, records consisting of those file names for which charges no one are placed in file (a potential source for finding users trying to avoid disk charges). If is given, file is the name of the password file. This option is not needed if the password file is (See diskusg(1M) for more details.) turns process accounting off if the optional file argument is omitted. If file is given, it must be the name of an existing file, to which the kernel appends process accounting records (see acct(2) and acct(4)). writes a utmp record to its standard output if the option is not used. If the option is used, writes a record to The record contains the current time and a string of characters that describe the reason for writing the record. A record type of is assigned (see utmp(4) and utmps(4)). The string argument reason must be 11 or fewer characters, numbers, or spaces if option is not used. Otherwise, it must be 63 or fewer characters, numbers, or spaces. For example, the following are suggestions for use in reboot and shutdown procedures, respec- tively: writes a record, for each user currently logged in, to the file This program is invoked by runacct to close the existing file before creat- ing a new one. writes a record, for each user currently logged in, to the file This program is invoked by runacct to initialize the newly created file. FILES
Holds all accounting commands listed in section(1M) of this manual. Current process accounting file. Used for converting login name to user ID Login/logoff history file. New login/logoff history database. SEE ALSO
acctcms(1M), acctcom(1M), acctcon(1M), acctmerg(1M), acctprc(1M), acctsh(1M), diskusg(1M), fwtmp(1M), runacct(1M), acct(2), acct(4), utmp(4). utmps(4), wtmps(4). STANDARDS CONFORMANCE
acct(1M)
All times are GMT -4. The time now is 03:42 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy