validate csv file load


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting validate csv file load
# 1  
Old 12-28-2005
Lightbulb validate csv file load

Hi All,

I am using sqlldr to load my csv files into the database. The code in the sh script is as follows.

sqlldr ${DBUSER}/${DBPASS}@${ORACLE_SID} \
data=myCSV.data \
bad=myCSV.bad \
control=myCSV.ctl \
log=myCSV.log > /dev/null 2>&1

Now, the log file has some lines at the end which state the following:

Total logical records read: 30
Total logical records rejected: 1

This does not indicate why the record was rejected.

Now I want to validate the csv files, either before load or during load. During load would be better as i can add code in the same loader sh script.

The validation should be as follows:
1) Check for premature EOF in the input csv file.
2) Check for the number of fields for all records in the csv file.
3) Check for Not Null validations in the csv file.
4) Check for datatype violations in the csv file according to the database table column definitions.

This way I can find the reason for the records not being loaded.

If you guyz have faced such a scenario, do let me know your inputs.

Thanks in advance,
Rahul.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script to validate header in a csv file

Hi All; I am struggling to write a script that validates file header. Header file would be like below with TAB separated TRX # TYPE REF # Source Piece Code Destination Piece Code every time I need to check the txt file if the header was same as above fields if validation success... (6 Replies)
Discussion started by: heye18
6 Replies

2. UNIX for Beginners Questions & Answers

Validate csv file

Hi guys, i want to validate the no.of colums in a csv file ,but if there is a comma(,) in any of the data values it should skip and count only valid (,) commas. e.g 1.abc,pqrs,1234,567,hhh result :4 2.abc,pqrs,1234,567,hhh,"in,valid",end12 result:6 here script should skip the comma inside... (10 Replies)
Discussion started by: harry123
10 Replies

3. Shell Programming and Scripting

Script to ingest a csv, validate data and insert into Oracle

Hi all i would appreciate your help... I am looking for a set of unix commands which i can use to 1) ingest a csv file with a known format 2) validate the filename 3) validate the data/datatypes 4) Insert into an oracle db Can you help get me started? yogz888 (1 Reply)
Discussion started by: yogz888
1 Replies

4. Shell Programming and Scripting

How to: Validate a CSV file using an XSD?

Hi All, I was wondering if there is a utility/perl library/way of validating the contents of a CSV file using an XSD. i.e. Customer csv (including Header & Trailer) Cust_num, Cust_nme, Cust_typ, Cust_act_dte, Cust_loc, 101,Joe's Pizza,Retail,10121979,Detroit, 102,Sony... (6 Replies)
Discussion started by: luinzi
6 Replies

5. Web Development

script to load data from csv file

hello i want a script to load the data line by line from a csv file into a mysql table (3 Replies)
Discussion started by: srpa01red
3 Replies

6. Shell Programming and Scripting

Need help in wrting Load Script for a Load-Resume type of load.

hi all need your help. I am wrting a script that will load data into the table. then on another load will append the data into the existing table. Regards Ankit (1 Reply)
Discussion started by: ankitgupta
1 Replies

7. Shell Programming and Scripting

Shell Script to Load data into the database using a .csv file and .ctl file

Since i'm new to scripting i'm findind it difficult to code a script. The script has to be an executable with 2 paramters passed to it.The Parameters are 1. The Control file name(.ctl file) 2. The Data file name(.csv file) Does anybody have an idea about it? :confused: (3 Replies)
Discussion started by: Csmani
3 Replies

8. Shell Programming and Scripting

How to validate a CSV file?

Hi. I think some people have already asked this, but the answers/questions seem to be about validating the contents inside a CSV file. I am simply after a simple variable solution (ie 0 = false, 1 = true) that I can use in my script to say that file so-and-so is actually a CSV file, or in some... (4 Replies)
Discussion started by: ElCaito
4 Replies

9. UNIX for Advanced & Expert Users

How to load comma seperated values file (*.csv) into Oracle table

Hi all I need to input values in a .csv file into my Oracle table running in Unix, I wonder what would be the command to do so... The values are recorded in an excel file and I tried using a formatted text file to do so but failed because one of the field is simply too large to fit in the... (4 Replies)
Discussion started by: handynas
4 Replies

10. UNIX for Dummies Questions & Answers

How to load comma seperated values file (*.csv) into Oracle table

Hi all I need to input values in a .csv file into my Oracle table running in Unix, I wonder what would be the command to do so... The values are recorded in an excel file and I tried using a formatted text file to do so but failed because one of the field is simply too large to fit in the... (5 Replies)
Discussion started by: handynas
5 Replies
Login or Register to Ask a Question
getc_putc(8)						      System Manager's Manual						      getc_putc(8)

NAME
getc_putc - program to test hard drive performance. SYNOPSIS
getc_putc [-d dir] [-s size(KiB)] [-m machine-name] [-u uid-to-use:gid-to-use] [-g gid-to-use] DESCRIPTION
This manual page documents briefly the getc_putc, program. This is a simple adjunct to the bonnie++ benchmark. It is used to test various ways of doing IO one byte at a time, usually you don't need to do enough of this for it to be a performance issue for it to matter much which way you do it. But sometimes it's necessary (for example whan parsing IO from a terminal and then launching another process which will take over all IO, such as a simple shell). The real benefits of this are to help settle some arguements about the performance of such things, and to educate novices about how bad per-byte IO really is. OPTIONS
For getc_putc every option is of the form of a hyphen followed by a letter and then the next parameter contains the value. -d the directory to use for the tests. -s the size of the file for byte IO performance measured in kilobytes. NB You can specify the size in mega-bytes if you add 'm' to the end of the number. The default for this test is to test with a 40MiB file. Of the file only 1/32 of it will be used for write() and read() system calls (anything else takes too long), and only 1/4 of it will be used for locked getc() and putc(). -m name of the machine - for display purposes only. -u user-id to use. When running as root specify the UID to use for the tests. It is not recommended to use root, so if you really want to run as root then use -u root. Also if you want to specify the group to run as then use the user:group format. If you spec- ify a user by name but no group then the primary group of that user will be chosen. If you specify a user by number and no group then the group will be nogroup. -g group-id to use. Same as using :group for the -u parameter, just a different way to specify it for compatibility with other pro- grams. -q quiet mode. If specified then some of the extra informational messages will be suppressed. Also the csv data will be the only output on standard out and the plain text data will be on standard error. This means you can run getc_putc -q >> file.csv to record your csv data. OUTPUT
The primary output is plain-text in 80 columns which is designed to fit well when pasted into email and which will work well with Braille displays. The second type of output is CSV (Comma Seperated Values). This can easily be imported into any spread-sheet or database program. For every test the result is a speed in KiB/s. I do not display the CPU time because it presumably is 99% of the power of a single CPU (or something very close to that). AUTHOR
This program, it's manual page, and the Debian package were written by Russell Coker <russell@coker.com.au>. The documentation, the Perl scripts, and all the code for testing the creation of thousands of files was written by Russell Coker, but the entire package is under joint copyright with Tim Bray. SIGNALS
Handles SIGINT and does a cleanup (which may take some time), a second SIGINT or a SIGQUIT will cause it to immidiately die. SIGXCPU and SIGXFSZ act like SIGINT. Ignores SIGHUP. AVAILABILITY
The source is available from http://www.coker.com.au/bonnie++ . See http://etbe.coker.com.au/category/benchmark for further information. SEE ALSO
bonnie++(8), zcav(8) getc_putc(8)