12-28-2005
validate csv file load
Hi All,
I am using sqlldr to load my csv files into the database. The code in the sh script is as follows.
sqlldr ${DBUSER}/${DBPASS}@${ORACLE_SID} \
data=myCSV.data \
bad=myCSV.bad \
control=myCSV.ctl \
log=myCSV.log > /dev/null 2>&1
Now, the log file has some lines at the end which state the following:
Total logical records read: 30
Total logical records rejected: 1
This does not indicate why the record was rejected.
Now I want to validate the csv files, either before load or during load. During load would be better as i can add code in the same loader sh script.
The validation should be as follows:
1) Check for premature EOF in the input csv file.
2) Check for the number of fields for all records in the csv file.
3) Check for Not Null validations in the csv file.
4) Check for datatype violations in the csv file according to the database table column definitions.
This way I can find the reason for the records not being loaded.
If you guyz have faced such a scenario, do let me know your inputs.
Thanks in advance,
Rahul.
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hi all
I need to input values in a .csv file into my Oracle table running in Unix, I wonder what would be the command to do so...
The values are recorded in an excel file and I tried using a formatted text file to do so but failed because one of the field is simply too large to fit in the... (5 Replies)
Discussion started by: handynas
5 Replies
2. UNIX for Advanced & Expert Users
Hi all
I need to input values in a .csv file into my Oracle table running in Unix, I wonder what would be the command to do so...
The values are recorded in an excel file and I tried using a formatted text file to do so but failed because one of the field is simply too large to fit in the... (4 Replies)
Discussion started by: handynas
4 Replies
3. Shell Programming and Scripting
Hi.
I think some people have already asked this, but the answers/questions seem to be about validating the contents inside a CSV file.
I am simply after a simple variable solution (ie 0 = false, 1 = true) that I can use in my script to say that file so-and-so is actually a CSV file, or in some... (4 Replies)
Discussion started by: ElCaito
4 Replies
4. Shell Programming and Scripting
Since i'm new to scripting i'm findind it difficult to code a script. The script has to be an executable with 2 paramters passed to it.The Parameters are
1. The Control file name(.ctl file)
2. The Data file name(.csv file)
Does anybody have an idea about it? :confused: (3 Replies)
Discussion started by: Csmani
3 Replies
5. Shell Programming and Scripting
hi all
need your help.
I am wrting a script that will load data into the table.
then on another load will append the data into the existing table.
Regards
Ankit (1 Reply)
Discussion started by: ankitgupta
1 Replies
6. Web Development
hello
i want a script to load the data line by line from a csv file into a mysql
table (3 Replies)
Discussion started by: srpa01red
3 Replies
7. Shell Programming and Scripting
Hi All,
I was wondering if there is a utility/perl library/way of validating the contents of a CSV file using an XSD.
i.e. Customer csv (including Header & Trailer)
Cust_num, Cust_nme, Cust_typ, Cust_act_dte, Cust_loc,
101,Joe's Pizza,Retail,10121979,Detroit,
102,Sony... (6 Replies)
Discussion started by: luinzi
6 Replies
8. Shell Programming and Scripting
Hi all i would appreciate your help...
I am looking for a set of unix commands which i can use to
1) ingest a csv file with a known format
2) validate the filename
3) validate the data/datatypes
4) Insert into an oracle db
Can you help get me started?
yogz888 (1 Reply)
Discussion started by: yogz888
1 Replies
9. UNIX for Beginners Questions & Answers
Hi guys,
i want to validate the no.of colums in a csv file ,but if there is a comma(,) in any of the data values it should skip and count only valid (,) commas.
e.g
1.abc,pqrs,1234,567,hhh
result :4
2.abc,pqrs,1234,567,hhh,"in,valid",end12
result:6
here script should skip the comma inside... (10 Replies)
Discussion started by: harry123
10 Replies
10. Shell Programming and Scripting
Hi All;
I am struggling to write a script that validates file header.
Header file would be like below with TAB separated
TRX # TYPE REF # Source Piece Code Destination Piece Code
every time I need to check the txt file if the header was same as above fields if validation success... (6 Replies)
Discussion started by: heye18
6 Replies
LEARN ABOUT DEBIAN
text::csv::encoded::coder::encodeguess
Text::CSV::Encoded::Coder::EncodeGuess(3pm) User Contributed Perl Documentation Text::CSV::Encoded::Coder::EncodeGuess(3pm)
NAME
Text::CSV::Encoded::Coder::EncodeGuess - Text::CSV::Encoded coder class using Encode::Guess
SYNOPSIS
use Text::CSV::Encoded coder_class => 'Text::CSV::Encoded::Coder::EncodeGuess';
use Spreadsheet::ParseExcel;
my $csv = Text::CSV::Encoded->new();
$csv->encoding( ['ucs2', 'ascii'] ); # guessing ucs2 or ascii?
$csv->encoding_to_combine('shiftjis');
my $excel = Spreadsheet::ParseExcel::Workbook->Parse( $file );
my $sheet = $excel->{Worksheet}->[0];
for my $row ( $sheet->{MinRow} .. $sheet->{MaxRow} ) {
my @fields;
for my $col ( $sheet->{MinCol} .. $sheet->{MaxCol} ) {
my $cell = $sheet->{Cells}[$row][$col];
push @fields, $cell->{Val};
}
$csv->combine( @fields ) or die;
print $csv->string, "
";
}
DESCRIPTION
This module is inherited from Text::CSV::Encoded::Coder::Encode.
USE
Except for 2 attributes, same as Text::CSV::Encoded::Coder::Encode.
encoding_in
$csv = $csv->encoding_in( $encoding_list_ref );
The accessor to an encoding for pre-parsing CSV strings. If no encoding is given, returns current $encoding, otherwise the object itself.
$encoding_list_ref = $csv->encoding_in()
When you pass a list reference, it might guess the encoding from the given list.
$csv->encoding_in( ['shiftjis', 'euc-jp', 'iso-20022-jp'] );
If it cannot guess the encoding, the first encoding of the list is used.
encoding
$csv = $csv->encoding( $encoding_list_ref );
$encoding_list_ref = $csv->encoding();
You can pass a list reference to this attribute only:
* For list data consumed by combine().
* For list reference returned by getline().
In other word, in "combine" and "print", it might guess an encoding for the passing list data. If it cannot guess the encoding, the first
encoding of the list is used.
SEE ALSO
Encode, Encode::Guess
AUTHOR
Makamaka Hannyaharamitu, <makamaka[at]cpan.org>
COPYRIGHT AND LICENSE
Copyright 2008-2010 by Makamaka Hannyaharamitu
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
perl v5.14.2 2010-04-26 Text::CSV::Encoded::Coder::EncodeGuess(3pm)