Sponsored Content
Top Forums Programming eliminate duplicate rows - sqlloader Post 302595698 by jim mcnamara on Saturday 4th of February 2012 08:47:11 AM
Old 02-04-2012
There is no duplicate detection in sqlldr itself. It depends on constraints defined on the the table.

You will have to prepare your inputfile. You want to reject duplicate rows, duplicate equal the first two columns in separate rows are equal. If there are differences - missing column 3 - then prefer the row with three columns, otherwise just use the first of the duplicates. You have to sort in reverse order to guarantee that, in the case of duplicates, the row with data in column 3 comes first. Then eliminate duplicates.
This does that for you:
Code:
sort -r  -o inputfile inputfile  # keep the file intact to review data if there is a problem
awk '!arr[ $1 $2 ]++' inputfile > file_to_load_into_oracle    # eliminate duplicates as defined by you

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

duplicate rows in a file

hi all can anyone please let me know if there is a way to find out duplicate rows in a file. i have a file that has hundreds of numbers(all in next row). i want to find out the numbers that are repeted in the file. eg. 123434 534 5575 4746767 347624 5575 i want 5575 please help (3 Replies)
Discussion started by: infyanurag
3 Replies

2. Shell Programming and Scripting

how to delete duplicate rows in a file

I have a file content like below. "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","","" "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","","" "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""... (5 Replies)
Discussion started by: vamshikrishnab
5 Replies

3. Shell Programming and Scripting

How to extract duplicate rows

I have searched the internet for duplicate row extracting. All I have seen is extracting good rows or eliminating duplicate rows. How do I extract duplicate rows from a flat file in unix. I'm using Korn shell on HP Unix. For.eg. FlatFile.txt ======== 123:456:678 123:456:678 123:456:876... (5 Replies)
Discussion started by: bobbygsk
5 Replies

4. HP-UX

How to get Duplicate rows in a file

Hi all, I have written one shell script. The output file of this script is having sql output. In that file, I want to extract the rows which are having multiple entries(duplicate rows). For example, the output file will be like the following way. ... (7 Replies)
Discussion started by: raghu.iv85
7 Replies

5. Shell Programming and Scripting

How to extract duplicate rows

Hi! I have a file as below: line1 line2 line2 line3 line3 line3 line4 line4 line4 line4 I would like to extract duplicate lines (not unique, triplicate or quadruplicate lines). Output will be as below: line2 line2 I would appreciate if anyone can help. Thanks. (4 Replies)
Discussion started by: chromatin
4 Replies

6. Shell Programming and Scripting

Delete duplicate rows

Hi, This is a followup to my earlier post him mno klm 20 76 . + . klm_mango unix_00000001; alp fdc klm 123 456 . + . klm_mango unix_0000103; her tkr klm 415 439 . + . klm_mango unix_00001043; abc tvr klm 20 76 . + . klm_mango unix_00000001; abc def klm 83 84 . + . klm_mango... (5 Replies)
Discussion started by: jacobs.smith
5 Replies

7. Shell Programming and Scripting

Get the average from column, and eliminate the duplicate values.

Dear Experts, Kindly help me please, I have a big file where there is duplicate values in col 11 till col 23, every 2 rows appers a new numbers, but in each row there is different coordinates x and y in col 57 till col 74. Please i will like to get a single value and average of the x and y... (8 Replies)
Discussion started by: jiam912
8 Replies

8. Shell Programming and Scripting

How to duplicate rows using awk or any other method?

I want to duplicate each row in my file Egfile.txt Name State Age Jack NJ 34 John MA 23 Jessica FL 45 I want the code to produce this output Name State Age Jack NJ 34 Jack NJ 34 John MA 23 John MA 23 Jessica FL 45 Jessica FL 45 (6 Replies)
Discussion started by: sidnow
6 Replies

9. Shell Programming and Scripting

Extract duplicate rows with conditions

Gents Can you help please. Input file 5490921425 1 7 1310342 54909214251 5490921425 2 1 1 54909214252 5491120937 1 1 3 54911209371 5491120937 3 1 1 54911209373 5491320785 1 ... (4 Replies)
Discussion started by: jiam912
4 Replies

10. UNIX for Beginners Questions & Answers

Get duplicate rows from a csv file

How can i get the duplicates rows from a file using unix, for example i have data like a,1 b,2 c,3 d,4 a,1 c,3 e,5 i want output to be like a,1 c,3 (4 Replies)
Discussion started by: ggupta
4 Replies
ALTER 
DOMAIN(7) PostgreSQL 9.2.7 Documentation ALTER DOMAIN(7) NAME
ALTER_DOMAIN - change the definition of a domain SYNOPSIS
ALTER DOMAIN name { SET DEFAULT expression | DROP DEFAULT } ALTER DOMAIN name { SET | DROP } NOT NULL ALTER DOMAIN name ADD domain_constraint [ NOT VALID ] ALTER DOMAIN name DROP CONSTRAINT [ IF EXISTS ] constraint_name [ RESTRICT | CASCADE ] ALTER DOMAIN name RENAME CONSTRAINT constraint_name TO new_constraint_name ALTER DOMAIN name VALIDATE CONSTRAINT constraint_name ALTER DOMAIN name OWNER TO new_owner ALTER DOMAIN name RENAME TO new_name ALTER DOMAIN name SET SCHEMA new_schema DESCRIPTION
ALTER DOMAIN changes the definition of an existing domain. There are several sub-forms: SET/DROP DEFAULT These forms set or remove the default value for a domain. Note that defaults only apply to subsequent INSERT commands; they do not affect rows already in a table using the domain. SET/DROP NOT NULL These forms change whether a domain is marked to allow NULL values or to reject NULL values. You can only SET NOT NULL when the columns using the domain contain no null values. ADD domain_constraint [ NOT VALID ] This form adds a new constraint to a domain using the same syntax as CREATE DOMAIN (CREATE_DOMAIN(7)). When a new constraint is added to a domain, all columns using that domain will be checked against the newly added constraint. These checks can be suppressed by adding the new constraint using the NOT VALID option; the constraint can later be made valid using ALTER DOMAIN ... VALIDATE CONSTRAINT. Newly inserted or updated rows are always checked against all constraints, even those marked NOT VALID. NOT VALID is only accepted for CHECK constraints. DROP CONSTRAINT [ IF EXISTS ] This form drops constraints on a domain. If IF EXISTS is specified and the constraint does not exist, no error is thrown. In this case a notice is issued instead. RENAME CONSTRAINT This form changes the name of a constraint on a domain. VALIDATE CONSTRAINT This form validates a constraint previously added as NOT VALID, that is, verify that all data in columns using the domain satisfy the specified constraint. OWNER This form changes the owner of the domain to the specified user. RENAME This form changes the name of the domain. SET SCHEMA This form changes the schema of the domain. Any constraints associated with the domain are moved into the new schema as well. You must own the domain to use ALTER DOMAIN. To change the schema of a domain, you must also have CREATE privilege on the new schema. To alter the owner, you must also be a direct or indirect member of the new owning role, and that role must have CREATE privilege on the domain's schema. (These restrictions enforce that altering the owner doesn't do anything you couldn't do by dropping and recreating the domain. However, a superuser can alter ownership of any domain anyway.) PARAMETERS
name The name (possibly schema-qualified) of an existing domain to alter. domain_constraint New domain constraint for the domain. constraint_name Name of an existing constraint to drop or rename. NOT VALID Do not verify existing column data for constraint validity. CASCADE Automatically drop objects that depend on the constraint. RESTRICT Refuse to drop the constraint if there are any dependent objects. This is the default behavior. new_name The new name for the domain. new_constraint_name The new name for the constraint. new_owner The user name of the new owner of the domain. new_schema The new schema for the domain. NOTES
Currently, ALTER DOMAIN ADD CONSTRAINT and ALTER DOMAIN SET NOT NULL will fail if the named domain or any derived domain is used within a composite-type column of any table in the database. They should eventually be improved to be able to verify the new constraint for such nested columns. EXAMPLES
To add a NOT NULL constraint to a domain: ALTER DOMAIN zipcode SET NOT NULL; To remove a NOT NULL constraint from a domain: ALTER DOMAIN zipcode DROP NOT NULL; To add a check constraint to a domain: ALTER DOMAIN zipcode ADD CONSTRAINT zipchk CHECK (char_length(VALUE) = 5); To remove a check constraint from a domain: ALTER DOMAIN zipcode DROP CONSTRAINT zipchk; To rename a check constraint on a domain: ALTER DOMAIN zipcode RENAME CONSTRAINT zipchk TO zip_check; To move the domain into a different schema: ALTER DOMAIN zipcode SET SCHEMA customers; COMPATIBILITY
ALTER DOMAIN conforms to the SQL standard, except for the OWNER, RENAME, SET SCHEMA, and VALIDATE CONSTRAINT variants, which are PostgreSQL extensions. The NOT VALID clause of the ADD CONSTRAINT variant is also a PostgreSQL extension. SEE ALSO
CREATE DOMAIN (CREATE_DOMAIN(7)), DROP DOMAIN (DROP_DOMAIN(7)) PostgreSQL 9.2.7 2014-02-17 ALTER DOMAIN(7)
All times are GMT -4. The time now is 01:59 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy