Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Discarding records with duplicate fields Post 303043653 by beca123456 on Monday 3rd of February 2020 06:19:13 AM
Old 02-03-2020
Discarding records with duplicate fields

Hi,

My input looks like this (tab-delimited):
Code:
grp1	name2	firstname	M	55	item1	item1.0
grp1	name2	firstname	F	55	item1	item1.0
grp2	name1	firstname	M	55	item1	item1.0
grp2	name2	firstname	M	55	item1	item1.0

Using awk, I am trying to discard the records with common fields 2, 4, 5, 6, 7 only for records from 'grp2' (field $1; i.e. records starting with 'grp1' should be kept no matter what) in order to get this output:
Code:
grp1	name2	firstname	M	55	item1	item1.0
grp1	name2	firstname	F	55	item1	item1.0
grp2	name1	firstname	M	55	item1	item1.0

Here is my code, but I don't see what is wrong with it.
Code:
awk '
BEGIN{FS=OFS="\t"}
{
    a[$2 FS $4 FS $5 FS $6 FS $7]

    if($1 ~ /grp2/){
        if(a[$2 FS $4 FS $5 FS $6 FS $7]++==0){
            print $0
         }
    }
    else{
        print $0
    }
}' input.tab

Any help would be greatly appreciated.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Records Duplicate

Hi Everyone, I have a flat file of 1000 unique records like following : For eg Andy,Flower,201-987-0000,12/23/01 Andrew,Smith,101-387-3400,11/12/01 Ani,Ross,401-757-8640,10/4/01 Rich,Finny,245-308-0000,2/27/06 Craig,Ford,842-094-8740,1/3/04 . . . . . . Now I want to duplicate... (9 Replies)
Discussion started by: ganesh123
9 Replies

2. Shell Programming and Scripting

compare fields in a file with duplicate records

Hi: I've been searching the net but didnt find a clue. I have a file in which, for some records, some fields coincide. I want to compare one (or more) of the dissimilar fields and retain the one record that fulfills a certain condition. For example, on this file: 99 TR 1991 5 06 ... (1 Reply)
Discussion started by: rleal
1 Replies

3. Shell Programming and Scripting

combine duplicate records

I have a .DAT file like below 23666483030000653-B94030001OLFXXX000000120081227 23797049900000654-E71060001OLFXXX000000220081227 23699281320000655 E71060002OLFXXX000000320081227 22885068900000652 B86860003OLFXXX592123320081227 22885068900000652 B86860003ODL-SP592123420081227... (8 Replies)
Discussion started by: kshuser
8 Replies

4. UNIX for Dummies Questions & Answers

Getting non-duplicate records

Hi, I have a file with these records abc xyz xyz pqr uvw cde cde In my o/p file , I want all the non duplicate rows to be shown. o/p abc pqr uvw Any suggestions how to do this? Thanks for the help. rs (2 Replies)
Discussion started by: rs123
2 Replies

5. UNIX for Dummies Questions & Answers

Need to keep duplicate records

Consider my input is 10 10 20 then, uniq -u will give 20 and uniq -dwill return 10. But i need the output as , 10 10 How we can achieve this? Thanks (4 Replies)
Discussion started by: pandeesh
4 Replies

6. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

7. Shell Programming and Scripting

Deleting duplicate records from file 1 if records from file 2 match

I have 2 files "File 1" is delimited by ";" and "File 2" is delimited by "|". File 1 below (3 record shown): Doc1;03/01/2012;New York;6 Main Street;Mr. Smith 1;Mr. Jones Doc2;03/01/2012;Syracuse;876 Broadway;John Davis;Barbara Lull Doc3;03/01/2012;Buffalo;779 Old Windy Road;Charles... (2 Replies)
Discussion started by: vestport
2 Replies

8. Shell Programming and Scripting

Remove duplicate records

Hi, i am working on a script that would remove records or lines in a flat file. The only difference in the file is the "NOT NULL" word. Please see below example of the input file. INPUT FILE:> CREATE a ( TRIAL_CLIENT NOT NULL VARCHAR2(60), TRIAL_FUND NOT NULL... (3 Replies)
Discussion started by: reignangel2003
3 Replies

9. Shell Programming and Scripting

Duplicate records

Gents, I have a file which contends duplicate records in column 1, but the values in column 2 are different. 3099753489 3 3099753489 5 3101954341 12 3101954341 14 3102153285 3 3102153285 5 3102153297 3 3102153297 5 I will like to get something like this: output desired... (16 Replies)
Discussion started by: jiam912
16 Replies

10. Shell Programming and Scripting

Duplicate records

Gents, Please give a help file --BAD STATUS NOT RESHOOTED-- *** VP 41255/51341 in sw 2973 *** VP 41679/51521 in sw 2973 *** VP 41687/51653 in sw 2973 *** VP 41719/51629 in sw 2976 --BAD COG NOT RESHOOTED-- *** VP 41689/51497 in sw 2974 *** VP 41699/51677 in sw 2974 *** VP... (18 Replies)
Discussion started by: jiam912
18 Replies
CUBRID_QUERY(3) 							 1							   CUBRID_QUERY(3)

cubrid_query - Send a CUBRID query

SYNOPSIS
resource cubrid_query (string $query, [resource $conn_identifier]) DESCRIPTION
cubrid_query(3) sends a unique query (multiple queries are not supported) to the currently active database on the server that's associated with the specified $conn_identifier. PARAMETERS
o $query - An SQL query Data inside the query should be properly escaped. o $conn_identifier - The CUBRID connection. If the connection identifier is not specified, the last connection opened by cubrid_connect(3) is assumed. RETURN VALUES
For SELECT, SHOW, DESCRIBE, EXPLAIN and other statements returning resultset, cubrid_query(3) returns a resource on success, or FALSE on error. For other type of SQL statements, INSERT, UPDATE, DELETE, DROP, etc, cubrid_query(3) returns TRUE on success or FALSE on error. The returned result resource should be passed to cubrid_fetch_array(3), and other functions for dealing with result tables, to access the returned data. Use cubrid_num_rows(3) to find out how many rows were returned for a SELECT statement or cubrid_affected_rows(3) to find out how many rows were affected by a DELETE, INSERT, REPLACE, or UPDATE statement. cubrid_query(3) will also fail and return FALSE if the user does not have permission to access the table(s) referenced by the query. EXAMPLES
Example #1 Invalid Query The following query is syntactically invalid, so cubrid_query(3) fails and returns FALSE. <?php $conn = cubrid_connect('localhost', 33000, 'demodb'); $result = cubrid_query('SELECT * WHERE 1=1'); if (!$result) { die('Invalid query: ' . cubrid_error()); } ?> Example #2 Valid Query The following query is valid, so cubrid_query(3) returns a resource. <?php // This could be supplied by a user, for example $firstname = 'fred'; $lastname = 'fox'; $conn = cubrid_connect('localhost', 33000, 'demodb'); cubrid_execute($conn,"DROP TABLE if exists friends"); cubrid_execute($conn,"create table friends(firstname varchar,lastname varchar,address char(24),age int)"); cubrid_execute($conn,"insert into friends values('fred','fox','home-1','20')"); cubrid_execute($conn,"insert into friends values('blue','cat','home-2','21')"); // Formulate Query // This is the best way to perform an SQL query // For more examples, see cubrid_real_escape_string() $query = sprintf("SELECT firstname, lastname, address, age FROM friends WHERE firstname='%s' AND lastname='%s'", cubrid_real_escape_string($firstname), cubrid_real_escape_string($lastname)); // Perform Query $result = cubrid_query($query); // Check result // This shows the actual query sent to CUBRID, and the error. Useful for debugging. if (!$result) { $message = 'Invalid query: ' . cubrid_error() . " "; $message .= 'Whole query: ' . $query; die($message); } // Use result // Attempting to print $result won't allow access to information in the resource // One of the cubrid result functions must be used // See also cubrid_result(), cubrid_fetch_array(), cubrid_fetch_row(), etc. while ($row = cubrid_fetch_assoc($result)) { echo $row['firstname']; echo $row['lastname']; echo $row['address']; echo $row['age']; } // Free the resources associated with the result set // This is done automatically at the end of the script cubrid_free_result($result); ?> SEE ALSO
cubrid_connect(3), cubrid_error(3), cubrid_real_escape_string(3), cubrid_result(3), cubrid_fetch_assoc(3), cubrid_unbuffered_query(3). PHP Documentation Group CUBRID_QUERY(3)
All times are GMT -4. The time now is 05:12 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy