Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Removing rows that contain non-unique column entry Post 302924198 by msatseqs on Thursday 6th of November 2014 10:34:25 PM
Old 11-06-2014
Removing rows that contain non-unique column entry

Background:
I have a file of thousands of potential SSR primers from Batch Primer 3.
I can't use primers that will contain the same sequence ID or sequence as another primer.
I have some basic shell scripting skills, but not enough to handle this.

What you need to know:
I need to remove the entire line(row) if its entry in column 3 or 13 is not unique when compared to the rest of its column. Or, I need to cat all lines that have a unique entry in columns 3 and 13 to a new file.

Note: I can't just remove the duplicate value, I have to remove the whole row after checking a value in that row against the rest of its column.

Example data is attached. Red values are duplicates.

Thank you very very very much!
Removing rows that contain non-unique column entry-example-msat-datapng
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Removing the rest of line from the second entry of an expression

Dear people, can you please enlighten: I need to do a (most probably) very simple thing but couldn't figure how. I have files with lots of lines starting with a fixed expression: Query=. (the dot is a space) followed by different combinations of characters including special ones such... (5 Replies)
Discussion started by: roussine
5 Replies

2. Shell Programming and Scripting

for each different entry in column 1 extract maximum values from column 2 in unix/awk

Hello, I have 2 columns (1st column has multiple entries but the corresponding values in the column 2 may be the same or different.) however I want to extract unique values for each entry in column 1 by assigning the max value from column 2 SDF4 -0.211654 SDF4 0.978068 ... (1 Reply)
Discussion started by: Diya123
1 Replies

3. Shell Programming and Scripting

Rename a header column by adding another column entry to the header column name URGENT!!

Hi All, I have a file example.csv which looks like this GrpID,TargetID,Signal,Avg_Num CSCH74_1_1,2007,61,256 CSCH74_1_1,212007,647,679 CSCH74_1_1,12007,3,32 CSCH74_1_1,207,299,777 I want the output as GrpID,TragetID,Signal-CSCH74_1_1,Avg_Num CSCH74_1_1,2007,61,256... (4 Replies)
Discussion started by: Vavad
4 Replies

4. Shell Programming and Scripting

unique entry add values

Hi, I have a file with 3 columns ABC 3 1 ABC 5 1 XYZ 4 2 DEF 3 2 DEF 4 1 DEF 6 1 MNO 5 5 JKL 3 2 JKL 4 2 PQR 12 1 For each unique entry in column 1 I want to add values in column 2 and column3 o/p ABC 8 2 XYZ 4 2 (1 Reply)
Discussion started by: Diya123
1 Replies

5. UNIX for Dummies Questions & Answers

Delete rows with unique value for specific column

Hi all I have a file which looks like this 1234|1|Jon|some text|some text 1234|2|Jon|some text|some text 3453|5|Jon|some text|some text 6533|2|Kate|some text|some text 4567|3|Chris|some text|some text 4567|4|Maggie|some text|some text 8764|6|Maggie|some text|some text My third column is my... (9 Replies)
Discussion started by: A-V
9 Replies

6. UNIX for Dummies Questions & Answers

merging rows into new file based on rows and first column

I have 2 files, file01= 7 columns, row unknown (but few) file02= 7 columns, row unknown (but many) now I want to create an output with the first field that is shared in both of them and then subtract the results from the rest of the fields and print there e.g. file 01 James|0|50|25|10|50|30... (1 Reply)
Discussion started by: A-V
1 Replies

7. Shell Programming and Scripting

Delete unique rows - optimize script

Hi all, I have the following input - the unique row key is 1st column cat file.txt A response C request C response D request C request C response E request The desired output should be C request (7 Replies)
Discussion started by: varu0612
7 Replies

8. Shell Programming and Scripting

Unique extraction of rows

I do have a tab delimited file of the following format: 431 kat1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 432 kat2 2 NA NA NA NA NA NA NA NA NA NA NA NA NA 433 KATe NA 3 NA NA 6 NA NA NA 10 11 NA NA NA NA 542 Kaed 2 NA NA NA NA NA NA NA NA NA NA NA NA NA 543 hkwuy NA NA NA NA 6 NA NA NA NA 11 NA NA... (11 Replies)
Discussion started by: Kanja
11 Replies

9. Shell Programming and Scripting

Count occurrence of column one unique value having unique second column value

Hello Team, I need your help on the following: My input file a.txt is as below: 3330690|373846|108471 3330690|373846|108471 0640829|459725|100001 0640829|459725|100001 3330690|373847|108471 Here row 1 and row 2 of column 1 are identical but corresponding column 2 value are... (4 Replies)
Discussion started by: angshuman
4 Replies

10. UNIX for Beginners Questions & Answers

Awk/sed summation of one column based on some entry in first column

Hi All , I am having an input file as stated below Input file 6 ddk/djhdj/djhdj/Q 10 0.5 dhd/jdjd.djd.nd/QB 01 0.5 hdhd/jd/jd/jdj/Q 10 0.5 512 hd/hdh/gdh/Q 01 0.5 jdjd/jd/ud/j/QB 10 0.5 HD/jsj/djd/Q 01 0.5 71 hdh/jjd/dj/jd/Q 10 0.5 ... (5 Replies)
Discussion started by: kshitij
5 Replies
DB2_STATISTICS(3)							 1							 DB2_STATISTICS(3)

db2_statistics - Returns a result set listing the index and statistics for a table

SYNOPSIS
resource db2_statistics (resource $connection, string $qualifier, string $schema, string $table-name, bool $unique) DESCRIPTION
Returns a result set listing the index and statistics for a table. PARAMETERS
o $connection - A valid connection to an IBM DB2, Cloudscape, or Apache Derby database. o $qualifier - A qualifier for DB2 databases running on OS/390 or z/OS servers. For other databases, pass NULL or an empty string. o $schema - The schema that contains the targeted table. If this parameter is NULL, the statistics and indexes are returned for the schema of the current user. o $table_name - The name of the table. o $unique - An integer value representing the type of index information to return. o $0 - Return only the information for unique indexes on the table. o $1 - Return the information for all indexes on the table. RETURN VALUES
Returns a statement resource with a result set containing rows describing the statistics and indexes for the base tables matching the specified parameters. The rows are composed of the following columns: +--------------+---------------------------------------------------+ | Column name | | | | | | | Description | | | | +--------------+---------------------------------------------------+ | TABLE_CAT | | | | | | | The catalog that contains the table. The value is | | | NULL if this table does not have catalogs. | | | | | TABLE_SCHEM | | | | | | | Name of the schema that contains the table. | | | | | TABLE_NAME | | | | | | | Name of the table. | | | | | NON_UNIQUE | | | | | | | An integer value representing whether the index | | | prohibits unique values, or whether the row rep- | | | resents statistics on the table itself: | | | | | | box, tab (|); c | c | . T{ Return value | | | | | | Parameter type | | | | +--------------+---------------------------------------------------+ |0 (SQL_FALSE) | | | | | | | The index allows duplicate values. | | | | |1 (SQL_TRUE) | | | | | | | The index values must be unique. | | | | | | | | NULL | | | | | | | This row is statistics information for the table | | | itself. | | | | +--------------+---------------------------------------------------+ T} T{ INDEX_QUALIFIER T} |T{ A string value representing the qualifier that would have to be prepended to INDEX_NAME to fully qualify the index. T} T{ INDEX_NAME T} |T{ A string representing the name of the index. T} T{ TYPE T} |T{ An integer value representing the type of information contained in this row of the result set: +------------------------+---------------------------------------------------+ | Return value | | | | | | | Parameter type | | | | +------------------------+---------------------------------------------------+ | 0 (SQL_TABLE_STAT) | | | | | | | The row contains statistics about the table | | | itself. | | | | |1 (SQL_INDEX_CLUSTERED) | | | | | | | The row contains information about a clustered | | | index. | | | | | 2 (SQL_INDEX_HASH) | | | | | | | The row contains information about a hashed | | | index. | | | | | 3 (SQL_INDEX_OTHER) | | | | | | | The row contains information about a type of | | | index that is neither clustered nor hashed. | | | | +------------------------+---------------------------------------------------+ T} T{ ORDINAL_POSITION T} |T{ The 1-indexed position of the column in the index. NULL if the row contains statistics information about the table itself. T} T{ COLUMN_NAME T} |T{ The name of the column in the index. NULL if the row contains statistics information about the table itself. T} T{ ASC_OR_DESC T} |T{ A if the column is sorted in ascending order, D if the column is sorted in descending order, NULL if the row contains statistics informa- tion about the table itself. T} T{ CARDINALITY T} |T{ If the row contains information about an index, this column contains an integer value representing the number of unique values in the index. If the row contains information about the table itself, this column contains an integer value representing the number of rows in the table. T} T{ PAGES T} |T{ If the row contains information about an index, this column contains an integer value representing the number of pages used to store the index. If the row contains information about the table itself, this column contains an integer value representing the number of pages used to store the table. T} T{ FILTER_CONDITION T} |T{ Always returns NULL. T} SEE ALSO
db2_column_privileges(3), db2_columns(3), db2_foreign_keys(3), db2_primary_keys(3), db2_procedure_columns(3), db2_procedures(3), db2_spe- cial_columns(3), db2_table_privileges(3), db2_tables(3). PHP Documentation Group DB2_STATISTICS(3)
All times are GMT -4. The time now is 06:24 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy