Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Remove duplicate rows when >10 based on single column value Post 302590795 by informaticist on Tuesday 17th of January 2012 02:52:30 PM
Old 01-17-2012
Neither of those worked, here is a sample of the input

Code:
Col1    Col2
2600.m01    194
2600.m01    332
2600.m01    595
2600.m01    664
2600.m01    673
2600.m01    685
2600.m01    6043
2600.m01    6158
2600.m01    6677
2600.m01    6897
2600.m01    6938
2600.m01    6969
2600.m01    7001
2600.m01    7014
2500.m01    7016
2500.m01    7064
2500.m01    7070
2500.m01    8166
2500.m01    9288
2500.m01    9291
2500.m01    9304
2500.m01    9316
2500.m01    9330
2500.m01    9365
2432.m0392    9369
2134.m01234    10525
2827.m033    67
2472.m001234    2643

and the correct output would be
Code:
Col1    Col2
2500.m01    7016
2500.m01    7064
2500.m01    7070
2500.m01    8166
2500.m01    9288
2500.m01    9291
2500.m01    9304
2500.m01    9316
2500.m01    9330
2500.m01    9365
2432.m0392    9369
2134.m01234    10525
2827.m033    67
2472.m001234    2643

Moderator's Comments:
Mod Comment Use code tags, please!

Last edited by radoulov; 01-17-2012 at 04:23 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remove duplicate rows of a file based on a value of a column

Hi, I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g. COL1 COL2 COL3 A 1234 1234 B 3k32 2322 C Xk32 TTT A NEW XX22 B 3k32 ... (7 Replies)
Discussion started by: risk_sly
7 Replies

2. Shell Programming and Scripting

how to delete duplicate rows based on last column

hii i have a huge amt of data stored in a file.Here in this file i need to remove duplicates rows in such a way that the last column has different data & i must check for greatest among last colmn data & print the largest data along with other entries but just one of other duplicate entries is... (16 Replies)
Discussion started by: reva
16 Replies

3. Shell Programming and Scripting

Remove duplicate line detail based on column one data

My input file: AVI.out <detail>named as the RRM .</detail> AVI.out <detail>Contains 1 RRM .</detail> AR0.out <detail>named as the tellurite-resistance.</detail> AWG.out <detail>Contains 2 HTH .</detail> ADV.out <detail>named as the DENR family.</detail> ADV.out ... (10 Replies)
Discussion started by: patrick87
10 Replies

4. Shell Programming and Scripting

duplicate row based on single column

I am a newbie to shell scripting .. I have a .csv file. It has 1000 some rows and about 7 columns... but before I insert this data to a table I have to parse it and clean it ..basing on the value of the first column..which a string of phone number type... example below.. column 1 ... (2 Replies)
Discussion started by: mitr
2 Replies

5. Shell Programming and Scripting

remove duplicates based on single column

Hello, I am new to shell scripting. I have a huge file with multiple columns for example: I have 5 columns below. HWUSI-EAS000_29:1:105 + chr5 76654650 AATTGGAA HHHHG HWUSI-EAS000_29:1:106 + chr5 76654650 AATTGGAA B@HYL HWUSI-EAS000_29:1:108 + ... (4 Replies)
Discussion started by: Diya123
4 Replies

6. Shell Programming and Scripting

Removing duplicate records in a file based on single column

Hi, I want to remove duplicate records including the first line based on column1. For example inputfile(filer.txt): ------------- 1,3000,5000 1,4000,6000 2,4000,600 2,5000,700 3,60000,4000 4,7000,7777 5,999,8888 expected output: ---------------- 3,60000,4000 4,7000,7777... (5 Replies)
Discussion started by: G.K.K
5 Replies

7. Shell Programming and Scripting

Removing duplicate records in a file based on single column explanation

I was reading this thread. It looks like a simpler way to say this is to only keep uniq lines based on field or column 1. https://www.unix.com/shell-programming-scripting/165717-removing-duplicate-records-file-based-single-column.html Can someone explain this command please? How are there no... (5 Replies)
Discussion started by: cokedude
5 Replies

8. UNIX for Dummies Questions & Answers

merging rows into new file based on rows and first column

I have 2 files, file01= 7 columns, row unknown (but few) file02= 7 columns, row unknown (but many) now I want to create an output with the first field that is shared in both of them and then subtract the results from the rest of the fields and print there e.g. file 01 James|0|50|25|10|50|30... (1 Reply)
Discussion started by: A-V
1 Replies

9. Shell Programming and Scripting

Converting Single Column into Multiple rows, but with strings to specific tab column

Dear fellows, I need your help. I'm trying to write a script to convert a single column into multiple rows. But it need to recognize the beginning of the string and set it to its specific Column number. Each Line (loop) begins with digit (RANGE). At this moment it's kind of working, but it... (6 Replies)
Discussion started by: AK47
6 Replies

10. Shell Programming and Scripting

Remove duplicate rows based on one column

Dear members, I need to filter a file based on the 8th column (that is id), and does not mather the other columns, because I want just one id (1 line of each id) and remove the duplicates lines based on this id (8th column), and does not matter wich duplicate will be removed. example of my file... (3 Replies)
Discussion started by: clarissab
3 Replies
MYSQLINDEXCHECK(1)						  MySQL Utilities						MYSQLINDEXCHECK(1)

NAME
mysqlindexcheck - Check for redundant or duplicate indexes on a list tables or databases SYNOPSIS
mysqlindexcheck [options] db[:table] ... DESCRIPTION
This utility reads the indexes for one or more tables and identifies duplicate and potentially redundant indexes. To check all tables in a database, specify only the database name. To check a specific table, name the table in db.table format. It is pos- sible to mix database and table names. You can scan tables in any database except the internal databases mysql, INFORMATION_SCHEMA, and performance_schema. Depending on the index type, the utility applies the following rules to compare indexes (designated as idx_a and idx_b): BTREE idx_b is redundant to idx_a if and only if the first n columns in idx_b also appear in idx_a. Order and uniqueness count. HASH idx_a and idx_b are duplicates if and only if they contain the same columns in the same order. Uniqueness counts. SPATIAL idx_a and idx_b are duplicates if and only if they contain the same column (only one column is permitted). FULLTEXT idx_b is redundant to idx_a if and only if all columns in idx_b are included in idx_a. Order counts. To see DROP statements to drop redundant indexes, specify the --show-drops option. To examine the existing indexes, use the --verbose option, which prints the equivalent CREATE INDEX (or ALTER TABLE for primary keys. To display the best or worst nonprimary key indexes for each table, use the --best or --worst option. This causes the output to show the best or worst indexes from tables with 10 or more rows. By default, each option shows five indexes. To override that, provide an integer value for the option. To change the format of the index lists displayed for the --show-indexes, --best, and --worst options, use one of the following values with the --format option: grid (default) Display output in grid or table format like that of the mysql monitor. csv Display output in comma-separated values format. tab Display output in tab-separated format. sql print SQL statements rather than a list. vertical Display output in single-column format like that of the G command for the mysql monitor. Note: The --best and --worst lists cannot be printed as SQL statements. OPTIONS
mysqlindexcheck accepts the following command-line options: --help Display a help message and exit. --best[=<N>] If --stats is given, limit index statistics to the best N indexes. The default value of N is 5 if omitted. --format=<index_format>, -f<index_format> Specify the index list display format for output produced by --stats. Permitted format values are grid, csv, tab, sql, and vertical. The default is grid. --server=<source> Connection information for the server in <user>[:<passwd>]@<host>[:<port>][:<socket>] format. --show-drops, -d Display DROP statements for dropping indexes. --show-indexes, -i Display indexes for each table. --skip, -s Skip tables that do not exist. --stats Show index performance statistics. --verbose, -v Specify how much information to display. Use this option multiple times to increase the amount of information. For example, -v = verbose, -vv = more verbose, -vvv = debug. --version Display version information and exit. --worst[=<N>] If --stats is given, limit index statistics to the worst N indexes. The default value of N is 5 if omitted. NOTES
You must provide connection parameters (user, host, password, and so forth) for an account that has the appropriate privileges to read all objects accessed during the operation. For the --format option, the permitted values are not case sensitive. In addition, values may be specified as any unambiguous prefix of a valid value. For example, --format=g specifies the grid format. An error occurs if a prefix matches more than one valid value. EXAMPLES
To check all tables in the employees database on the local server to see the possible redundant and duplicate indexes, use this command: $ mysqlindexcheck --server=root@localhost employees # Source on localhost: ... connected. # The following indexes are duplicates or redundant for table employees.dept_emp: # CREATE INDEX emp_no ON employees.dept_emp (emp_no) USING BTREE # may be redundant or duplicate of: ALTER TABLE employees.dept_emp ADD PRIMARY KEY (emp_no, dept_no) # The following indexes are duplicates or redundant for table employees.dept_manager: # CREATE INDEX emp_no ON employees.dept_manager (emp_no) USING BTREE # may be redundant or duplicate of: ALTER TABLE employees.dept_manager ADD PRIMARY KEY (emp_no, dept_no) # The following indexes are duplicates or redundant for table employees.salaries: # CREATE INDEX emp_no ON employees.salaries (emp_no) USING BTREE # may be redundant or duplicate of: ALTER TABLE employees.salaries ADD PRIMARY KEY (emp_no, from_date) # The following indexes are duplicates or redundant for table employees.titles: # CREATE INDEX emp_no ON employees.titles (emp_no) USING BTREE # may be redundant or duplicate of: ALTER TABLE employees.titles ADD PRIMARY KEY (emp_no, title, from_date) COPYRIGHT
Copyright (c) 2010, 2012, Oracle and/or its affiliates. All rights reserved. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MER- CHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA AUTHOR
MySQL Utilities Team COPYRIGHT
2010, Oracle and/or its affiliates. All rights reserved. 1.0.3 May 09, 2012 MYSQLINDEXCHECK(1)
All times are GMT -4. The time now is 03:29 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy