Sponsored Content
Top Forums Shell Programming and Scripting finding duplicates in csv based on key columns Post 302576186 by jayan_jay on Thursday 24th of November 2011 02:04:44 AM
Old 11-24-2011
Give some input content and expected output ..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

removing duplicates based on key

HI I am having a file like this 1234 12345678 1234567890123 4321 43215678 432156789028433435 I want to get ouput as 1234567890123 432156789028433435 based on key position 1-4 I am using ksh can anyone give me an idea Thanks pukars (1 Reply)
Discussion started by: pukars4u
1 Replies

2. Shell Programming and Scripting

finding duplicates in columns and removing lines

I am trying to figure out how to scan a file like so: 1 ralphs office","555-555-5555","ralph@mail.com","www.ralph.com 2 margies office","555-555-5555","ralph@mail.com","www.ralph.com 3 kims office","555-555-5555","kims@mail.com","www.ralph.com 4 tims... (17 Replies)
Discussion started by: totus
17 Replies

3. Shell Programming and Scripting

Remove duplicates based on the two key columns

Hi All, I needs to fetch unique records based on a keycolumn(ie., first column1) and also I needs to get the records which are having max value on column2 in sorted manner... and duplicates have to store in another output file. Input : Input.txt 1234,0,x 1234,1,y 5678,10,z 9999,10,k... (7 Replies)
Discussion started by: kmsekhar
7 Replies

4. Shell Programming and Scripting

Search based on 1,2,4,5 columns and remove duplicates in the same file.

Hi, I am unable to search the duplicates in a file based on the 1st,2nd,4th,5th columns in a file and also remove the duplicates in the same file. Source filename: Filename.csv "1","ccc","information","5000","temp","concept","new" "1","ddd","information","6000","temp","concept","new"... (2 Replies)
Discussion started by: onesuri
2 Replies

5. UNIX for Dummies Questions & Answers

Removing duplicates based on key

Hi, I have the input file with the below data: 12345|12|34 12345|13|23 3456|12|90 15670|12|13 12345|10|14 3456|12|13 I need to remove the duplicates based on the first field only. I need the output like: 12345|12|34 3456|12|90 15670|12|13 The first field needs to be unique . (4 Replies)
Discussion started by: pandeesh
4 Replies

6. Shell Programming and Scripting

CSV with commas in field values, remove duplicates, cut columns

Hi Description of input file I have: ------------------------- 1) CSV with double quotes for string fields. 2) Some string fields have Comma as part of field value. 3) Have Duplicate lines 4) Have 200 columns/fields 5) File size is more than 10GB Description of output file I need:... (4 Replies)
Discussion started by: krishnix
4 Replies

7. Shell Programming and Scripting

Removing duplicates in fixed width file which has multiple key columns

Hi All , I have a requirement where I need to remove duplicates from a fixed width file which has multiple key columns .Also , need to capture the duplicate records into another file . File has 8 columns. Key columns are col1 and col2. Col1 has the length of 8 col 2 has the length of 3. ... (5 Replies)
Discussion started by: saj
5 Replies

8. Shell Programming and Scripting

Remove Duplicates on multiple Key Columns and get the Latest Record from Date/Time Column

Hi Experts , we have a CDC file where we need to get the latest record of the Key columns Key Columns will be CDC_FLAG and SRC_PMTN_I and fetch the latest record from the CDC_PRCS_TS Can we do it with a single awk command. Please help.... (3 Replies)
Discussion started by: vijaykodukula
3 Replies

9. Shell Programming and Scripting

UNIX scripting for finding duplicates and null records in pk columns

Hi, I have a requirement.for eg: i have a text file with pipe symbol as delimiter(|) with 4 columns a,b,c,d. Here a and b are primary key columns.. i want to process that file to find the duplicates and null values are in primary key columns(a,b) . I want to write the unique records in which... (5 Replies)
Discussion started by: praveenraj.1991
5 Replies

10. UNIX for Beginners Questions & Answers

Sort and remove duplicates in directory based on first 5 columns:

I have /tmp dir with filename as: 010020001_S-FOR-Sort-SYEXC_20160229_2212101.marker 010020001_S-FOR-Sort-SYEXC_20160229_2212102.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212104.marker 010020001-S-XOR-Sort-SYEXC_20160229_2212105.marker 010020001_S-ZOR-Sort-SYEXC_20160229_2212106.marker... (4 Replies)
Discussion started by: gnnsprapa
4 Replies
MYSQLINDEXCHECK(1)						  MySQL Utilities						MYSQLINDEXCHECK(1)

NAME
mysqlindexcheck - Check for redundant or duplicate indexes on a list tables or databases SYNOPSIS
mysqlindexcheck [options] db[:table] ... DESCRIPTION
This utility reads the indexes for one or more tables and identifies duplicate and potentially redundant indexes. To check all tables in a database, specify only the database name. To check a specific table, name the table in db.table format. It is pos- sible to mix database and table names. You can scan tables in any database except the internal databases mysql, INFORMATION_SCHEMA, and performance_schema. Depending on the index type, the utility applies the following rules to compare indexes (designated as idx_a and idx_b): BTREE idx_b is redundant to idx_a if and only if the first n columns in idx_b also appear in idx_a. Order and uniqueness count. HASH idx_a and idx_b are duplicates if and only if they contain the same columns in the same order. Uniqueness counts. SPATIAL idx_a and idx_b are duplicates if and only if they contain the same column (only one column is permitted). FULLTEXT idx_b is redundant to idx_a if and only if all columns in idx_b are included in idx_a. Order counts. To see DROP statements to drop redundant indexes, specify the --show-drops option. To examine the existing indexes, use the --verbose option, which prints the equivalent CREATE INDEX (or ALTER TABLE for primary keys. To display the best or worst nonprimary key indexes for each table, use the --best or --worst option. This causes the output to show the best or worst indexes from tables with 10 or more rows. By default, each option shows five indexes. To override that, provide an integer value for the option. To change the format of the index lists displayed for the --show-indexes, --best, and --worst options, use one of the following values with the --format option: grid (default) Display output in grid or table format like that of the mysql monitor. csv Display output in comma-separated values format. tab Display output in tab-separated format. sql print SQL statements rather than a list. vertical Display output in single-column format like that of the G command for the mysql monitor. Note: The --best and --worst lists cannot be printed as SQL statements. OPTIONS
mysqlindexcheck accepts the following command-line options: --help Display a help message and exit. --best[=<N>] If --stats is given, limit index statistics to the best N indexes. The default value of N is 5 if omitted. --format=<index_format>, -f<index_format> Specify the index list display format for output produced by --stats. Permitted format values are grid, csv, tab, sql, and vertical. The default is grid. --server=<source> Connection information for the server in <user>[:<passwd>]@<host>[:<port>][:<socket>] format. --show-drops, -d Display DROP statements for dropping indexes. --show-indexes, -i Display indexes for each table. --skip, -s Skip tables that do not exist. --stats Show index performance statistics. --verbose, -v Specify how much information to display. Use this option multiple times to increase the amount of information. For example, -v = verbose, -vv = more verbose, -vvv = debug. --version Display version information and exit. --worst[=<N>] If --stats is given, limit index statistics to the worst N indexes. The default value of N is 5 if omitted. NOTES
You must provide connection parameters (user, host, password, and so forth) for an account that has the appropriate privileges to read all objects accessed during the operation. For the --format option, the permitted values are not case sensitive. In addition, values may be specified as any unambiguous prefix of a valid value. For example, --format=g specifies the grid format. An error occurs if a prefix matches more than one valid value. EXAMPLES
To check all tables in the employees database on the local server to see the possible redundant and duplicate indexes, use this command: $ mysqlindexcheck --server=root@localhost employees # Source on localhost: ... connected. # The following indexes are duplicates or redundant for table employees.dept_emp: # CREATE INDEX emp_no ON employees.dept_emp (emp_no) USING BTREE # may be redundant or duplicate of: ALTER TABLE employees.dept_emp ADD PRIMARY KEY (emp_no, dept_no) # The following indexes are duplicates or redundant for table employees.dept_manager: # CREATE INDEX emp_no ON employees.dept_manager (emp_no) USING BTREE # may be redundant or duplicate of: ALTER TABLE employees.dept_manager ADD PRIMARY KEY (emp_no, dept_no) # The following indexes are duplicates or redundant for table employees.salaries: # CREATE INDEX emp_no ON employees.salaries (emp_no) USING BTREE # may be redundant or duplicate of: ALTER TABLE employees.salaries ADD PRIMARY KEY (emp_no, from_date) # The following indexes are duplicates or redundant for table employees.titles: # CREATE INDEX emp_no ON employees.titles (emp_no) USING BTREE # may be redundant or duplicate of: ALTER TABLE employees.titles ADD PRIMARY KEY (emp_no, title, from_date) COPYRIGHT
Copyright (c) 2010, 2012, Oracle and/or its affiliates. All rights reserved. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; version 2 of the License. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MER- CHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA AUTHOR
MySQL Utilities Team COPYRIGHT
2010, Oracle and/or its affiliates. All rights reserved. 1.0.3 May 09, 2012 MYSQLINDEXCHECK(1)
All times are GMT -4. The time now is 07:15 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy