Sponsored Content
Top Forums Shell Programming and Scripting Find duplicate values in specific column and delete all the duplicate values Post 302985868 by sajmar on Wednesday 16th of November 2016 12:05:26 PM
Old 11-16-2016
Dear senhia83

Thank you so much, the awk command works exactly perfect. However, the order is important for me. my desire order is based on the second column.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

have to retrieve the distinct values (not duplicate) from 2nd column and display

I have a text file names test2 with 3 columns as below . We have to retrieve the distinct values (not duplicate) from 2nd column and display. I have used the below command but giving some error. NS3303 NS CRAFT LTD NS3303 NS CHIRON VACCINES LTD NS3303 NS ALLIED MEDICARE LTD NS3303 NS... (16 Replies)
Discussion started by: shirdi
16 Replies

2. Shell Programming and Scripting

Find and replace duplicate column values in a row

I have file which as 12 columns and values like this 1,2,3,4,5 a,b,c,d,e b,c,a,e,f a,b,e,a,h if you see the first column has duplicate values, I need to identify (print it to console) the duplicate value (which is 'a') and also remove duplicate values like below. I could be in two... (5 Replies)
Discussion started by: nuthalapati
5 Replies

3. Shell Programming and Scripting

Perl: filtering lines based on duplicate values in a column

Hi I have a file like this. I need to eliminate lines with first column having the same value 10 times. 13 18 1 + chromosome 1, 122638287 AGAGTATGGTCGCGGTTG 13 18 1 + chromosome 1, 128904080 AGAGTATGGTCGCGGTTG 13 18 1 - chromosome 14, 13627938 CAACCGCGACCATACTCT 13 18 1 + chromosome 1,... (5 Replies)
Discussion started by: polsum
5 Replies

4. UNIX for Dummies Questions & Answers

[SOLVED] remove lines that have duplicate values in column two

Hi, I've got a file that I'd like to uniquely sort based on column 2 (values in column 2 begin with "comp"). I tried sort -t -nuk2,3 file.txtBut got: sort: multi-character tab `-nuk2,3' "man sort" did not help me out Any pointers? Input: Output: (5 Replies)
Discussion started by: pathunkathunk
5 Replies

5. Shell Programming and Scripting

Check to identify duplicate values at first column in csv file

Hello experts, I have a requirement where I have to implement two checks on a csv file: 1. Check to see if the value in first column is duplicate, if any value is duplicate script should exit. 2. Check to verify if the value at second column is between "yes" or "no", if it is anything else... (4 Replies)
Discussion started by: avikaljain
4 Replies

6. Shell Programming and Scripting

Get the average from column, and eliminate the duplicate values.

Dear Experts, Kindly help me please, I have a big file where there is duplicate values in col 11 till col 23, every 2 rows appers a new numbers, but in each row there is different coordinates x and y in col 57 till col 74. Please i will like to get a single value and average of the x and y... (8 Replies)
Discussion started by: jiam912
8 Replies

7. Shell Programming and Scripting

Identify duplicate values at first column in csv file

Input 1,ABCD,no 2,system,yes 3,ABCD,yes 4,XYZ,no 5,XYZ,yes 6,pc,noCode used to find duplicate with regard to 2nd column awk 'NR == 1 {p=$2; next} p == $2 { print "Line" NR "$2 is duplicated"} {p=$2}' FS="," ./input.csv Now is there a wise way to de-duplicate the entire line (remove... (4 Replies)
Discussion started by: deadyetagain
4 Replies

8. Shell Programming and Scripting

Filter file to remove duplicate values in first column

Hello, I have a script that is generating a tab delimited output file. num Name PCA_A1 PCA_A2 PCA_A3 0 compound_00 -3.5054 -1.1207 -2.4372 1 compound_01 -2.2641 0.4287 -1.6120 3 compound_03 -1.3053 1.8495 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies

9. Shell Programming and Scripting

Remove duplicate values in a column(not in the file)

Hi Gurus, I have a file(weblog) as below abc|xyz|123|agentcode=sample code abcdeeess,agentcode=sample code abcdeeess,agentcode=sample code abcdeeess|agentadd=abcd stereet 23343,agentadd=abcd stereet 23343 sss|wwq|999|agentcode=sample1 code wqwdeeess,gentcode=sample1 code... (4 Replies)
Discussion started by: ratheeshjulk
4 Replies

10. UNIX for Beginners Questions & Answers

Find lines with duplicate values in a particular column

I have a file with 5 columns. I want to pull out all records where the value in column 4 is not unique. For example in the sample below, I would want it to print out all lines except for the last two. 40991764 2419 724 47182 Cand A 40992936 3591 724 47182 Cand B 40993016 3671 724 47182 Cand C... (5 Replies)
Discussion started by: kaktus
5 Replies
tbl-dctrl(1)						   Debian administrator's manual					      tbl-dctrl(1)

NAME
tbl-dctrl - generate tabular representations of data in dctrl format SYNOPSIS
tbl-dctrl [ options ] [ -c column-specification ... ] [ filename ] ... tbl-dctrl --version tbl-dctrl --help DESCRIPTION
tbl-dctrl creates tabular representations of data given to it in Debian control file format. By default, tbl-dctrl reads the whole database, looking for the longest entry in each requested column; it then outputs a table, with bor- ders and column titles, where each column is just wide enough to fit the longest entry. Most of this behaviour can be customized as described below. A column is requested by specifying the -c (--column) switch with a column specification. The simplest kind of a column specification con- sists solely of the name of a field. In such a case, tbl-dctrl will include in the output a column whose title is the literal column spec- ification and whose data is drawn from fields with that name. If no -c options are given, tbl-dctrl will use all fields in the input in the order in which they first appear. There are two optional additions one can make to a column specification. Prefixing the field name with some text followed by an equality sign (for example, -c 'Package name=Package') modifies the column in such a way that the text before the equality sign is used as the col- umn title, while the text after the equality sign is used as the name of the field from which data is drawn. One can also append a colon followed by a positive whole number to the field name. In such a case, the number after the colon specifies the width of the column. These two additions can be used separately or together. If there are more than one colon, the last one is significant. If there are more than one equals sign, the first one is significant. Other colons and equals signs are used simply as data. Note that the whole column specification must be given to tbl-dctrl as one argument, so if it contains spaces, it must be quoted for the shell. If all requested columns have a specified width, tbl-dctrl will produce output immediately, not waiting for the whole input to be read in. OPTIONS
-d delimiter, --delimiter=delimiter Instead of drawing nice borders to the table, use the specified delimiter string to delimit columns in a row. -H, --no-heading Do not print a table heading (column titles). -l level, --errorlevel=level Set log level to level. level is one of fatal, important, informational and debug, but the last may not be available, depending on the compile-time options. These categories are given here in order; every message that is emitted when fatal is in effect, will be emitted in the important error level, and so on. The default is important. -V, --version Print out version information. -C, --copying Print out the copyright license. This produces much output; be sure to redirect or pipe it somewhere (such as your favourite pager). -h, --help Print out a help summary. OPERANDS
tbl-dctrl will read its input from the files named on the command line, in the specified order. A file called - represents the program's standard input stream. If no files are named, the program behaves as if - alone had been named, that is, input is read from the standard input stream. STDIN
The standard input stream may be used as input as specified above in the OPERANDS section. INPUT FILES
All input to tbl-dctrl is in the format of a Debian control file. A Debian control (dctrl) file is a semistructured single-table database stored in a machine-parseable text file. Such a database consists of a set of records; each record is a mapping from field names to field content. Textually, records are separated by empty lines, while each field is encoded as one or more nonempty lines inside a record. A field starts with its name, followed by a colon, followed by the field content. The colon must reside on the first line of the field, and the first line must start with no whitespace. Subsequent lines, in contrast, always start with linear whitespace (one or more space or tab characters). When input is read from multiple files, a record separator is implicit between two adjacent files. ENVIRONMENT VARIABLES
The standard locale environment, specifically its character set setting, affects the interpretation of input and output as character streams. ASYNCHRONOUS EVENTS
Standard UNIX signals have their usual meaning. STDOUT
All output is sent to the standard output stream. The output is a tabular representation of the input database restricted to the specified fields. Logically, the output is a table; when the -d option is used, this table is represented simply by separating columns in each row by the specified delimiter; when the option is not used, a frame is drawn around the table. The order of the columns is the same as the order of the column specifications on the command line. OUTPUT FILES
There are no output files. EXIT STATUS
This utility exits with 0 when successful. It uses a nonzero exit code inconsistently when an error is noticed (this is a bug). CONSEQUENCES OF ERRORS
In case of errors in the input, the output will be partially or completely garbage. In case of errors in invocation, the program will refuse to function. EXAMPLES
The following command line pipe outputs a table of all packages, with their maintainer data, sorted by the maintainer data, that have no content: % grep-available -FInstalled-Size --eq 0 | sort-dctrl -kMaintainer - | tbl-dctrl -cPackage -cMaintainer AUTHOR
The tbl-dctrl program and this manual page were written by Antti-Juhani Kaijanaho. SEE ALSO
apt-cache(1), ara(1), dpkg-awk(1), dpkg-query(1), grep-dctrl(1), sort-dctrl(1), dpkg(8) Debian Project 2006-04-02 tbl-dctrl(1)
All times are GMT -4. The time now is 01:52 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy