Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Remove duplicate rows of a file based on a value of a column Post 302240574 by risk_sly on Friday 26th of September 2008 05:18:02 AM
Old 09-26-2008
Remove duplicate rows of a file based on a value of a column

Hi,

I am processing a file and would like to delete duplicate records as indicated by one of its column. e.g.

COL1 COL2 COL3
A 1234 1234
B 3k32 2322
C Xk32 TTT
A NEW XX22
B 3k32 2322


I want the file not to contain duplicate COL1. i.e. the file should only contain the ff:

COL1 COL2 COL3
A 1234 1234
B 3k32 2322
C Xk32 TTT


The records with duplicate COL1 were deleted.

Anybody who has suggestions on how to do this?

Thank you.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

To remove date and duplicate rows from a log file using unix commands

Hi, I have a log file having size of 48mb. For such a large log file. I want to get the message in a particular format which includes only unique error and exception messages. The following things to be done : 1) To remove all the date and time from the log file 2) To remove all the... (1 Reply)
Discussion started by: Pank10
1 Replies

2. Shell Programming and Scripting

how to delete duplicate rows based on last column

hii i have a huge amt of data stored in a file.Here in this file i need to remove duplicates rows in such a way that the last column has different data & i must check for greatest among last colmn data & print the largest data along with other entries but just one of other duplicate entries is... (16 Replies)
Discussion started by: reva
16 Replies

3. Shell Programming and Scripting

Remove duplicate line detail based on column one data

My input file: AVI.out <detail>named as the RRM .</detail> AVI.out <detail>Contains 1 RRM .</detail> AR0.out <detail>named as the tellurite-resistance.</detail> AWG.out <detail>Contains 2 HTH .</detail> ADV.out <detail>named as the DENR family.</detail> ADV.out ... (10 Replies)
Discussion started by: patrick87
10 Replies

4. UNIX for Dummies Questions & Answers

How to get remove duplicate of a file based on many conditions

Hii Friends.. I have a huge set of data stored in a file.Which is as shown below a.dat: RAO 1869 12 19 0 0 0.00 17.9000 82.3000 10.0 0 0.00 0 3.70 0.00 0.00 0 0.00 3.70 4 NULL LEE 1870 4 11 1 0 0.00 30.0000 99.0000 0.0 0 0.00 0 0.00 0.00 0.00 0 ... (3 Replies)
Discussion started by: reva
3 Replies

5. UNIX for Dummies Questions & Answers

Remove duplicate rows when >10 based on single column value

Hello, I'm trying to delete duplicates when there are more than 10 duplicates, based on the value of the first column. e.g. a 1 a 2 a 3 b 1 c 1 gives b 1 c 1 but requires 11 duplicates before it deletes. Thanks for the help Video tutorial on how to use code tags in The UNIX... (11 Replies)
Discussion started by: informaticist
11 Replies

6. UNIX for Dummies Questions & Answers

merging rows into new file based on rows and first column

I have 2 files, file01= 7 columns, row unknown (but few) file02= 7 columns, row unknown (but many) now I want to create an output with the first field that is shared in both of them and then subtract the results from the rest of the fields and print there e.g. file 01 James|0|50|25|10|50|30... (1 Reply)
Discussion started by: A-V
1 Replies

7. Shell Programming and Scripting

Remove duplicate rows based on one column

Dear members, I need to filter a file based on the 8th column (that is id), and does not mather the other columns, because I want just one id (1 line of each id) and remove the duplicates lines based on this id (8th column), and does not matter wich duplicate will be removed. example of my file... (3 Replies)
Discussion started by: clarissab
3 Replies

8. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

9. Shell Programming and Scripting

Filter file to remove duplicate values in first column

Hello, I have a script that is generating a tab delimited output file. num Name PCA_A1 PCA_A2 PCA_A3 0 compound_00 -3.5054 -1.1207 -2.4372 1 compound_01 -2.2641 0.4287 -1.6120 3 compound_03 -1.3053 1.8495 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies

10. Shell Programming and Scripting

Remove duplicate values in a column(not in the file)

Hi Gurus, I have a file(weblog) as below abc|xyz|123|agentcode=sample code abcdeeess,agentcode=sample code abcdeeess,agentcode=sample code abcdeeess|agentadd=abcd stereet 23343,agentadd=abcd stereet 23343 sss|wwq|999|agentcode=sample1 code wqwdeeess,gentcode=sample1 code... (4 Replies)
Discussion started by: ratheeshjulk
4 Replies
STRFMON(3)						     Linux Programmer's Manual							STRFMON(3)

NAME
strfmon - convert monetary value to a string SYNOPSIS
#include <monetary.h> ssize_t strfmon(char *s, size_t max, const char *format, ...); DESCRIPTION
The strfmon() function formats the specified amounts according to the format specification format and places the result in the character array s of size max. Ordinary characters in format are copied to s without conversion. Conversion specifiers are introduced by a `%' character. Immediately following it there can be zero or more of the following flags: =f The single-byte character f is used as the numeric fill character (to be used with a left precision, see below). When not speci- fied, the space character is used. ^ Do not use any grouping characters that might be defined for the current locale. By default, grouping is enabled. ( or + The ( flag indicates that negative amounts should be enclosed between parentheses. The + flag indicates that signs should be handled in the default way, that is, amounts are preceded by the locale's sign indication, e.g., nothing for positive, "-" for negative. ! Omit the currency symbol. - Left justify all fields. The default is right justification. Next, there may be a field width: a decimal digit string specifying a minimum field width in bytes. The default is 0. A result smaller than this width is padded with spaces (on the left, unless the left-justify flag was given). Next, there may be a left precision of the form "#" followed by a decimal digit string. If the number of digits left of the radix character is smaller than this, the representation is padded on the left with the numeric fill character. Grouping characters are not counted in this field width. Next, there may be a right precision of the form "." followed by a decimal digit string. The amount being formatted is rounded to the spec- ified number of digits prior to formatting. The default is specified in the frac_digits and int_frac_digits items of the current locale. If the right precision is 0, no radix character is printed. (The radix character here is determined by LC_MONETARY, and may differ from that specified by LC_NUMERIC.) Finally, the conversion specification must be ended with a conversion character. The three conversion characters are % (In this case the entire specification must be exactly "%%".) Put a `%' character in the result string. i One argument of type double is converted using the locale's international currency format. n One argument of type double is converted using the locale's national currency format. RETURN VALUE
The strfmon() function returns the number of characters placed in the array s, not including the terminating NUL character, provided the string, including the terminating NUL, fits. Otherwise, it sets errno to E2BIG, returns -1, and the contents of the array is undefined. EXAMPLE
The call strfmon(buf, sizeof(buf), "[%^=*#6n] [%=*#6i]", 1234.567, 1234.567); outputs [ fl **1234,57] [ NLG **1 234,57] in the Dutch locale (with fl for "florijnen" and NLG for Netherlands Guilders). The grouping character is very ugly because it takes as much space as a digit, while it should not take more than half that, and will no doubt cause confusion. Surprisingly, the "fl" is preceded and followed by a space, and "NLG" is preceded by one and followed by two spaces. This may be a bug in the locale files. The Italian, Aus- tralian, Swiss and Portuguese locales yield [ L. **1235] [ ITL **1.235] [ $**1234.57] [ AUD **1,234.57] [Fr. **1234,57] [CHF **1.234,57] [ **1234$57Esc] [ **1.234$57PTE ] SEE ALSO
setlocale(3), sprintf(3), locale(7) Linux 2000-12-05 STRFMON(3)
All times are GMT -4. The time now is 08:49 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy