Sponsored Content
Full Discussion: removing duplicates.
Top Forums UNIX for Advanced & Expert Users removing duplicates. Post 302504580 by raju4u on Tuesday 15th of March 2011 12:18:37 AM
Old 03-15-2011
removing duplicates.

Hi All

In unix ,we have a file ,there we have to remove the duplicates by using one specific column.
Can any body tell me the command.

ex:
file1
id,name
1,ww
2,qwq
2,asas
3,asa
4,asas
4,asas


o/p:
1,ww
2,qwq
3,asa
4,asaa
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing duplicates

Hi, I've been trying to removed duplicates lines with similar columns in a fixed width file and it's not working. I've search the forum but nothing comes close. I have a sample file: 27147140631203RA CCD * 27147140631203RA PPN * 37147140631207RD AAA 47147140631203RD JNA... (12 Replies)
Discussion started by: giannicello
12 Replies

2. UNIX for Dummies Questions & Answers

removing duplicates and sort -k

Hello experts, I am trying to remove all lines in a csv file where the 2nd columns is a duplicate. I am try to use sort with the key parameter sort -u -k 2,2 File.csv > Output.csv File.csv File Name|Document Name|Document Title|Organization Word Doc 1.doc|Word Document|Sample... (3 Replies)
Discussion started by: orahi001
3 Replies

3. Shell Programming and Scripting

removing duplicates

Hi I have a file that are a list of people & their credentials i recieve frequently The issue is that whne I catnet this list that duplicat entries exists & are NOT CONSECUTIVE (i.e. uniq -1 may not weork here ) I'm trying to write a scrip that will remove duplicate entries the script can... (5 Replies)
Discussion started by: stevie_velvet
5 Replies

4. Shell Programming and Scripting

Removing duplicates

Hi, I have a file in the below format., test test (10) to to (25) see see (45) and i need the output in the format of test 10 to 25 see 45 Some one help me? (6 Replies)
Discussion started by: imdadulla
6 Replies

5. Shell Programming and Scripting

Removing duplicates

I have a test file with the following 2 columns: Col 1 | Col 2 T1 | 1 <= remove T5 | 1 T4 | 2 T1 | 3 T3 | 3 T4 | 1 <= remove T1 | 2 <= remove T3 ... (7 Replies)
Discussion started by: gctex
7 Replies

6. Emergency UNIX and Linux Support

Removing all the duplicates

i want to remove all the duplictaes in a file.I dont want even a single entry. For the input data: 12345|12|34 12345|13|23 3456|12|90 15670|12|13 12345|10|14 3456|12|13 i need the below data in one file 15670|12|13 and the below data in another file (9 Replies)
Discussion started by: pandeesh
9 Replies

7. Shell Programming and Scripting

Help in removing duplicates

I have an input file abc.txt with info like: abcd rateuse inklite robet rateuse abcd I need to remove duplicates from the file (eg: abcd,rateuse) from the file and need to place the contents in same file abc.txt if needed can be placed in another file. can anyone help me in this :( (4 Replies)
Discussion started by: rkrish
4 Replies

8. UNIX for Dummies Questions & Answers

Removing duplicates from a file

Hi All, I am merging files coming from 2 different systems ,while doing that I am getting duplicates entries in the merged file I,01,000131,764,2,4.00 I,01,000131,765,2,4.00 I,01,000131,772,2,4.00 I,01,000131,773,2,4.00 I,01,000168,762,2,2.00 I,01,000168,763,2,2.00... (5 Replies)
Discussion started by: Sri3001
5 Replies

9. Shell Programming and Scripting

Removing duplicates except the last occurrence

Hi All, i have a file like below, @DB_FCTS\src\Data\Scripts\Delete_CU_OM_BIL_PRT_STMT_TYP.sql @DB_FCTS\src\Data\Scripts\Delete_CDP_BILL_LBL_MSG.sql @DB_FCTS\src\Data\Scripts\Delete_OM_BIDDR.sql @DB_FCTS\src\Data\Scripts\Insert_CU_OM_LBL_MSG.sql... (11 Replies)
Discussion started by: mechvijays
11 Replies

10. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3 (2 Replies)
Discussion started by: sagar_1986
2 Replies
ASA(1P) 						     POSIX Programmer's Manual							   ASA(1P)

PROLOG
This manual page is part of the POSIX Programmer's Manual. The Linux implementation of this interface may differ (consult the correspond- ing Linux manual page for details of Linux behavior), or the interface may not be implemented on Linux. NAME
asa -- interpret carriage-control characters SYNOPSIS
asa [file...] DESCRIPTION
The asa utility shall write its input files to standard output, mapping carriage-control characters from the text files to line-printer control sequences in an implementation-defined manner. The first character of every line shall be removed from the input, and the following actions are performed. If the character removed is: <space> The rest of the line is output without change. 0 A <newline> is output, then the rest of the input line. 1 One or more implementation-defined characters that causes an advance to the next page shall be output, followed by the rest of the input line. + The <newline> of the previous line shall be replaced with one or more implementation-defined characters that causes printing to return to column position 1, followed by the rest of the input line. If the '+' is the first character in the input, it shall be equivalent to <space>. The action of the asa utility is unspecified upon encountering any character other than those listed above as the first character in a line. OPTIONS
None. OPERANDS
file A pathname of a text file used for input. If no file operands are specified, the standard input shall be used. STDIN
The standard input shall be used if no file operands are specified, and shall be used if a file operand is '-' and the implementation treats the '-' as meaning standard input. Otherwise, the standard input shall not be used. See the INPUT FILES section. INPUT FILES
The input files shall be text files. ENVIRONMENT VARIABLES
The following environment variables shall affect the execution of asa: LANG Provide a default value for the internationalization variables that are unset or null. (See the Base Definitions volume of POSIX.1-2008, Section 8.2, Internationalization Variables for the precedence of internationalization variables used to determine the values of locale categories.) LC_ALL If set to a non-empty string value, override the values of all the other internationalization variables. LC_CTYPE Determine the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multi-byte characters in arguments and input files). LC_MESSAGES Determine the locale that should be used to affect the format and contents of diagnostic messages written to standard error. NLSPATH Determine the location of message catalogs for the processing of LC_MESSAGES. ASYNCHRONOUS EVENTS
Default. STDOUT
The standard output shall be the text from the input file modified as described in the DESCRIPTION section. STDERR
None. OUTPUT FILES
None. EXTENDED DESCRIPTION
None. EXIT STATUS
The following exit values shall be returned: 0 All input files were output successfully. >0 An error occurred. CONSEQUENCES OF ERRORS
Default. The following sections are informative. APPLICATION USAGE
None. EXAMPLES
1. The following command: asa file permits the viewing of file (created by a program using FORTRAN-style carriage-control characters) on a terminal. 2. The following command: a.out | asa | lp formats the FORTRAN output of a.out and directs it to the printer. RATIONALE
The asa utility is needed to map ``standard'' FORTRAN 77 output into a form acceptable to contemporary printers. Usually, asa is used to pipe data to the lp utility; see lp. This utility is generally used only by FORTRAN programs. The standard developers decided to retain asa to avoid breaking the historical large base of FORTRAN applications that put carriage-control characters in their output files. There is no requirement that a system have a FORTRAN compiler in order to run applications that need asa. Historical implementations have used an ASCII <form-feed> in response to a 1 and an ASCII <carriage-return> in response to a '+'. It is suggested that implementations treat characters other than 0, 1, and '+' as <space> in the absence of any compelling reason to do other- wise. However, the action is listed here as ``unspecified'', permitting an implementation to provide extensions to access fast multiple- line slewing and channel seeking in a non-portable manner. FUTURE DIRECTIONS
None. SEE ALSO
fort77, lp The Base Definitions volume of POSIX.1-2008, Chapter 8, Environment Variables COPYRIGHT
Portions of this text are reprinted and reproduced in electronic form from IEEE Std 1003.1, 2013 Edition, Standard for Information Technol- ogy -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 7, Copyright (C) 2013 by the Institute of Electrical and Electronics Engineers, Inc and The Open Group. (This is POSIX.1-2008 with the 2013 Technical Corrigendum 1 applied.) In the event of any discrepancy between this version and the original IEEE and The Open Group Standard, the original IEEE and The Open Group Stan- dard is the referee document. The original Standard can be obtained online at http://www.unix.org/online.html . Any typographical or formatting errors that appear in this page are most likely to have been introduced during the conversion of the source files to man page format. To report such errors, see https://www.kernel.org/doc/man-pages/reporting_bugs.html . IEEE
/The Open Group 2013 ASA(1P)
All times are GMT -4. The time now is 09:23 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy