Sponsored Content
Top Forums Shell Programming and Scripting remove duplicates based on single column Post 302525365 by Diya123 on Thursday 26th of May 2011 12:46:50 PM
Old 05-26-2011
Thank you.. It worked exactly as what i needed.
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How can i delete the duplicates based on one column of a line

I have my data something like this (08/03/2009 22:57:42.414)(:) king aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb (08/03/2009 22:57:42.416)(:) John cccccccccccc cccccvssssssssss baaaaa (08/03/2009 22:57:42.417)(:) Michael ddddddd tststststtststts (08/03/2009 22:57:42.425)(:) Ravi... (11 Replies)
Discussion started by: rdhanek
11 Replies

2. UNIX for Dummies Questions & Answers

Remove duplicates based on a column in fixed width file

Hi, How to output the duplicate record to another file. We say the record is duplicate based on a column whose position is from 2 and its length is 11 characters. The file is a fixed width file. ex of Record: DTYU12333567opert tjhi kkklTRG9012 The data in bold is the key on which... (1 Reply)
Discussion started by: Qwerty123
1 Replies

3. Shell Programming and Scripting

Remove duplicates based on the two key columns

Hi All, I needs to fetch unique records based on a keycolumn(ie., first column1) and also I needs to get the records which are having max value on column2 in sorted manner... and duplicates have to store in another output file. Input : Input.txt 1234,0,x 1234,1,y 5678,10,z 9999,10,k... (7 Replies)
Discussion started by: kmsekhar
7 Replies

4. Shell Programming and Scripting

need to remove duplicates based on key in first column and pattern in last column

Given a file such as this I need to remove the duplicates. 00060011 PAUL BOWSTEIN ad_waq3_921_20100826_010517.txt 00060011 PAUL BOWSTEIN ad_waq3_921_20100827_010528.txt 0624-01 RUT CORPORATION ad_sade3_10_20100827_010528.txt 0624-01 RUT CORPORATION ... (13 Replies)
Discussion started by: script_op2a
13 Replies

5. UNIX for Dummies Questions & Answers

Remove duplicate rows when >10 based on single column value

Hello, I'm trying to delete duplicates when there are more than 10 duplicates, based on the value of the first column. e.g. a 1 a 2 a 3 b 1 c 1 gives b 1 c 1 but requires 11 duplicates before it deletes. Thanks for the help Video tutorial on how to use code tags in The UNIX... (11 Replies)
Discussion started by: informaticist
11 Replies

6. UNIX for Dummies Questions & Answers

remove duplicates based on a field and criteria

Hi, I have a file with fields like below: A;XYZ;102345;222 B;XYZ;123243;333 C;ABC;234234;444 D;MNO;103345;222 E;DEF;124243;333 desired output: C;ABC;234234;444 D;MNO;103345;222 E;DEF;124243;333 ie, if the 4rth field is a duplicate.. i need only those records where... (5 Replies)
Discussion started by: wanderingmind16
5 Replies

7. Shell Programming and Scripting

Remove duplicates based on a field's value

Hi All, I have a text file with three columns. I would like a simple script that removes lines in which column 1 has duplicate entries, but use the largest value in column 3 to decide which one to keep. For example: Input file: 12345a rerere.rerere len=23 11111c fsdfdf.dfsdfdsf len=33 ... (3 Replies)
Discussion started by: anniecarv
3 Replies

8. Shell Programming and Scripting

Trying to remove duplicates based on field and row

I am trying to see if I can use awk to remove duplicates from a file. This is the file: -==> Listvol <== deleting /vol/eng_rmd_0941 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_1006 deleting /vol/eng_rmd_1012 rearrange /vol/eng_rmd_0943 ... (6 Replies)
Discussion started by: newbie2010
6 Replies

9. Shell Programming and Scripting

Remove duplicates according to their frequency in column

Hi all, I have huge a tab-delimited file with the following format and I want to remove the duplicates according to their frequency based on Column2 and Column3. Column1 Column2 Column3 Column4 Column5 Column6 Column7 1 user1 access1 word word 3 2 2 user2 access2 ... (10 Replies)
Discussion started by: corfuitl
10 Replies
RCSCLEAN(1)						      General Commands Manual						       RCSCLEAN(1)

NAME
rcsclean - clean up working files SYNOPSIS
rcsclean [options] [ file ... ] DESCRIPTION
rcsclean removes files that are not being worked on. rcsclean -u also unlocks and removes files that are being worked on but have not changed. For each file given, rcsclean compares the working file and a revision in the corresponding RCS file. If it finds a difference, it does nothing. Otherwise, it first unlocks the revision if the -u option is given, and then removes the working file unless the working file is writable and the revision is locked. It logs its actions by outputting the corresponding rcs -u and rm -f commands on the standard output. Files are paired as explained in ci(1). If no file is given, all working files in the current directory are cleaned. Filenames matching an RCS suffix denote RCS files; all others denote working files. The number of the revision to which the working file is compared may be attached to any of the options -n, -q, -r, or -u. If no revision number is specified, then if the -u option is given and the caller has one revision locked, rcsclean uses that revision; otherwise rcsclean uses the latest revision on the default branch, normally the root. rcsclean is useful for clean targets in makefiles. See also rcsdiff(1), which prints out the differences, and ci(1), which normally reverts to the previous revision if a file was not changed. OPTIONS
-ksubst Use subst style keyword substitution when retrieving the revision for comparison. See co(1) for details. -n[rev] Do not actually remove any files or unlock any revisions. Using this option will tell you what rcsclean would do without actually doing it. -q[rev] Do not log the actions taken on standard output. -r[rev] This option has no effect other than specifying the revision for comparison. -T Preserve the modification time on the RCS file even if the RCS file changes because a lock is removed. This option can suppress extensive recompilation caused by a make(1) dependency of some other copy of the working file on the RCS file. Use this option with care; it can suppress recompilation even when it is needed, i.e. when the lock removal would mean a change to keyword strings in the other working file. -u[rev] Unlock the revision if it is locked and no difference is found. -V Print RCS's version number. -Vn Emulate RCS version n. See co(1) for details. -xsuffixes Use suffixes to characterize RCS files. See ci(1) for details. -zzone Use zone as the time zone for keyword substitution; see co(1) for details. EXAMPLES
rcsclean *.c *.h removes all working files ending in .c or .h that were not changed since their checkout. rcsclean removes all working files in the current directory that were not changed since their checkout. FILES
rcsclean accesses files much as ci(1) does. ENVIRONMENT
RCSINIT Options prepended to the argument list, separated by spaces. A backslash escapes spaces within an option. The RCSINIT options are prepended to the argument lists of most RCS commands. Useful RCSINIT options include -q, -V, -x, and -z. RCS_MEM_LIMIT An integer lim, measured in kilobytes, specifying the threshold under which commands will try to use memory-based operations for processing the RCS file. (For RCS files of size lim kilobytes or greater, RCS will use the slower standard input/output routines.) Default value is 256. TMPDIR Name of the temporary directory. If not set, the environment variables TMP and TEMP are inspected instead and the first value found is taken; if none of them are set, a host-dependent default is used, typically /tmp. DIAGNOSTICS
The exit status is zero if and only if all operations were successful. Missing working files and RCS files are silently ignored. IDENTIFICATION
Author: Walter F. Tichy. Manual Page Revision: 5.8.1; Release Date: 2012-06-06. Copyright (C) 2010-2012 Thien-Thi Nguyen. Copyright (C) 1990, 1991, 1992, 1993 Paul Eggert. Copyright (C) 1982, 1988, 1989 Walter F. Tichy. SEE ALSO
ci(1), co(1), ident(1), rcs(1), rcsdiff(1), rcsmerge(1), rlog(1), rcsfile(5). Walter F. Tichy, RCS--A System for Version Control, Software--Practice & Experience 15, 7 (July 1985), 637-654. BUGS
At least one file must be given in older Unix versions that do not provide the needed directory scanning operations. GNU RCS 5.8.1 2012-06-06 RCSCLEAN(1)
All times are GMT -4. The time now is 05:39 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy