Sponsored Content
Top Forums Shell Programming and Scripting Find duplicate based on 'n' fields and mark the duplicate as 'D' Post 302593698 by machomaddy on Saturday 28th of January 2012 02:07:24 AM
Old 01-28-2012
Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi,

In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'.

In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create a new field at the end of the records as "C" or "D")

Code:
File 1
====
Man_ID|Man_Dt|Ship_Id|Ship_Dt|ItemID|Noof ITEMS|ItemNam
001|2010-12-31|11|2010-12-31|111|2|Jackets
002|2010-12-31|12|2010-12-31|111|1|Caps
001|2010-12-31|11|2009-11-31|111|2|Jackets
001|2010-12-31|11|2011-12-31|111|2|Jackets
003|2010-11-01|13|2011-12-31|111|1|Shoes

Expected Output

File 1
=====
Man_ID|Man_Dt|Ship_Id|Ship_Dt|ItemID|Noof ITEMS|ItemNam
 001|2010-12-31|11|2010-12-31|111|2|Jackets|D
002|2010-12-31|12|2010-12-31|111|1|Caps
001|2010-12-31|11|2009-11-31|111|2|Jackets|D
 001|2010-12-31|11|2011-12-31|111|2|Jackets|C
003|2010-11-01|13|2011-12-31|111|1|Shoes


Last edited by machomaddy; 01-28-2012 at 06:37 AM.. Reason: Edited wrong Input "2010-12-31" to "2011-12-31" in the 4th record
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract duplicate fields in rows

I have a input file with formating: 6000000901 ;36200103 ;h3a01f496 ; 2000123605 ;36218982 ;heefa1328 ; 2000273132 ;36246985 ;h08c5cb71 ; 2000041207 ;36246985 ;heef75497 ; Each fields is seperated by semi-comma. Sometime, the second files is... (6 Replies)
Discussion started by: anhtt
6 Replies

2. Shell Programming and Scripting

compare fields in a file with duplicate records

Hi: I've been searching the net but didnt find a clue. I have a file in which, for some records, some fields coincide. I want to compare one (or more) of the dissimilar fields and retain the one record that fulfills a certain condition. For example, on this file: 99 TR 1991 5 06 ... (1 Reply)
Discussion started by: rleal
1 Replies

3. Shell Programming and Scripting

awk 2 fields duplicate and 1 different

I have file that I need to remove the duplicates. The problem is, I need to only keep the one which has a unique 3rd field. Here is a sample file: xxx.xxx:x:CISCO1.CLEVE61W:ERIE.NET:x:x:x:x: xxx.xxx:x:CISCO2.CLEVE62W:OHIO.NET:x:x:x:x: xxx.xxx:x:CISCO2.CLEVE62W:NORTH.NET:x:x:x:x:... (1 Reply)
Discussion started by: numele
1 Replies

4. Shell Programming and Scripting

Filter or remove duplicate block of text without distinguishing marks or fields

Hello, Although I have found similar questions, I could not find advice that could help with our problem. The issue: We have several hundreds text files containing repeated blocks of text (I guess back at the time they were prepared like that to optmize printing). The block of texts... (13 Replies)
Discussion started by: samask
13 Replies

5. Shell Programming and Scripting

Remove duplicate based on Group

Hi, How can I remove duplicates from a file based on group on other column? for example: Test1|Test2|Test3|Test4|Test5 Test1|Test6|Test7|Test8|Test5 Test1|Test9|Test10|Test11|Test12 Test1|Test13|Test14|Test15|Test16 Test17|Test18|Test19|Test20|Test21 Test17|Test22|Test23|Test24|Test5 ... (2 Replies)
Discussion started by: yale_work
2 Replies

6. Shell Programming and Scripting

Join fields from files with duplicate lines

I have two files, file1.txt: 1 abc 2 def 2 dgh 3 ijk 4 lmn file2.txt 1 opq 2 rst 3 uvw My desired output is: 1 abc opq 2 def rst 2 dgh rst 3 ijk uvw (2 Replies)
Discussion started by: xan.amini
2 Replies

7. Shell Programming and Scripting

How To Remove Duplicate Based on the Value?

Hi , Some time i got duplicated value in my files , bundle_identifier= B Sometext=ABC bundle_identifier= A bundle_unit=500 Sometext123=ABCD bundle_unit=400 i need to check if there is a duplicated values or not if yes , i need to check if the value is A or B when Bundle_Identified ,... (2 Replies)
Discussion started by: OTNA
2 Replies

8. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

9. Shell Programming and Scripting

Find duplicate values in specific column and delete all the duplicate values

Dear folks I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same... (4 Replies)
Discussion started by: sajmar
4 Replies

10. UNIX for Beginners Questions & Answers

Discarding records with duplicate fields

Hi, My input looks like this (tab-delimited): grp1 name2 firstname M 55 item1 item1.0 grp1 name2 firstname F 55 item1 item1.0 grp2 name1 firstname M 55 item1 item1.0 grp2 name2 firstname M 55 item1 item1.0 Using awk, I am trying to discard the records with common fields 2, 4, 5, 6, 7... (4 Replies)
Discussion started by: beca123456
4 Replies
NUMRANGE(1)						User Contributed Perl Documentation					       NUMRANGE(1)

NAME
numrange - Print out a range of numbers for use in for loops and such. SYNOPSIS
numrange [-dhV] /<expression>/ DESCRIPTION
numrange will print out a list of numbers based on an expression that you specify. This is useful for making a list of numbers for use in for loops and so on. Ranges are inclusive. Ranges of numbers are specified using the .. operator, like this /20..50/, which means all integers from 20 to 50 inclusive. More complex expressions can be generated using the commas and the 'i' increment operator. OPTIONS
-e <set> Exclude the <set> of numbers from the range output. <set> is a set of numbers separated by commas. -n <n> Use <n> as the separator between numbers. By default, it will use a space. Use ' ' or \n for a newline character or use the -N option. -N Just a quick option for using a newline as the separator. -h Help: You're looking at it. -V Increase verbosity. -d Debug mode. For developers EXAMPLES
All numbers from 1 to 10. $ numrange /1..10/ 1 2 3 4 5 6 7 8 9 10 From 10 to 1. Counting down. $ numrange /10..1/ 10 9 8 7 6 5 4 3 2 1 From 1 to 10 and from 15 to 20. $ numrange /1..10,15..20/ 1 2 3 4 5 6 7 8 9 10 15 16 17 18 19 20 Even numbers from 0 to 10 $ numrange /0..10i2/ 0 2 4 6 8 10 Odd numbers. Notice the starting number in the range expression. $ numrange /1..10i2/ 1 3 5 7 9 Factors of 3 between 99 and 120. $ numrange /99..120i3/ 99 102 105 108 111 114 117 120 Decimal numbers $ numrange /1.1..2.5i0.1/ 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 2.1 2.2 2.3 2.4 2.5 And negative numbers too. $ numrange /1.0..-2.0i0.3/ 1 0.7 0.4 0.1 -0.2 -0.5 -0.8 -1.1 -1.4 -1.7 -2 You can also pad numbers when you are counting up. This is a trick of how the Perl programming language deals with ranges: $ numrange /01..15/ 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 BUGS
Even though you can do zero padding on simple ranges, like 001..100, it will not pad zeros on complex ranges like 001..100i2, or for counting downwards. SEE ALSO
seq(1), numaverage(1), numbound(1), numinterval(1), numnormalize(1), numgrep(1), numprocess(1), numsum(1), numrandom(1), numround(1) COPYRIGHT
numrange is part of the num-utils package, which is copyrighted by Suso Banderas and released under the GPL license. Please read the COPYING and LICENSE files that came with the num-utils package Developers can read the GOALS file and contact me about providing submitions or help for the project. MORE INFO
More info on numrange can be found at: http://suso.suso.org/programs/num-utils/ perl v5.10.1 2009-10-31 NUMRANGE(1)
All times are GMT -4. The time now is 03:01 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy