Sponsored Content
Top Forums Shell Programming and Scripting Finding the duplicate in a file.... Post 302369796 by Reddy482 on Monday 9th of November 2009 03:33:27 PM
Old 11-09-2009
Finding the duplicate in a file....

Hi Unix Guru's

I had generated the uniqe code for every day date ranging from 20000101 to 21990101(200 years alomost 73000 uniqe codes ) and redirected it to text file.

Now My problem is i want to check whether there are any duplicates in unique code not PRESENT in the textfile ?

unique.txt Text file is in the following format.

YYYYMMDD|YYYYMM|UNIQUE CODE(ignore this row)

21120101|211201|F5CA1DD7746029E9C1CEF3137345D987
21120102|211201|F98804977D03F72DBC0AA0163B26F89E
21120103|211201|F01F29EC62E943978C934BCA79CD0140
21120104|211201|C943B6AB6BE9275D4D52B06BB484C59E
21120105|211201|A42873466FD7EF8FD211C82C52B2E1B2
21120106|211201|0179BB5B69E1433758E17DCA7D5A7D10
21120107|211201|30801625DDF75D0CC74E0255E994629E
21120108|211201|B758F26C1DCBC48F5BA62F38CED8B880

And also i want to print all the duplicates found. in the same formate of the record.

Thanks in Advance.
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Finding duplicate data in a file

A pogram named LOGGEDON returns an output of: Ref_num IP Address Logged on User 12000 10.10.12.12 12-02-2002 11:00 john 12004 10.10.12.13 12-03-2002 14:00 mary 12012 10.10.12.14 12-03-2002 11:30 bob 12024 ... (1 Reply)
Discussion started by: dinplant
1 Replies

2. Shell Programming and Scripting

finding duplicate files by size and finding pattern matching and its count

Hi, I have a challenging task,in which i have to find the duplicate files by its name and size,then i need to take anyone of the file.Then i need to open the file and find for more than one pattern and count of that pattern. Note:These are the samples of two files,but i can have more... (2 Replies)
Discussion started by: jerome Sukumar
2 Replies

3. Shell Programming and Scripting

Finding duplicate lines and deleting folders based on them

Hi, I have research data, which is organized to 100 folders numbered 00-99. I have many sets of 100 folders, for different values of initial parameters. For some reason, the computer that ran the program to gather the data, didn't always create a unique seed for each folder. I anticipated that... (1 Reply)
Discussion started by: Jopi
1 Replies

4. Shell Programming and Scripting

Finding Duplicate files

How do you delete and and find duplicate files? (1 Reply)
Discussion started by: Jicom4
1 Replies

5. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

6. Shell Programming and Scripting

Perl- Finding average "frequency" of occurrence of duplicate lines

Hello, I am working with a perl script that tries to find the average "frequency" in which lines are duplicated. So far I've only managed to find the way to count how many times the lines are repeated, the code is as follows: perl -ae' my $filename= $ENV{'i'}; open (FILE, "$filename") or... (10 Replies)
Discussion started by: acsg
10 Replies

7. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

8. Programming

Finding duplicate files in two base directories

Hello All, I have got some assignment to complete till this Monday and problem statement is as follow :- Problem :- Find duplicate files (especially .c and .cpp) from two project base directories with following requirement :- 1.Should be extendable to search in multiple base... (4 Replies)
Discussion started by: anand.shah
4 Replies

9. Shell Programming and Scripting

FINDING DUPLICATE PROJECT ( directory project )

I have a project tree like that. after running find command with the -no -empty option, i am able to have a list of non empty directory DO_MY_SEARCH="find . -type d -not -empty -print0" MY_EXCLUDE_DIR1=" -e NOT_IN_USE -e RTMAP -e NOT_USEFULL " echo " " > $MY_TEMP_RESULT_1 while... (2 Replies)
Discussion started by: jcdole
2 Replies
MSGUNIQ(1)								GNU								MSGUNIQ(1)

NAME
msguniq - unify duplicate translations in message catalog SYNOPSIS
msguniq [OPTION] [INPUTFILE] DESCRIPTION
Unifies duplicate translations in a translation catalog. Finds duplicate translations of the same message ID. Such duplicates are invalid input for other programs like msgfmt, msgmerge or msgcat. By default, duplicates are merged together. When using the --repeated option, only duplicates are output, and all other messages are discarded. Comments and extracted comments will be cumulated, except that if --use-first is specified, they will be taken from the first translation. File positions will be cumulated. When using the --unique option, duplicates are discarded. Mandatory arguments to long options are mandatory for short options too. Input file location: INPUTFILE input PO file -D, --directory=DIRECTORY add DIRECTORY to list for input files search If no input file is given or if it is -, standard input is read. Output file location: -o, --output-file=FILE write output to specified file The results are written to standard output if no output file is specified or if it is -. Message selection: -d, --repeated print only duplicates -u, --unique print only unique messages, discard duplicates Input file syntax: -P, --properties-input input file is in Java .properties syntax --stringtable-input input file is in NeXTstep/GNUstep .strings syntax Output details: -t, --to-code=NAME encoding for output --use-first use first available translation for each message, don't merge several translations -e, --no-escape do not use C escapes in output (default) -E, --escape use C escapes in output, no extended chars --force-po write PO file even if empty -i, --indent write the .po file using indented style --no-location do not write '#: filename:line' lines -n, --add-location generate '#: filename:line' lines (default) --strict write out strict Uniforum conforming .po file -p, --properties-output write out a Java .properties file --stringtable-output write out a NeXTstep/GNUstep .strings file -w, --width=NUMBER set output page width --no-wrap do not break long message lines, longer than the output page width, into several lines -s, --sort-output generate sorted output -F, --sort-by-file sort output by file location Informative output: -h, --help display this help and exit -V, --version output version information and exit AUTHOR
Written by Bruno Haible. REPORTING BUGS
Report bugs to <bug-gnu-gettext@gnu.org>. COPYRIGHT
Copyright (C) 2001-2007 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. SEE ALSO
The full documentation for msguniq is maintained as a Texinfo manual. If the info and msguniq programs are properly installed at your site, the command info msguniq should give you access to the complete manual. GNU gettext-tools 0.17 November 2007 MSGUNIQ(1)
All times are GMT -4. The time now is 09:58 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy