11-09-2009
Finding the duplicate in a file....
Hi Unix Guru's
I had generated the uniqe code for every day date ranging from 20000101 to 21990101(200 years alomost 73000 uniqe codes ) and redirected it to text file.
Now My problem is i want to check whether there are any duplicates in unique code not PRESENT in the textfile ?
unique.txt Text file is in the following format.
YYYYMMDD|YYYYMM|UNIQUE CODE(ignore this row)
21120101|211201|F5CA1DD7746029E9C1CEF3137345D987
21120102|211201|F98804977D03F72DBC0AA0163B26F89E
21120103|211201|F01F29EC62E943978C934BCA79CD0140
21120104|211201|C943B6AB6BE9275D4D52B06BB484C59E
21120105|211201|A42873466FD7EF8FD211C82C52B2E1B2
21120106|211201|0179BB5B69E1433758E17DCA7D5A7D10
21120107|211201|30801625DDF75D0CC74E0255E994629E
21120108|211201|B758F26C1DCBC48F5BA62F38CED8B880
And also i want to print all the duplicates found. in the same formate of the record.
Thanks in Advance.
9 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
A pogram named LOGGEDON returns an output of:
Ref_num IP Address Logged on User
12000 10.10.12.12 12-02-2002 11:00 john
12004 10.10.12.13 12-03-2002 14:00 mary
12012 10.10.12.14 12-03-2002 11:30 bob
12024 ... (1 Reply)
Discussion started by: dinplant
1 Replies
2. Shell Programming and Scripting
Hi,
I have a challenging task,in which i have to find the duplicate files by its name and size,then i need to take anyone of the file.Then i need to open the file and find for more than one pattern and count of that pattern.
Note:These are the samples of two files,but i can have more... (2 Replies)
Discussion started by: jerome Sukumar
2 Replies
3. Shell Programming and Scripting
Hi,
I have research data, which is organized to 100 folders numbered 00-99. I have many sets of 100 folders, for different values of initial parameters. For some reason, the computer that ran the program to gather the data, didn't always create a unique seed for each folder. I anticipated that... (1 Reply)
Discussion started by: Jopi
1 Replies
4. Shell Programming and Scripting
How do you delete and and find duplicate files? (1 Reply)
Discussion started by: Jicom4
1 Replies
5. UNIX for Dummies Questions & Answers
Hi Unix gurus,
Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me.
File format: CSV file
File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies
6. Shell Programming and Scripting
Hello,
I am working with a perl script that tries to find the average "frequency" in which lines are duplicated. So far I've only managed to find the way to count how many times the lines are repeated, the code is as follows:
perl -ae'
my $filename= $ENV{'i'};
open (FILE, "$filename") or... (10 Replies)
Discussion started by: acsg
10 Replies
7. Shell Programming and Scripting
Hi,
In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'.
In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies
8. Programming
Hello All,
I have got some assignment to complete till this Monday and problem statement is as follow :-
Problem :- Find duplicate files (especially .c and .cpp) from two project base directories with following requirement :-
1.Should be extendable to search in multiple base... (4 Replies)
Discussion started by: anand.shah
4 Replies
9. Shell Programming and Scripting
I have a project tree like that.
after running find command with the -no -empty option, i am able to have a list of non empty directory
DO_MY_SEARCH="find . -type d -not -empty -print0"
MY_EXCLUDE_DIR1=" -e NOT_IN_USE -e RTMAP -e NOT_USEFULL "
echo " " > $MY_TEMP_RESULT_1
while... (2 Replies)
Discussion started by: jcdole
2 Replies
MSGUNIQ(1) GNU MSGUNIQ(1)
NAME
msguniq - unify duplicate translations in message catalog
SYNOPSIS
msguniq [OPTION] [INPUTFILE]
DESCRIPTION
Unifies duplicate translations in a translation catalog. Finds duplicate translations of the same message ID. Such duplicates are invalid
input for other programs like msgfmt, msgmerge or msgcat. By default, duplicates are merged together. When using the --repeated option,
only duplicates are output, and all other messages are discarded. Comments and extracted comments will be cumulated, except that if
--use-first is specified, they will be taken from the first translation. File positions will be cumulated. When using the --unique
option, duplicates are discarded.
Mandatory arguments to long options are mandatory for short options too.
Input file location:
INPUTFILE
input PO file
-D, --directory=DIRECTORY
add DIRECTORY to list for input files search
If no input file is given or if it is -, standard input is read.
Output file location:
-o, --output-file=FILE
write output to specified file
The results are written to standard output if no output file is specified or if it is -.
Message selection:
-d, --repeated
print only duplicates
-u, --unique
print only unique messages, discard duplicates
Input file syntax:
-P, --properties-input
input file is in Java .properties syntax
--stringtable-input
input file is in NeXTstep/GNUstep .strings syntax
Output details:
-t, --to-code=NAME
encoding for output
--use-first
use first available translation for each message, don't merge several translations
-e, --no-escape
do not use C escapes in output (default)
-E, --escape
use C escapes in output, no extended chars
--force-po
write PO file even if empty
-i, --indent
write the .po file using indented style
--no-location
do not write '#: filename:line' lines
-n, --add-location
generate '#: filename:line' lines (default)
--strict
write out strict Uniforum conforming .po file
-p, --properties-output
write out a Java .properties file
--stringtable-output
write out a NeXTstep/GNUstep .strings file
-w, --width=NUMBER
set output page width
--no-wrap
do not break long message lines, longer than the output page width, into several lines
-s, --sort-output
generate sorted output
-F, --sort-by-file
sort output by file location
Informative output:
-h, --help
display this help and exit
-V, --version
output version information and exit
AUTHOR
Written by Bruno Haible.
REPORTING BUGS
Report bugs to <bug-gnu-gettext@gnu.org>.
COPYRIGHT
Copyright (C) 2001-2007 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
SEE ALSO
The full documentation for msguniq is maintained as a Texinfo manual. If the info and msguniq programs are properly installed at your
site, the command
info msguniq
should give you access to the complete manual.
GNU gettext-tools 0.17 November 2007 MSGUNIQ(1)