Sponsored Content
Top Forums Shell Programming and Scripting Identifying dupes within a database and creating unique sub-sets Post 302879976 by gimley on Monday 16th of December 2013 11:00:35 PM
Old 12-17-2013
Many thanks. It was pretty fast. Zipped through 20,000 lines in a few seconds. I doubt that there are any issues, since I tested the output file for dupes and there were none.
Many thanks.
 

9 More Discussions You Might Find Interesting

1. Programming

Creating a Unique ID on distributed systems

Hi, How do you actually create a unique ID on a distributed system. I looked at gethostid but the man page says that its not guaranteed to be unique. Also using the IP address does not seem to be a feasible solution. Is there a function call or mechanism by which this is possible when even the... (4 Replies)
Discussion started by: pic
4 Replies

2. Virtualization and Cloud Computing

Clouds (Partially Order Sets) - Streams (Linearly Ordered Sets) - Part 2

timbass Sat, 28 Jul 2007 10:07:53 +0000 Originally posted in Yahoo! CEP-Interest Here is my follow-up note on posets (partially ordered sets) and tosets (totally or linearly ordered sets) as background set theory for event processing, and in particular CEP and ESP. In my last note, we... (0 Replies)
Discussion started by: Linux Bot
0 Replies

3. UNIX for Dummies Questions & Answers

split a file with unique sets

This may sound like a trivial problem, but I still need some help: I have a file with ids and I want to split it 'n' ways (could be any number) into files: 1 1 1 2 2 3 3 4 5 5 Let's assume 'n' is 3, and we cannot have the same id in two different partitions. So the partitions may... (8 Replies)
Discussion started by: ChicagoBlues
8 Replies

4. UNIX for Dummies Questions & Answers

Identifying the commands creating subshells

Hi all, This is the basic question. I have read many books which advised to avoid creating sub shells. e.g: use wc -l<filename rather than using cat file|wc -l. So, how to identify whether a command creates subshell or not? so,is it better to use tail -n+1 file in stead of using cat.... (3 Replies)
Discussion started by: pandeesh
3 Replies

5. Shell Programming and Scripting

Script for identifying and deleting dupes in a line

I am compiling a synonym dictionary which has the following structure Headword=Synonym1,Synonym2 and so on, with each synonym separated by a comma. As is usual in such cases manual preparation of synonyms results in repeating the synonym which results in dupes as in the example below:... (3 Replies)
Discussion started by: gimley
3 Replies

6. Programming

Unique Number Identifying

I'm trying to solve the below problem for a number: Enter a number and if it has all unique digits print unique number else non-unique number. Eg: Input=123; Output=unique number Input=112; Output=Non-unique number The thing i tried is splitting the number into digits by using % operator... (2 Replies)
Discussion started by: Gautham
2 Replies

7. Shell Programming and Scripting

Help with Perl script for identifying dupes in column1

Dear all, I have a large dictionary database which has the following structure source word=target word e.g. book=livre Since the database is very large in spite of all the care taken, it so happens that at times the source word is repeated e.g. book=livre book=tome Since I want to... (7 Replies)
Discussion started by: gimley
7 Replies

8. Shell Programming and Scripting

Identifying single words in a dictionary database

I am reworking a Marathi-English dictionary to be out on open-source. My dictionary has the Headword in Marathi, followed by its Part of Speech and subsequently by its English glosses as in the examples below; अकरसणें v i To contract, shrink. अकरा a Eleven. अकराळ a Frightful, terrible. विकराळ... (2 Replies)
Discussion started by: gimley
2 Replies

9. Shell Programming and Scripting

Regex to identify unique words in a dictionary database

Hello, I have a dictionary which I am building for the Open Source Community. The data structure is as under HEADWORD=PARTOFSPEECH=ENGLISH MEANING as shown in the example below अ=m=Prefix signifying negation. अँहँ=ind=Interjection expressing disapprobation. अं=int=An interjection... (2 Replies)
Discussion started by: gimley
2 Replies
EMGRIP-DUPES(1) 					User Contributed Perl Documentation					   EMGRIP-DUPES(1)

NAME
emgrip-dupes - find packages listed in more than one component Synopsis Syntax: emgrip-dupes -b PATH [OPTIONS] emgrip-dupes -b PATH -m|--merge NAME [OPTIONS] emgrip-dupes -b PATH -p|--purge NAME [OPTIONS] emgrip-dupes -?|-h|--help|--version Commands: -b|--base-path PATH: path to the top level grip directory [required] -a|--arch ARCHITECTURE: architecture to test [default: i386] -m|--merge NAMES: retain this duplicate at the latest version in all -p|--purge NAMES: remove the duplicates from 'main' -t|--trim NAMES: retain the duplicates in main only -?|-h|--help|--version: print this help message and exit Options: --grip-name STRING: alternative name for the grip repository -s|--suite SUITE: suite to check (default: unstable) -n|--dry-run: print the reprepro commands that would be used. Description emgrip-dupes scans the Grip repository Packages data and configuration, identifies the supported list of components in the requested suite. In some cases, these duplicates are useful and only a small amount of space is taken up by the extra listing. However, the version in one component can easily be out of sync with the version in another. The main emphasis is on the size of the Packages file for the 'main' component (the one that every user needs to download). Purge mode will remove the listing of the specified package from 'main'. Merge mode will bring the outdated version into line with the most recent version of the package so that all components list the most recent version. Limitations Next step is to automate the "correction" of the duplicates but this does need care. Manual corrections involve identifying the packages to retain in main (where the duplicate in dev, doc or debug is not wanted) and pass those to --trim. The more complex case is to remove from main (e.g. package name suffix is -dev or -doc or -dbg or the Section is devel, dbg, doc or libdevel). emgrip-dupes --purge removes each binary separately because removing the package from main in a single operation will also remove the source. This is a particular problem if the source package also builds binary packages that are intended for main, e.g. dbus. Copyright and Licence Copyright (C) 2009 Neil Williams <codehelp@debian.org> This package is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>. perl v5.12.3 2011-03-27 EMGRIP-DUPES(1)
All times are GMT -4. The time now is 01:54 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy