Sponsored Content
Full Discussion: removing duplicates
Top Forums Shell Programming and Scripting removing duplicates Post 302211958 by stevie_velvet on Saturday 5th of July 2008 11:41:24 AM
Old 07-05-2008
removing duplicates

Hi I have a file that are a list of people & their credentials i recieve frequently The issue is that whne I catnet this list that duplicat entries exists & are NOT CONSECUTIVE (i.e. uniq -1 may not weork here )
I'm trying to write a scrip that will remove duplicate entries
the script can typically made up of the following :
--------------
Ms AA
Unique to A
More of A

Mr BB

Mr CC

Ms AA
Unique to A
More of A

Mr DD

Mr EE

Mr BB


------------

Some of my technqiues of just are't working quite right especially with ignoring white spaces (maybe sed here)
(e.g. awk -F, '! mail[$3]++' inputfile )

any tips ?

ts

ms s
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing duplicates

Hi, I've been trying to removed duplicates lines with similar columns in a fixed width file and it's not working. I've search the forum but nothing comes close. I have a sample file: 27147140631203RA CCD * 27147140631203RA PPN * 37147140631207RD AAA 47147140631203RD JNA... (12 Replies)
Discussion started by: giannicello
12 Replies

2. UNIX for Dummies Questions & Answers

removing duplicates and sort -k

Hello experts, I am trying to remove all lines in a csv file where the 2nd columns is a duplicate. I am try to use sort with the key parameter sort -u -k 2,2 File.csv > Output.csv File.csv File Name|Document Name|Document Title|Organization Word Doc 1.doc|Word Document|Sample... (3 Replies)
Discussion started by: orahi001
3 Replies

3. Shell Programming and Scripting

Removing duplicates

Hi, I have a file in the below format., test test (10) to to (25) see see (45) and i need the output in the format of test 10 to 25 see 45 Some one help me? (6 Replies)
Discussion started by: imdadulla
6 Replies

4. UNIX for Advanced & Expert Users

removing duplicates.

Hi All In unix ,we have a file ,there we have to remove the duplicates by using one specific column. Can any body tell me the command. ex: file1 id,name 1,ww 2,qwq 2,asas 3,asa 4,asas 4,asas o/p: 1,ww 2,qwq 3,asa (7 Replies)
Discussion started by: raju4u
7 Replies

5. Shell Programming and Scripting

Removing duplicates

I have a test file with the following 2 columns: Col 1 | Col 2 T1 | 1 <= remove T5 | 1 T4 | 2 T1 | 3 T3 | 3 T4 | 1 <= remove T1 | 2 <= remove T3 ... (7 Replies)
Discussion started by: gctex
7 Replies

6. Emergency UNIX and Linux Support

Removing all the duplicates

i want to remove all the duplictaes in a file.I dont want even a single entry. For the input data: 12345|12|34 12345|13|23 3456|12|90 15670|12|13 12345|10|14 3456|12|13 i need the below data in one file 15670|12|13 and the below data in another file (9 Replies)
Discussion started by: pandeesh
9 Replies

7. Shell Programming and Scripting

Help in removing duplicates

I have an input file abc.txt with info like: abcd rateuse inklite robet rateuse abcd I need to remove duplicates from the file (eg: abcd,rateuse) from the file and need to place the contents in same file abc.txt if needed can be placed in another file. can anyone help me in this :( (4 Replies)
Discussion started by: rkrish
4 Replies

8. UNIX for Dummies Questions & Answers

Removing duplicates from a file

Hi All, I am merging files coming from 2 different systems ,while doing that I am getting duplicates entries in the merged file I,01,000131,764,2,4.00 I,01,000131,765,2,4.00 I,01,000131,772,2,4.00 I,01,000131,773,2,4.00 I,01,000168,762,2,2.00 I,01,000168,763,2,2.00... (5 Replies)
Discussion started by: Sri3001
5 Replies

9. Shell Programming and Scripting

Removing duplicates except the last occurrence

Hi All, i have a file like below, @DB_FCTS\src\Data\Scripts\Delete_CU_OM_BIL_PRT_STMT_TYP.sql @DB_FCTS\src\Data\Scripts\Delete_CDP_BILL_LBL_MSG.sql @DB_FCTS\src\Data\Scripts\Delete_OM_BIDDR.sql @DB_FCTS\src\Data\Scripts\Insert_CU_OM_LBL_MSG.sql... (11 Replies)
Discussion started by: mechvijays
11 Replies

10. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3 (2 Replies)
Discussion started by: sagar_1986
2 Replies
GCONFTOOL-2(1)							   User Commands						    GCONFTOOL-2(1)

NAME
gconftool-2 - GNOME configuration tool SYNOPSIS
gconftool-2 [-?] [-?|--help] [--usage] [-s|--set] [-g|--get] [--set-schema] [-u|--unset] [--recursive-unset] [-a|--all-entries] [--all-dirs] [--dump] [--load=STRING] [-R|--recursive-list] [--dir-exists=STRING] [--shutdown] [-p|--ping] [--spawn] [-t|--type int|bool|float|string|list|pair] [-T|--get-type] [--get-list-size] [--get-list-element] [--list-type=int|bool|float|string] [--car-type=int|bool|float|string] [--cdr-type=int|bool|float|string] [--short-desc=DESCRIPTION] [--long-desc=DESCRIPTION] [--owner=OWNER] [--install-schema-file=FILENAME] [--config-source=SOURCE] [--direct] [--makefile-install-rule] [--makefile-uninstall-rule] [--break-key] [--break-directory] [--short-docs] [--long-docs] [--get-schema-name] [--apply-schema] [--unapply-schema] [--get-default-source] [-v|--ver- sion] DESCRIPTION
Each preference in the GConf repository is expressed as a key-value pair. A GConf preference key is an element in the GConf repository that corresponds to an application prefer-ence. Preference keys typically have simple values such as strings, integers, or lists of strings and integers. The default key-value pairs are installed on the system, and are known as schemas. gconftool-2 is the command-line interface tool that enables you to set the values of keys, display the values of keys, and install schemas from schema definition files when you install an application. The GConf preference keys are stored and managed by the GConf daemon (gconfd-2). These keys are cached in memory, and saved to disk in XML format when appropriate. OPTIONS
-s, --set Set a key to a value and sync. Use with --type. -g, --get Print the value of a key to standard output. -u, --unset Unset the keys on the command line. --recursive-unset Recursively unset all keys at or below the key/directory names on the command line. -a, --all-entries Print all key/value pairs in a directory. --all-dirs Print all subdirectories in a directory. -R, --recursive-list Print all subdirectories and entries under a dir, recursively. --dir-exists=STRING Return 0 if the directory exists, 2 if it does not. -t, --type=int|bool|float|string|list|pair Specify the type of the value being set, or the type of the value a schema describes. Unique abbreviations are allowed. --list-type=int|bool|float|string Specify the type of the list value being set, or the type of the value a schema describes. Unique abbreviations are allowed. --car-type=int|bool|float|string Specify the type of the car pair value being set, or the type of the value a schema describes. Unique abbreviations are allowed. --cdr-type=int|bool|float|string Specify the type of the cdr pair value being set, or the type of the value a schema describes. Unique abbreviations are allowed. --short-desc=DESCRIPTION Specify a short half-line description to go in a schema. --long-desc=DESCRIPTION Specify a several-line description to go in a schema. --owner=OWNER Specify the owner of a schema. --install-schema-file=FILENAME Specify a schema file to be installed --config-source=SOURCE Specify a configuration source to use rather than the default path --direct Access the config database directly, bypassing server. Requires that gconfd is not running. --set-schema Set a schema and sync. Use with --short-desc, --long-desc, --owner, and --type. --makefile-install-rule Properly installs schema files on the command line into the database. GCONF_CONFIG_SOURCE environment variable should be set to a non-default config source or set to the empty string to use the default. --break-key Torture-test an application by setting and unsetting a bunch of values of different types for keys on the command line. --break-directory Torture-test an application by setting and unsetting a bunch of keys inside the directories on the command line. --short-docs Get the short doc string for a key --long-docs Get the long doc string for a key --get-schema-name Get the name of the schema applied to this key --apply-schema Specify the schema name followed by the key to apply the schema name to --unapply-schema Remove any schema name applied to the given keys --get-default-source Get the name of the default source. --shutdown Shut down gconfd. Don't use this option without good reason. -p, --ping Return 0 if gconfd is running, 2 if not. --spawn Launch the config server (gconfd). (Normally happens automatically when needed.) -v, --version Print version Help options -?, --help Show the help message. --usage Display brief usage message. EXAMPLES
Display the path to the current desktop wallpaper. gconftool-2 --get /desktop/gnome/background/picture_filename Change the widget font to "Serif 12" gconftool-2 --type string --set /desktop/gnome/interface/font_name "Serif 12" AUTHOR
GConf was written by Havoc Pennington <hp@redhat.com>. SEE ALSO
The GConf web site, http://www.gnome.org/projects/gconf/. gconftool-2 2.2.1 Feburary 2006 GCONFTOOL-2(1)
All times are GMT -4. The time now is 06:32 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy