02-25-2009
Shell script to remove duplicates lines in a file
Hi,
I am writing a shell script that needs to remove duplicate lines within a file by category.
example:
section a
a
c
b
a
section b
a
b
a
c
I need to remove the duplicates within th category with out removing the duplicates from the 2 different sections (one of the a's in section a, and one of the a's in section b).
I wanted to use uniq but i have to sort the file first...which takes the lines out of the sections they are supposed to be in and sorts the entire file which I don't want.....but I wouldn't mind sorting it by category if possible.
any help is appreciated.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi gurus,
I have this file with blank lines in it.
How do i remove them in shell?
I tried these commands but not working:
sed '/^ *$/d'
or
sed '/^$/d'
Anybody has a better idea pls?
Also there are lines which starts with a single space, how do we remove the space in those lines?... (3 Replies)
Discussion started by: gholdbhurg
3 Replies
2. Shell Programming and Scripting
hi,,
i hav a file with many lines.i need to remove all lines before a line begginning with a specific pattern from the file because these lines are not required.
Can u help me out with either a perl script or shell script
example:-
if file initially contains lines:
a
b
c
d
.1.2
d
e
f... (2 Replies)
Discussion started by: raksha.s
2 Replies
3. Shell Programming and Scripting
Input:
a
b
b
c
d
d
I need:
a
c
I know how to get this (the lines that have duplicates) :
b
d
sort file | uniq -d
But i need opossite of this. I have searched the forum and other places as well, but have found solution for everything except this variant of the problem. (3 Replies)
Discussion started by: necroman08
3 Replies
4. Shell Programming and Scripting
Can some one provide me a shell script.
I have file with many columns and many rows. need to sort the first column and then remove the duplicates records if exists.. finally print the full data with first coulm as unique.
Sort BASED ON FIRST FIELD and remove the duplicates if exists... (2 Replies)
Discussion started by: tuffEnuff
2 Replies
5. Shell Programming and Scripting
Hi
I need a script that removes the duplicate records and write it to a new file
for example I have a file named test.txt and it looks like
abcd.23
abcd.24
abcd.25
qwer.25
qwer.26
qwer.98
I want to pick only $1 and compare with the next record and the output should be
abcd.23... (6 Replies)
Discussion started by: antointoronto
6 Replies
6. Shell Programming and Scripting
I have existing XML file as below, now based on input string in shell script on workordercode i need to create a seprate xml file
for e.g if we pass the input string as 184851 then it find the tag data from <workOrder>..</workOrder> and write to a new file and similarly next time if i pass the... (3 Replies)
Discussion started by: balrajg
3 Replies
7. UNIX for Dummies Questions & Answers
Hello experts!
I'd like a way to remove duplicates per line. Strings are enclosed in brackets, and I would prefer to maintain the order of the file:
example input
(56)(63)
(56)(70)(56)(70)(24)
(25)(78)
(12)(33)(12)
(10)
(10)
desired output
(56)(63)
(56)(70)(24)
(25)(78)... (5 Replies)
Discussion started by: torchij
5 Replies
8. UNIX for Dummies Questions & Answers
I have a file called FILE
cat FILE
11/11/2012
11/11/2012
12/11/2012
15/11/2012
need to remove the duplicates dates ( ie 11/11/2012 is present two times i need remove one duplicates date )
Need outputs like this
11/11/2012
12/11/2012
15/11/2012
I have tried using awk... (8 Replies)
Discussion started by: Venkatesh1
8 Replies
9. Shell Programming and Scripting
Hi All,
My name is Prathyu and I am working as a ETL develper. I have one requirement to create a XML file based on the provided XSD file. As per the Datastage standards Key(repeatable) field does not contain any Null values so I am inserting some dummy tag line to that XML file.
... (14 Replies)
Discussion started by: Prathyu
14 Replies
10. Shell Programming and Scripting
Please help me on this
My script name is uniqueArray.csh
#!/bin/csh
set ARRAY = ( one teo three one three )
set ARRAY = ( $ARRAY one five three five )
How to remove the duplicates in this array ,sort and save those in the same variable or different variable.
Thanks in the advance
... (3 Replies)
Discussion started by: SA_Palani
3 Replies
inifile(3tcl) Parsing of Windows INI files inifile(3tcl)
__________________________________________________________________________________________________________________________________________________
NAME
inifile - Parsing of Windows INI files
SYNOPSIS
package require Tcl 8.2
package require inifile ?0.2.4?
::ini::open file ?access?
::ini::close ini
::ini::commit ini
::ini::revert ini
::ini::filename ini
::ini::sections ini
::ini::keys ini section
::ini::get ini section
::ini::exists ini section ?key?
::ini::value ini section key ?default?
::ini::set ini section key value
::ini::delete ini section ?key?
::ini::comment ini section ?key? ?text?
::ini::commentchar ?char?
_________________________________________________________________
DESCRIPTION
This package provides an interface for easy manipulation of Windows INI files.
::ini::open file ?access?
Opens an INI file and returns a handle that is used by other commands. access is the same as the first form (non POSIX) of the open
command, with the exception that mode a is not supported. The default mode is r+.
::ini::close ini
Close the specified handle. If any changes were made and not written by commit they are lost.
::ini::commit ini
Writes the file and all changes to disk. The sections are written in arbitrary order. The keys in a section are written in alphabet-
ical order. If the ini was opened in read only mode an error will be thrown.
::ini::revert ini
Rolls all changes made to the inifile object back to the last committed state.
::ini::filename ini
Returns the name of the file the ini object is associated with.
::ini::sections ini
Returns a list of all the names of the existing sections in the file handle specified.
::ini::keys ini section
Returns a list of all they key names in the section and file specified.
::ini::get ini section
Returns a list of key value pairs that exist in the section and file specified.
::ini::exists ini section ?key?
Returns a boolean value indicating the existance of the specified section as a whole or the specified key within that section.
::ini::value ini section key ?default?
Returns the value of the named key and section. If specified, the default value will be returned if the key does not exist. If the
key does not exist and no default is specified an error will be thrown.
::ini::set ini section key value
Sets the value of the key in the specified section. If the section does not exist then a new one is created.
::ini::delete ini section ?key?
Removes the key or the entire section and all its keys. A section is not automatically deleted when it has no remaining keys.
::ini::comment ini section ?key? ?text?
Reads and modifies comments for sections and keys. To write a section comment use an empty string for the key. To remove all com-
ments use an empty string for text. text may consist of a list of lines or one single line. Any embedded newlines in text are prop-
erly handled. Comments may be written to nonexistant sections or keys and will not return an error. Reading a comment from a nonex-
istant section or key will return an empty string.
::ini::commentchar ?char?
Reads and sets the comment character. Lines that begin with this character are treated as comments. When comments are written out
each line is preceded by this character. The default is ;.
BUGS, IDEAS, FEEDBACK
This document, and the package it describes, will undoubtedly contain bugs and other problems. Please report such in the category inifile
of the Tcllib SF Trackers [http://sourceforge.net/tracker/?group_id=12883]. Please also report any ideas for enhancements you may have for
either package and/or documentation.
CATEGORY
Text processing
inifile 0.2.4 inifile(3tcl)