06-09-2012
To do this a bit more information is needed:
1) is the file sorted, or are the lines you wish to 'keep' adjacent to each other in the file?
2) is the order of the output important? Do the lines 'kept' need to be in the same order that they appeared in the input?
3) do some lines appear more than twice, and should those be kept as well, or do you want to keep the lines that appear exactly twice?
4) how big is the file in terms of number of lines?
Last edited by agama; 06-09-2012 at 02:57 PM..
Reason: typo
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I am doing KSH script to remove duplicate lines in a file. Let say the file has format below.
FileA
1253-6856
3101-4011
1827-1356
1822-1157
1822-1157
1000-1410
1000-1410
1822-1231
1822-1231
3101-4011
1822-1157
1822-1231
and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies
2. UNIX for Advanced & Expert Users
Hi,
I have a file with duplicate lines in it. I want to keep only the duplicate lines and delete the non duplicates. Can some one please help me?
Regards
Narayana Gupta (3 Replies)
Discussion started by: guptan
3 Replies
3. UNIX for Dummies Questions & Answers
Hi,
I am trying to remove duplicate lines from a file. For example the contents of example.txt is:
this is a test
2342
this is a test
34343
this is a test
43434
and i want to remove the "this is a test" lines only and end up with the numbers in the file, that is, end up with:
2342... (4 Replies)
Discussion started by: ocelot
4 Replies
4. UNIX for Dummies Questions & Answers
Hi,
I am having a file which contains many duplicate lines. I wanted to redirect these duplicate lines into another file.
Suppose I have a file called file_dup.txt which contains some line as
file_dup.txt
A100-R1
ACCOUNTING-CONTROL
ACTONA-ACTASTOR
ADMIN-AUTH-STATS
ACTONA-ACTASTOR... (3 Replies)
Discussion started by: zing_foru
3 Replies
5. UNIX for Dummies Questions & Answers
I have a log file "logreport" that contains several lines as seen below:
04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping
06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping
07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies
6. Shell Programming and Scripting
Hi All,
I am trying to remove the duplicate entries in a file and print them just once. For example, if my input file has:
00:44,37,67,56,15,12
00:44,34,67,56,15,12
00:44,58,67,56,15,12
00:44,35,67,56,15,12
00:59,37,67,56,15,12
00:59,34,67,56,15,12
00:59,35,67,56,15,12... (7 Replies)
Discussion started by: faiz1985
7 Replies
7. UNIX for Advanced & Expert Users
Hi All,
I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space.
I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies
8. Shell Programming and Scripting
Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file.
I wrote the following script, but does not work. What is the problem?
The output file should only contain five lines:
Later! (5 Replies)
Discussion started by: Ernst
5 Replies
9. UNIX for Dummies Questions & Answers
I have a file with following data
A
B
C
I would like to print like this n times(For eg:5 times)
A
B
C
A
B
C
A
B
C
A
B
C
A (7 Replies)
Discussion started by: nsuresh316
7 Replies
10. Shell Programming and Scripting
Hi,
I have a csv file which contains some millions of lines in it.
The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line).
I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies
LEARN ABOUT DEBIAN
syslogout
SYSLOGOUT(8) System Manager's Manual SYSLOGOUT(8)
NAME
syslogout - modular centralized shell logout mechanism
DESCRIPTION
syslogout is a generic approach to enable centralized shell logout actions for all users of a given system in a modular and centralized way
mostly aimed at avoiding work for lazy sysadmins. It has only been tested to work with the bash shell.
It basically consists of the small /etc/syslogout shell script which invokes other small shell scripts having a .bash suffix which are con-
tained in the /etc/syslogout.d/ directory. The system administrator can drop in any script he wants without any naming convention other
than that the scripts need to have a .bash suffix to enable automagic sourcing by the /etc/syslogout script.
For shell sessions, the contents of /etc/syslogout.d/" will be sourced by every user at logout if the following lines are present in his
$HOME/.bash_logout:
if [ -f /etc/syslogout ]; then
. /etc/syslogout
fi
If used for X sessions it is advisable to include the former statement into the Xreset script of the X display manager instead to prevent
that closing of an terminal emulator window yields unexpected results in your running X session if your X11 terminal emulator is using a
login shell. Be sure then to run it under the user-id of the X session's user. See the example files in /usr/share/doc/syslogout/ for
illustration.
Users not wanting /etc/syslogout to be sourced for their environment can easily disable it's automatic mechanism. It can be disabled by
simply creating an empty file called $HOME/.nosyslogout in the user's home directory using e.g. the touch(1) command.
Any single configuration file in /etc/syslogout.d/ can simply be overridden by any user by creating a private $HOME/.syslogout.d/ directory
which may contain a user's own version of any configuration file to be sourced instead of the system default. It's names have just to
match exactly the system's default /etc/syslogout.d/ configuration files. Empty versions of these files contained in the $HOME/.syslo-
gout.d/ directory automatically disable sourcing of the system wide version.
Naturally, users can add and include their own private scripts to be automagically executed by /etc/syslogout at logout time.
OPTIONS
There are no options other than those dictated by shell conventions. Anything is defined within the configuration scripts themselves.
SEE ALSO
The README files and configuration examples contained in /usr/share/doc/syslogout/ and the manual page for bash(1), xdm(1x),
xdm.options(5), and wdm(1x). Recommended further reading is everything related with shell programming.
If you need a similar mechanism for executing code at login time check out the related package sysprofile(8) which is a very close compan-
ion to syslogout.
BUGS
syslogout in its current form is mainly restricted to bash(1) syntax. In fact it is actually a rather embarrassing quick and dirty hack
than anything else - but it works. It serves the practical need to enable a centralized bash configuration until something better
becomes available. Your constructive criticism in making this into something better" is very welcome. Before i forget to mention it: we
take patches... ;-)
AUTHOR
syslogout was developed by Paul Seelig <pseelig@debian.org> specifically for the Debian GNU/Linux system. Feel free to port it to and use
it anywhere else under the conditions of either the GNU public license or the BSD license or both. Better yet, please help to make it into
something more worthwhile than it currently is.
SYSLOGOUT(8)