Sponsored Content
Full Discussion: bash - remove duplicates
Top Forums Shell Programming and Scripting bash - remove duplicates Post 302701283 by RudiC on Saturday 15th of September 2012 12:02:31 PM
Old 09-15-2012
Which URL should stay? And is the list a text file?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

fastest way to remove duplicates.

I have searched the FAQ - by using sort, duplicates, etc.... but I didn't get any articles or results on it. Currently, I am using: sort -u file1 > file2 to remove duplicates. For a file size of 1giga byte approx. time taken to remove duplicates is 1hr 21 mins. Is there any other faster way... (15 Replies)
Discussion started by: radhika
15 Replies

2. UNIX for Dummies Questions & Answers

How to remove duplicates without sorting

Hello, I can remove duplicate entries in a file by: sort File1 | uniq > File2 but how can I remove duplicates without sorting the file? I tried cat File1 | uniq > File2 but it doesn't work thanks (4 Replies)
Discussion started by: orahi001
4 Replies

3. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

4. UNIX for Dummies Questions & Answers

Remove duplicates from a file

Can u tell me how to remove duplicate records from a file? (11 Replies)
Discussion started by: saga20
11 Replies

5. Shell Programming and Scripting

Bash - remove duplicates without sort

I need to use bash to remove duplicates without using sort first. I can not use: cat file | sort | uniq But when I use only cat file | uniq some duplicates are not removed. (4 Replies)
Discussion started by: locoroco
4 Replies

6. Shell Programming and Scripting

Awk: Remove Duplicates

I have the following code for removing duplicate records based on fields in inputfile file & moves the duplicate records in duplicates file(1st Awk) & in 2nd awk i fetch the non duplicate entries in inputfile to tmp file and use move to update the original file. Requirement: Can both the awk... (4 Replies)
Discussion started by: siramitsharma
4 Replies

7. Shell Programming and Scripting

Help with merge and remove duplicates

Hi all, I need some help to remove duplicates from a file before merging. I have got 2 files: file1 has data in format 4300 23456 4301 2357 the 4 byte values on the right hand side is uniq, and are not repeated anywhere in the file file 2 has data in same format but is not in... (10 Replies)
Discussion started by: roy121
10 Replies

8. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

9. Shell Programming and Scripting

Sort and Remove duplicates

Here is my task : I need to sort two input files and remove duplicates in the output files : Sort by 13 characters from 97 Ascending Sort by 1 characters from 96 Ascending If duplicates are found retain the first value in the file the input files are variable length, convert... (4 Replies)
Discussion started by: ysvsr1
4 Replies

10. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies
SYSLOGOUT(8)						      System Manager's Manual						      SYSLOGOUT(8)

NAME
syslogout - modular centralized shell logout mechanism DESCRIPTION
syslogout is a generic approach to enable centralized shell logout actions for all users of a given system in a modular and centralized way mostly aimed at avoiding work for lazy sysadmins. It has only been tested to work with the bash shell. It basically consists of the small /etc/syslogout shell script which invokes other small shell scripts having a .bash suffix which are con- tained in the /etc/syslogout.d/ directory. The system administrator can drop in any script he wants without any naming convention other than that the scripts need to have a .bash suffix to enable automagic sourcing by the /etc/syslogout script. For shell sessions, the contents of /etc/syslogout.d/" will be sourced by every user at logout if the following lines are present in his $HOME/.bash_logout: if [ -f /etc/syslogout ]; then . /etc/syslogout fi If used for X sessions it is advisable to include the former statement into the Xreset script of the X display manager instead to prevent that closing of an terminal emulator window yields unexpected results in your running X session if your X11 terminal emulator is using a login shell. Be sure then to run it under the user-id of the X session's user. See the example files in /usr/share/doc/syslogout/ for illustration. Users not wanting /etc/syslogout to be sourced for their environment can easily disable it's automatic mechanism. It can be disabled by simply creating an empty file called $HOME/.nosyslogout in the user's home directory using e.g. the touch(1) command. Any single configuration file in /etc/syslogout.d/ can simply be overridden by any user by creating a private $HOME/.syslogout.d/ directory which may contain a user's own version of any configuration file to be sourced instead of the system default. It's names have just to match exactly the system's default /etc/syslogout.d/ configuration files. Empty versions of these files contained in the $HOME/.syslo- gout.d/ directory automatically disable sourcing of the system wide version. Naturally, users can add and include their own private scripts to be automagically executed by /etc/syslogout at logout time. OPTIONS
There are no options other than those dictated by shell conventions. Anything is defined within the configuration scripts themselves. SEE ALSO
The README files and configuration examples contained in /usr/share/doc/syslogout/ and the manual page for bash(1), xdm(1x), xdm.options(5), and wdm(1x). Recommended further reading is everything related with shell programming. If you need a similar mechanism for executing code at login time check out the related package sysprofile(8) which is a very close compan- ion to syslogout. BUGS
syslogout in its current form is mainly restricted to bash(1) syntax. In fact it is actually a rather embarrassing quick and dirty hack than anything else - but it works. It serves the practical need to enable a centralized bash configuration until something better becomes available. Your constructive criticism in making this into something better" is very welcome. Before i forget to mention it: we take patches... ;-) AUTHOR
syslogout was developed by Paul Seelig <pseelig@debian.org> specifically for the Debian GNU/Linux system. Feel free to port it to and use it anywhere else under the conditions of either the GNU public license or the BSD license or both. Better yet, please help to make it into something more worthwhile than it currently is. SYSLOGOUT(8)
All times are GMT -4. The time now is 11:32 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy