Sponsored Content
Full Discussion: Script to remove duplicates
Top Forums Shell Programming and Scripting Script to remove duplicates Post 302417090 by Corona688 on Wednesday 28th of April 2010 12:56:07 PM
Old 04-28-2010
Code:
#!/bin/sh

PREV_A=""
while IFS="." read A B
do
        [ "${PREV_A}" == "$A" ] || echo "$A.$B"
        PREV_A="$A"
done < file

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicates

Hello Experts, I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra... (4 Replies)
Discussion started by: forumthreads
4 Replies

2. Shell Programming and Scripting

Shell script to remove duplicates lines in a file

Hi, I am writing a shell script that needs to remove duplicate lines within a file by category. example: section a a c b a section b a b a c I need to remove the duplicates within th category with out removing the duplicates from the 2 different sections (one of the a's in section... (1 Reply)
Discussion started by: RichElks
1 Replies

3. Shell Programming and Scripting

Remove duplicates from a file

Hi, I need to remove duplicates from a file. The file will be like this 0003 10101 20100120 abcdefghi 0003 10101 20100121 abcdefghi 0003 10101 20100122 abcdefghi 0003 10102 20100120 abcdefghi 0003 10103 20100120 abcdefghi 0003 10103 20100121 abcdefghi Here if the first colum and... (6 Replies)
Discussion started by: gpaulose
6 Replies

4. Shell Programming and Scripting

remove duplicates and sort

Hi, I'm using the below command to sort and remove duplicates in a file. But, i need to make this applied to the same file instead of directing it to another. Thanks (6 Replies)
Discussion started by: dvah
6 Replies

5. Shell Programming and Scripting

bash - remove duplicates

I need to use a bash script to remove duplicate files from a download list, but I cannot use uniq because the urls are different. I need to go from this: http://***/fae78fe/file1.wmv http://***/39du7si/file1.wmv http://***/d8el2hd/file2.wmv http://***/h893js3/file2.wmv to this: ... (2 Replies)
Discussion started by: locoroco
2 Replies

6. UNIX for Dummies Questions & Answers

script to remove duplicates per line

Hello experts! I'd like a way to remove duplicates per line. Strings are enclosed in brackets, and I would prefer to maintain the order of the file: example input (56)(63) (56)(70)(56)(70)(24) (25)(78) (12)(33)(12) (10) (10) desired output (56)(63) (56)(70)(24) (25)(78)... (5 Replies)
Discussion started by: torchij
5 Replies

7. Shell Programming and Scripting

Remove duplicates

I have a file with the following format: fields seperated by "|" title1|something class|long...content1|keys title2|somhing class|log...content1|kes title1|sothing class|lon...content1|kes title3|shing cls|log...content1|ks I want to remove all duplicates with the same "title field"(the... (3 Replies)
Discussion started by: dtdt
3 Replies

8. Shell Programming and Scripting

Remove top 3 duplicates

hello , I have a requirement with input in below format abc 123 xyz bcd 365 kii abc 987 876 cdf 987 uii abc 456 yuu bcd 654 rrr Expecting Output abc 456 yuu bcd 654 rrr cdf 987 uii (1 Reply)
Discussion started by: Tomlight
1 Replies

9. Shell Programming and Scripting

Remove duplicates

Hi I have a below file structure. 200,1245,E1,1,E1,,7611068,KWH,30, ,,,,,,,, 200,1245,E1,1,E1,,7611070,KWH,30, ,,,,,,,, 300,20140223,0.001,0.001,0.001,0.001,0.001 300,20140224,0.001,0.001,0.001,0.001,0.001 300,20140225,0.001,0.001,0.001,0.001,0.001 300,20140226,0.001,0.001,0.001,0.001,0.001... (1 Reply)
Discussion started by: tejashavele
1 Replies

10. Shell Programming and Scripting

How to remove duplicates using for loop?

values=(1 2 3 5 4 2 3 1 6 8 3 5 ) #i need the output like this by removing the duplicates 1 2 3 5 4 6 8 #i dont need sorting in my program #plz explain me as simple using for loop #os-ubuntu ,shell=bash (5 Replies)
Discussion started by: Meeran Rizvi
5 Replies
incrontab(5)						       incron documentation						      incrontab(5)

NAME
incrontab - tables for driving inotify cron (incron) DESCRIPTION
An incrontab file contains instructions to the incrond(8) daemon of the general form: "run this command on these file events". There are two categories of tables: system tables (with root privileges) and user tables (with user privileges). System tables are (by default) located in /etc/incron.d and may have any names. Each system table exists separately inside incron and their watches never collide. Each user has their own table, and commands in any given incrontab will be executed as the user who owns the incrontab. System users (such as apache, postfix, nobody etc.) may have their own incrontab. incrontab files are read when the incrond(8) daemon starts and after any change (incrontab file are being hooked when incrond is running). Blank lines are ignored. The general line format is the following: <path> <mask> <command> Where path is an absolute filesystem path, mask is an event mask (in symbolic or numeric form) and command is an executable file (or a script) with its arguments. See bellow for event mask symbols. The executable file may be noted as an absolute path or only as the name itself (PATH locations are examined). Please remember that the same path may occur only once per table (otherwise only the first occurrence takes effect and an error message is emitted to the system log). EVENT SYMBOLS
These basic event mask symbols are defined: IN_ACCESS File was accessed (read) (*) IN_ATTRIB Metadata changed (permissions, timestamps, extended attributes, etc.) (*) IN_CLOSE_WRITE File opened for writing was closed (*) IN_CLOSE_NOWRITE File not opened for writing was closed (*) IN_CREATE File/directory created in watched directory (*) IN_DELETE File/directory deleted from watched directory (*) IN_DELETE_SELF Watched file/directory was itself deleted IN_MODIFY File was modified (*) IN_MOVE_SELF Watched file/directory was itself moved IN_MOVED_FROM File moved out of watched directory (*) IN_MOVED_TO File moved into watched directory (*) IN_OPEN File was opened (*) When monitoring a directory, the events marked with an asterisk (*) above can occur for files in the directory, in which case the name field in the returned event data identifies the name of the file within the directory. The IN_ALL_EVENTS symbol is defined as a bit mask of all of the above events. Two additional convenience symbols are IN_MOVE, which is a combination of IN_MOVED_FROM and IN_MOVED_TO, and IN_CLOSE which combines IN_CLOSE_WRITE and IN_CLOSE_NOWRITE. The following further symbols can be specified in the mask: IN_DONT_FOLLOW Don't dereference pathname if it is a symbolic link IN_ONESHOT Monitor pathname for only one event IN_ONLYDIR Only watch pathname if it is a directory Additionally, there is a symbol which doesn't appear in the inotify symbol set. It it IN_NO_LOOP. This symbol disables monitoring events until the current one is completely handled (until its child process exits). WILDCARDS
The following wildards may be used inside command specification: $$ dollar sign $@ watched filesystem path (see above) $# event-related file name $% event flags (textually) $& event flags (numerically) EXAMPLE
These are some example rules which can be used in an incrontab file: /tmp IN_ALL_EVENTS abcd $@/$# $% /usr/bin IN_ACCESS,IN_NO_LOOP abcd $# /home IN_CREATE /usr/local/bin/abcd $# /var/log 12 abcd $@/$# The first line monitors all events on the /tmp directory. When an event occurs it runs a application called 'abcd' with the full path of the file as the first arguments and the event flags as the second one. The second line monitors accesses (readings) on the /usr/bin directory. The application 'abcd' is run as a handler and the appropriate event watch is disabled until the program finishes. The file name (without the directory path) is passed in as an argument. The third example is used for monitoring the /home directory for newly create files or directories (it practically means an event is sent when a new user is added). This event is processed by a program specified by an absolute path. And the final line shows how to use numeric event mask instead of textual one. The value 12 is exactly the same as IN_ATTRIB,IN_CLOSE_WRITE. SEE ALSO
incrond(8), incrontab(1), incron.conf(5) AUTHOR
Lukas Jelinek <lukas@aiken.cz> (please report bugs to http://bts.aiken.cz or <bugs@aiken.cz>). COPYING
This program is free software. It can be used, redistributed and/or modified under the terms of the GNU General Public License, version 2. Lukas Jelinek 0.5.10 incrontab(5)
All times are GMT -4. The time now is 03:12 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy