Sponsored Content
Full Discussion: Checking file for duplicates
Top Forums Shell Programming and Scripting Checking file for duplicates Post 302422737 by curleb on Wednesday 19th of May 2010 09:11:25 AM
Old 05-19-2010
Being a theoretical question as opposed to a real how-to question, what are you already doing to process the first file? Bear in mind that what you're asking for is more of a framework question than it is a scripting issue.

There are plenty of ways to skin the cat, but which way have you started to do it? No sense in us providing a method that doesn't fit your approach.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Avoid Duplicates in a file

Hi Gurus, I had a question regarding avoiding duplicates.i have a file abc.txt abc.txt ------- READER_1_1_1> HIER_28056 XML Reader: Error occurred while parsing:; line number ; column number READER_1_3_1> Sun Mar 23 23:52:48 2008 READER_1_3_1> HIER_28056 XML Reader: Error occurred while... (7 Replies)
Discussion started by: pssandeep
7 Replies

2. Shell Programming and Scripting

Remove duplicates from a file

Hi, I need to remove duplicates from a file. The file will be like this 0003 10101 20100120 abcdefghi 0003 10101 20100121 abcdefghi 0003 10101 20100122 abcdefghi 0003 10102 20100120 abcdefghi 0003 10103 20100120 abcdefghi 0003 10103 20100121 abcdefghi Here if the first colum and... (6 Replies)
Discussion started by: gpaulose
6 Replies

3. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

4. Shell Programming and Scripting

Removing Duplicates from file

Hi Experts, Please check the following new requirement. I got data like the following in a file. FILE_HEADER 01cbbfde7898410| 3477945| home| 1 01cbc275d2c122| 3478234| WORK| 1 01cbbe4362743da| 3496386| Rich Spare| 1 01cbc275d2c122| 3478234| WORK| 1 This is pipe separated file with... (3 Replies)
Discussion started by: tinufarid
3 Replies

5. Shell Programming and Scripting

Duplicates in an XML file

Hi All, I have an xml file that contains information like this <ID>574922<COMMENT>TEXT TEXT TEXT</COMMENT></ID> <ID>574922<COMMENT>TEXT TEXT TEXT</COMMENT></ID> <ID>412659<COMMENT>TEXT TEXT TEXT TEXT TEXT</COMMENT></ID> <ID>873520<COMMENT>TEXT</COMMENT></ID>... (5 Replies)
Discussion started by: TasosARISFC
5 Replies

6. Shell Programming and Scripting

Remove the partial duplicates by checking the length of a field

Hi Folks - I'm quite new to awk and didn't come across such issues before. The problem statement is that, I've a file with duplicate records in 3rd and 4th fields. The sample is as below: aaaaaa|a12|45|56 abbbbaaa|a12|45|56 bbaabb|b1|51|45 bbbbbabbb|b2|51|45 aaabbbaaaa|a11|45|56 ... (3 Replies)
Discussion started by: asyed
3 Replies

7. Programming

[Solved] Removing duplicates from the file and saving as new file

Dear All I have 200 data files and each files has many duplicates. I am looking for the automated awk script such that it checks and removes the duplicates from the each file and saving them as new files for all 200 files in the respective folder. For example my data looks like this.. ... (12 Replies)
Discussion started by: bala06
12 Replies

8. UNIX for Dummies Questions & Answers

Remove duplicates from a file

Can u tell me how to remove duplicate records from a file? (11 Replies)
Discussion started by: saga20
11 Replies

9. UNIX for Dummies Questions & Answers

Removing duplicates from a file

Hi All, I am merging files coming from 2 different systems ,while doing that I am getting duplicates entries in the merged file I,01,000131,764,2,4.00 I,01,000131,765,2,4.00 I,01,000131,772,2,4.00 I,01,000131,773,2,4.00 I,01,000168,762,2,2.00 I,01,000168,763,2,2.00... (5 Replies)
Discussion started by: Sri3001
5 Replies

10. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3.I have tried previous post also,but in that complete line must be similar.In this case i have to verify first column only regardless what is the content in succeeding columns. (3 Replies)
Discussion started by: sagar_1986
3 Replies
PERIODIC(8)						    BSD System Manager's Manual 					       PERIODIC(8)

NAME
periodic -- run periodic system functions SYNOPSIS
periodic directory ... DESCRIPTION
The periodic utility is intended to be called by launchd(8) to execute shell scripts located in the specified directory. One or more of the following arguments must be specified: daily Perform the standard daily periodic executable run. This usually occurs early in the morning (local time). weekly Perform the standard weekly periodic executable run. This usually occurs very early on Saturday mornings. monthly Perform the standard monthly periodic executable run. This usually occurs on the first day of the month. path An arbitrary directory containing a set of executables to be run. If an argument is an absolute directory name it is used as is, otherwise it is searched for under /etc/periodic and any other directories specified by the local_periodic setting in periodic.conf(5) (see below). The periodic utility will run each executable file in the directory or directories specified. If a file does not have the executable bit set, it is silently ignored. Each script is required to exit with one of the following values: 0 The script has produced nothing notable in its output. The <basedir>_show_success variable controls the masking of this output. 1 The script has produced some notable information in its output. The <basedir>_show_info variable controls the masking of this output. 2 The script has produced some warnings due to invalid configuration settings. The <basedir>_show_badconfig variable controls the mask- ing of this output. >2 The script has produced output that must not be masked. If the relevant variable (where <basedir> is the base directory in which the script resides) is set to ``NO'' in periodic.conf, periodic will mask the script output. If the variable is not set to either ``YES'' or ``NO'', it will be given a default value as described in periodic.conf(5). All remaining script output is delivered based on the value of the <basedir>_output setting. If this is set to a path name (beginning with a '/' character), output is simply logged to that file. newsyslog(8) knows about the files /var/log/daily.log, /var/log/weekly.log and /var/log/monthly.log, and if they exist, it will rotate them at the appropriate times. These are therefore good values if you wish to log periodic output. If the <basedir>_output value does not begin with a '/' and is not empty, it is assumed to contain a list of email addresses, and the output is mailed to them. If <basedir>_show_empty_output is set to ``NO'', then no mail will be sent if the output was empty. If <basedir>_output is not set or is empty, output is sent to standard output. ENVIRONMENT
The periodic utility sets the PATH environment to include all standard system directories, but no additional directories, such as /usr/local/bin. If executables are added which depend upon other path components, each executable must be responsible for configuring its own appropriate environment. FILES
/System/Library/LaunchDaemons/com.apple.periodic-*.plist the periodic utility is typically called via these launchd(8) jobs /etc/periodic the top level directory containing daily, weekly, and monthly subdirectories which contain standard system peri- odic executables /etc/defaults/periodic.conf the periodic.conf system registry contains variables that control the behaviour of periodic and the standard daily, weekly, and monthly scripts /etc/periodic.conf this file contains local overrides for the default periodic configuration EXIT STATUS
Exit status is 0 on success and 1 if the command fails. EXAMPLES
The /etc/defaults/periodic.conf system registry will typically have a local_periodic variable reading: local_periodic="/usr/local/etc/periodic" To log periodic output instead of receiving it as email, add the following lines to /etc/periodic.conf: daily_output=/var/log/daily.log weekly_output=/var/log/weekly.log monthly_output=/var/log/monthly.log To only see important information from daily periodic jobs, add the following lines to /etc/periodic.conf: daily_show_success=NO daily_show_info=NO daily_show_badconfig=NO DIAGNOSTICS
The command may fail for one of the following reasons: usage: periodic <directory of files to execute> No directory path argument was passed to periodic to specify where the script fragments reside. <directory> not found Self explanatory. SEE ALSO
sh(1), periodic.conf(5), launchd(8), newsyslog(8) HISTORY
The periodic utility first appeared in FreeBSD 3.0. AUTHORS
Paul Traina <pst@FreeBSD.org> Brian Somers <brian@Awfulhak.org> BUGS
Since one specifies information about a directory using shell variables containing the string, <basedir>, <basedir> must only contain charac- ters that are valid within a sh(1) variable name, alphanumerics and underscores, and the first character may not be numeric. BSD
August 30, 2007 BSD
All times are GMT -4. The time now is 08:42 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy