Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Remove duplicate records

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove duplicate records
# 8  
Old 03-03-2010
Code:
>cut -d";" -f1,2,4 <scottn.txt | sort | uniq -u | gawk 'IFS=OFS=FS=";"{print $1,$2,".*",$3}' >match.txt
>grep -f match.txt <scottn.txt

sort -t\; -k1,1 -k2,2 -k4,4 scottn.txt | sed 's/[^;]*;/*;/3' | uniq -u > match.txt
grep -f match.txt scottn.txt

it is failing for the fields having $ ( ) characters.
Last edited by Scott; 03-03-2010 at 12:52 PM.. Reason: Code tags, please...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Duplicate records

Gents, Please give a help file --BAD STATUS NOT RESHOOTED-- *** VP 41255/51341 in sw 2973 *** VP 41679/51521 in sw 2973 *** VP 41687/51653 in sw 2973 *** VP 41719/51629 in sw 2976 --BAD COG NOT RESHOOTED-- *** VP 41689/51497 in sw 2974 *** VP 41699/51677 in sw 2974 *** VP... (18 Replies)
Discussion started by: jiam912
18 Replies

2. Shell Programming and Scripting

Remove duplicate records

Hi, i am working on a script that would remove records or lines in a flat file. The only difference in the file is the "NOT NULL" word. Please see below example of the input file. INPUT FILE:> CREATE a ( TRIAL_CLIENT NOT NULL VARCHAR2(60), TRIAL_FUND NOT NULL... (3 Replies)
Discussion started by: reignangel2003
3 Replies

3. Shell Programming and Scripting

Deleting duplicate records from file 1 if records from file 2 match

I have 2 files "File 1" is delimited by ";" and "File 2" is delimited by "|". File 1 below (3 record shown): Doc1;03/01/2012;New York;6 Main Street;Mr. Smith 1;Mr. Jones Doc2;03/01/2012;Syracuse;876 Broadway;John Davis;Barbara Lull Doc3;03/01/2012;Buffalo;779 Old Windy Road;Charles... (2 Replies)
Discussion started by: vestport
2 Replies

4. UNIX for Dummies Questions & Answers

Need to keep duplicate records

Consider my input is 10 10 20 then, uniq -u will give 20 and uniq -dwill return 10. But i need the output as , 10 10 How we can achieve this? Thanks (4 Replies)
Discussion started by: pandeesh
4 Replies

5. Shell Programming and Scripting

Remove somewhat Duplicate records from a flat file

I have a flat file that contains records similar to the following two lines; 1984/11/08 7 700000 123456789 2 1984/11/08 1941/05/19 7 700000 123456789 2 The 123456789 2 represents an account number, this is how I identify the duplicate record. The ### signs represent... (4 Replies)
Discussion started by: jolney
4 Replies

6. Shell Programming and Scripting

Remove Duplicate Records

Hi frinds, Need your help. item , color ,desc ==== ======= ==== 1,red ,abc 1,red , a b c 2,blue,x 3,black,y 4,brown,xv 4,brown,x v 4,brown, x v I have to elemnet the duplicate rows on the basis of item. the final out put will be 1,red ,abc (6 Replies)
Discussion started by: imipsita.rath
6 Replies

7. Shell Programming and Scripting

How to remove duplicate records with out sort

Can any one give me command How to delete duplicate records with out sort. Suppose if the records like below: 345,bcd,789 123,abc,456 234,abc,456 712,bcd,789 out tput should be 345,bcd,789 123,abc,456 Key for the records is 2nd and 3rd fields.fields are seperated by colon(,). (19 Replies)
Discussion started by: svenkatareddy
19 Replies

8. Solaris

How to remove duplicate records with out sort

Can any one give me command How to delete duplicate records with out sort. Suppose if the records like below: 345,bcd,789 123,abc,456 234,abc,456 712,bcd,789 out tput should be 345,bcd,789 123,abc,456 Key for the records is 2nd and 3rd fields.fields are seperated by colon(,). (2 Replies)
Discussion started by: svenkatareddy
2 Replies

9. Shell Programming and Scripting

Remove all instances of duplicate records from the file

Hi experts, I am new to scripting. I have a requirement as below. File1: A|123|NAME1 A|123|NAME2 B|123|NAME3 File2: C|123|NAME4 C|123|NAME5 D|123|NAME6 1) I have 2 merge both the files. 2) need to do a sort ( key fields are first and second field) 3) remove all the instances... (3 Replies)
Discussion started by: vukkusila
3 Replies

10. Shell Programming and Scripting

Records Duplicate

Hi Everyone, I have a flat file of 1000 unique records like following : For eg Andy,Flower,201-987-0000,12/23/01 Andrew,Smith,101-387-3400,11/12/01 Ani,Ross,401-757-8640,10/4/01 Rich,Finny,245-308-0000,2/27/06 Craig,Ford,842-094-8740,1/3/04 . . . . . . Now I want to duplicate... (9 Replies)
Discussion started by: ganesh123
9 Replies
Login or Register to Ask a Question

LEARN ABOUT HPUX

runacct

runacct(1M)															       runacct(1M)

NAME

       runacct - run daily accounting

SYNOPSIS

       [mmdd[state]]

DESCRIPTION

       runacct	is  the  main daily accounting shell procedure.  It is normally initiated via cron(1M).  runacct processes connect, fee, disk, and
       process accounting files.  It also prepares summary files for prdaily or billing purposes.

       runacct takes care not to damage active accounting files or summary files in the event of errors.   It  records	its  progress  by  writing
       descriptive  diagnostic messages into When an error is detected, a message is written to mail (see mail(1), mailx(1), or elm(1)) is sent to
       and and runacct terminates.  runacct uses a series of lock files to protect against re-invocation.  The	files  and  are  used  to  prevent
       simultaneous invocation, and is used to prevent more than one invocation per day.

       runacct	breaks its processing into separate, restartable states using to remember the last state completed.  It accomplishes this by writ-
       ing the state name into runacct then looks in to see what it has done and to determine what to process next.  states are  executed  in  the
       following order:

	      Move active accounting files into working files.

	      Verify integrity of
			     file, correcting date changes if necessary.

	      Produce connect session records in
			     format.

	      Convert process accounting records into
			     format.

	      Merge the connect and process accounting records.

	      Convert output of
			     chargefee into format and merge with connect and process accounting records.

	      Merge disk accounting records with connect, process, and fee accounting
			     records.

	      Merge the daily total accounting records in
			     with the summary total accounting records in

	      Produce command summaries.

	      Any installation-dependent accounting programs can be
			     included here.

	      Cleanup temporary files and exit.

       To  restart  runacct  after  a failure, first check the file for diagnostics, then fix up any corrupted data files such as or The files and
       file must be removed before runacct can be restarted.  The argument mmdd is necessary if runacct is  being  restarted,  and  specifies  the
       month  and  day	for  which  runacct  will  rerun the accounting.  Entry point for processing is based on the contents of to override this,
       include the desired state on the command line to designate where processing should begin.

EXAMPLES

       To start runacct.

       To restart runacct.

       To restart runacct at a specific state.

WARNINGS

       Normally it is not a good idea to restart runacct in its state.	Run manually, then restart via:

       If runacct failed in its state, remove the last file because it will not be complete.

FILES

SEE ALSO

       mail(1), acct(1M), acctcms(1M), acctcom(1M), acctcon(1M), acctmerg(1M), acctprc(1M), acctsh(1M),  cron(1M),  fwtmp(1M),	acct(2),  acct(4),
       utmp(4).

STANDARDS CONFORMANCE

																       runacct(1M)