Remove duplicate ???


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove duplicate ???
# 1  
Old 03-09-2006
Remove duplicate ???

Hi all,
I have a out.log file


CARR|02/26/2006 10:58:30.107|CDxAcct=1405157051
CARR|02/26/2006 11:11:30.107|CDxAcct=1405157051
CARR|02/26/2006 11:18:30.107|CDxAcct=7659579782
CARR|02/26/2006 11:28:30.107|CDxAcct=9534922327
CARR|02/26/2006 11:38:30.107|CDxAcct=9534922327
CARR|02/26/2006 11:49:30.107|CDxAcct=1405157067
LMWC|02/26/2006 11:59:28.307|CDxAcct=7659579780
LMWC|02/26/2006 12:01:28.307|Case Modify 10021 20024
LMWC|02/26/2006 12:11:28.307|Case Access 10021 20025
LMWC|02/26/2006 12:14:28.307|Case Modify 10022 20026
LMWC|02/26/2006 12:29:28.307|Case Modify 10023 20027

How do I write a script to get the result like this ??

CARR,02/26/2006 10:58:30.107,CDxAcct=1405157051|CDxAcct=7659579782|CDxAcct=9534922327|CDxAcct=1405157067
LMWC,02/26/2006 11:59:28.307,CDxAcct=7659579780|Case Modify 10021|Case Modify 10022|Case Modify 10023

Here is my bad code Smilie

#!/bin/ksh
cd /DAT/
cat out.log |

awk -F"|" '
function writeit()
{ printf "%s,%s\,%s\n\n",outputId,datetime,outputAction;
outputAction = "";
return
} {
id=$1;
datetime=$2;
action = $3;
lastacct = "";
if (outputId == ""){
outputId = id;

}

if (id != outputId ) {
printf "!!!! A: %s %s\n",outputId;
writeit();
outputId = id;

}
if (action ~ /CDxAcct/) {
acct = substr(action,9,10);

if (acct != lastacct) {
outputAction = outputAction "CDxAcct=" acct "|";
lastacct = acct;
} ## Remove duplicate acct like CDxAcct=1405157051|CDxAcct=1405157051|CDxAcct=7659579782

} else if (action ~ /Modify/) {

outputAction = action "|";
type = substr(action,1,index(action,"Modify")-2); #get Case
mainkey = substr(action,index(action,"Modify")+7,index(substr(action,index(action,"Modify")+7,20)," ")-1); #get number after Modify
searchspec1 = type " " mainkey;
searchspec2 = type " Modify " mainkey;
searchspec3 = type " Access " mainkey;

if ((outputAction ~ searchspec1) || (outputAction ~ searchspec2)) {

} else if ((outputAction ~ searchspec3)) {
# NEEDS FURTHER WORK To remove the Case Access XX from Case Modify XX
outputAction = outputAction action "|";

} else {
outputAction = outputAction action "|";
}

}
} END {
writeit();

}'



Thanks for your help.

Last edited by sabercats; 03-10-2006 at 05:15 PM..
# 2  
Old 03-10-2006
Not sure that I understand what you want. But maybe this does it...
Code:
#! /usr/bin/ksh
IFS="|"
first=1
exec < out.log
while read  name field2 field3; do
        if ((first)) ; then
                first=0
                oldname=$name
                output="${name}${field2}|${field3}"
        elif [[ $name != $oldname ]] ; then
                echo "$output"
                output="${name}${field2}|${field3}"
                oldname=$name
        else
                output="${output}|${field3}"
        fi

done
echo "$output"
exit 0

# 3  
Old 03-10-2006
Ruby:
Code:
gets(nil).scan( /^(.*?)\|(.*?)\|(.*\n(?:\1.*\n)*)/ ){|b|
  b.last.gsub!( /^.*(?=\|)/,"" ).gsub!( /\n/,"" )
  puts b.join(",")
}

# 4  
Old 03-10-2006
Oh cool, I learn another way, I will post you my code when I am done Smilie
Thanks,
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR (10 Replies)
Discussion started by: tara123
10 Replies

3. UNIX for Dummies Questions & Answers

Remove duplicate

Hi, How can I replace || with space and then remove duplicate from following text? T111||T222||T444||T222||T555 Thanks in advance (10 Replies)
Discussion started by: tinku981
10 Replies

4. Shell Programming and Scripting

Remove duplicate

Hi , I have a pipe seperated file repo.psv where i need to remove duplicates based on the 1st column only. Can anyone help with a Unix script ? Input: 15277105||Common Stick|ESHR||Common Stock|CYRO AB 15277105||Common Stick|ESHR||Common Stock|CYRO AB 16111278||Common Stick|ESHR||Common... (12 Replies)
Discussion started by: samrat dutta
12 Replies

5. Shell Programming and Scripting

How to remove duplicate ID's?

HI I have file contains 1000'f of duplicate id's with (upper and lower first character) as below i/p: a411532A411532a508661A508661c411532C411532 Requirement: But i need to ignore lowercase id's and need only below id's o/p: A411532 A508661 C411532 (9 Replies)
Discussion started by: buzzme
9 Replies

6. Shell Programming and Scripting

Help with remove duplicate content

Input file data_1 10 US data_1 2 US data_1 5 UK data_2 20 ENGLAND data_2 12 KOREA data_3 4 CHINA . . data_60 123 US data_60 23 UK data_60 45 US Desired output file data_1 10 US data_1 5 UK data_2 20 ENGLAND data_2 12 KOREA (2 Replies)
Discussion started by: perl_beginner
2 Replies

7. Shell Programming and Scripting

remove duplicate

Hi, I am tryung to use shell or perl to remove duplicate characters for example , if I have " I love google" it will become I love ggle" or even "I loveggle" if removing duplicate white space Thanks CC (6 Replies)
Discussion started by: ccp
6 Replies

8. UNIX for Dummies Questions & Answers

Remove duplicate in array

Hi, I have a list of numbers stored in an array as below. 5 7 10 30 30 40 50 Please advise how could I remove the duplicate value in the array ? Thanks in advance. (5 Replies)
Discussion started by: Rock
5 Replies

9. Shell Programming and Scripting

Remove duplicate

Hi all, I have a text file fileA.txt DXRV|02/28/2006 11:36:49.049|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:37:06.404|SAC||||CDxAcct=6070970034 DXRV|02/28/2006 11:37:25.740|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:38:32.633|SAC||||CDxAcct=6070970034 DXRV|02/28/2006... (2 Replies)
Discussion started by: sabercats
2 Replies

10. Shell Programming and Scripting

remove duplicate

i have a text its contain many record, but its written in one line, i want to remove from that line the duplicate record, not record have fixed width ex: width = 4 inputfile test.txt =abc cdf abc abc cdf fgh fgh abc abc i want the outputfile =abc cdf fgh only those records can any one help... (4 Replies)
Discussion started by: kazanoova2
4 Replies
Login or Register to Ask a Question