Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Delete duplicates in CA bundle

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Delete duplicates in CA bundle
# 1  
Old 06-28-2014
Error Delete duplicates in CA bundle
I do have a big CA bundle certificate file and each time if i get request to add new certificate to the existing bundle i need to make sure it is not present already. How i can validate the duplicates.

The alignment of the certificate within the bundle seems to be different.

Example:
Cert 1
Code:
MIIF7DCCBNSgAwIBAgIQbsx6pacDIAm4zrz06VLUkTANBgkqhkiG9w0BAQUFADCByjELMAkGA1UE
BhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBO

Cert 2
Code:
MIIF7DCCBNSgAwIBAgIQbsx6pacDIAm4zrz06VLUkTANBgkqhkiG9w0BAQUFADCB
yjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL


These are duplicates and since they are aligned differently i could not do an extract and compare.

Your suggestions and inputs are most welcome.
# 2  
Old 06-28-2014
Quote:
Originally Posted by diva_thilak
I do have a big CA bundle certificate file and each time if i get request to add new certificate to the existing bundle i need to make sure it is not present already. How i can validate the duplicates.

The alignment of the certificate within the bundle seems to be different.

Example:
Cert 1
Code:
MIIF7DCCBNSgAwIBAgIQbsx6pacDIAm4zrz06VLUkTANBgkqhkiG9w0BAQUFADCByjELMAkGA1UE
BhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBO

Cert 2
Code:
MIIF7DCCBNSgAwIBAgIQbsx6pacDIAm4zrz06VLUkTANBgkqhkiG9w0BAQUFADCB
yjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL


These are duplicates and since they are aligned differently i could not do an extract and compare.

Your suggestions and inputs are most welcome.
If we ignore the line breaks, we can see that these two certs start with the same strings. But, the 1st cert ends with the characters:
Code:
ExZWZXJpU2lnbiBUcnVzdCBO

that do not appear in the 2nd cert. I don't see how you can say these certs are the same when one of them contains 24 more characters than the other one.
# 3  
Old 06-28-2014
Thank you Don for your quick reply.

I have tried to post the first two lines to highlight the alignment issue. Yes you are right, removing the line breaks the certificates would be same.

Any suggestions on getting the duplicates removed from the file and direct them to a different file to show the number of duplicates.
# 4  
Old 06-28-2014
With what you have given us, I have absolutely no idea what you are trying to do.

What is a bundle?

Are both of these certs (and a bunch of others stored in a single file? If so, how are the certs delimited?

Do you have a bunch of files that each contain a cert?

Give us something concrete to work with! Show us sample input files to be processed. Explain the format of those input files. Show us the output that should be produced for those input files. Explain the format of the outputs to be produced.
# 5  
Old 07-15-2014
In informatica server for authentication purposes the list of certificates are stored in a file. It is called as Certificate bundle.
Code:
-----BEGIN CERTIFICATE-----
MIIDAzCCAmygAwIBAgIBATANBgkqhkiG9w0BAQQFADCBuzELMAkGA1UEBhMCQkUx
ETAPBgNVBAcTCEJydXNzZWxzMRMwEQYDVQQKEwpCZWxTaWduIE5WMTgwNgYDVQQL
Ey9CZWxTaWduIE9iamVjdCBQdWJsaXNoaW5nIENlcnRpZmljYXRlIEF1dGhvcml0
eTElMCMGA1UEAxMcQmVsU2lnbiBPYmplY3QgUHVibGlzaGluZyBDQTEjMCEGCSqG
SIb3DQEJARYUd2VibWFzdGVyQGJlbHNpZ24uYmUwHhcNOTcwOTE5MjIwMzAwWhcN
MDcwOTE5MjIwMzAwWjCBuzELMAkGA1UEBhMCQkUxETAPBgNVBAcTCEJydXNzZWxz
MRMwEQYDVQQKEwpCZWxTaWduIE5WMTgwNgYDVQQLEy9CZWxTaWduIE9iamVjdCBQ
dWJsaXNoaW5nIENlcnRpZmljYXRlIEF1dGhvcml0eTElMCMGA1UEAxMcQmVsU2ln
biBPYmplY3QgUHVibGlzaGluZyBDQTEjMCEGCSqGSIb3DQEJARYUd2VibWFzdGVy
QGJlbHNpZ24uYmUwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMQuH7a/7oJA
3fm3LkHVngWxWtAmfGJVA5v8y2HeS+/+6Jn+h7mIz5DaDwk8dt8Xl7bLPyVF/bS8
WAC+sFq2FIeP7mdkrR2Ig7tnn2VhAFgIgFCfgMkx9iqQHC33SmwQ9iNDXTgJYIhX
As0WbBj8zfuSKnfQnpOjXYhk0Mj4XVRRAgMBAAGjFTATMBEGCWCGSAGG+EIBAQQE
AwIABzANBgkqhkiG9w0BAQQFAAOBgQBjdhd8lvBTpV0BHFPOKcJ+daxMDaIIc7Rq
Mf0CBhSZ3FQEpL/IloafMUMyJVf2hfYluze+oXkjyVcGJXFrRU/49AJAFoIir1Tq
Mij2De6ZuksIUQ9uhiMhTC0liIHELg7xEyw4ipUCJMM6lWPkk45IuwhHcl+u5jpa
R9Zxxp6aUg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDKTCCApKgAwIBAgIENnAVljANBgkqhkiG9w0BAQUFADBGMQswCQYDVQQGEwJV
UzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMREwDwYDVQQL
EwhEU1RDQSBFMTAeFw05ODEyMTAxODEwMjNaFw0xODEyMTAxODQwMjNaMEYxCzAJ
BgNVBAYTAlVTMSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4x
ETAPBgNVBAsTCERTVENBIEUxMIGdMA0GCSqGSIb3DQEBAQUAA4GLADCBhwKBgQCg
bIGpzzQeJN3+hijM3oMv+V7UQtLodGBmE5gGHKlREmlvMVW5SXIACH7TpWJENySZ
j9mDSI+ZbZUTu0M7LklOiDfBu1h//uG9+LthzfNHwJmm8fOR6Hh8AMthyUQncWlV
Sn5JTe2io74CTADKAqjuAQIxZA9SLRN0dja1erQtcQIBA6OCASQwggEgMBEGCWCG
SAGG+EIBAQQEAwIABzBoBgNVHR8EYTBfMF2gW6BZpFcwVTELMAkGA1UEBhMCVVMx
JDAiBgNVBAoTG0RpZ2l0YWwgU2lnbmF0dXJlIFRydXN0IENvLjERMA8GA1UECxMI
RFNUQ0EgRTExDTALBgNVBAMTBENSTDEwKwYDVR0QBCQwIoAPMTk5ODEyMTAxODEw
MjNagQ8yMDE4MTIxMDE4MTAyM1owCwYDVR0PBAQDAgEGMB8GA1UdIwQYMBaAFGp5
fpFpRhgTCgJ3pVlbYJglDqL4MB0GA1UdDgQWBBRqeX6RaUYYEwoCd6VZW2CYJQ6i
+DAMBgNVHRMEBTADAQH/MBkGCSqGSIb2fQdBAAQMMAobBFY0LjADAgSQMA0GCSqG
SIb3DQEBBQUAA4GBACIS2Hod3IEGtgllsofIH160L+nEHvI8wbsEkBFKg05+k7lN
QseSJqBcNJo4cvj9axY+IO6CizEqkzaFI4iKPANo08kJD038bKTaKHKTDomAsH3+
gG9lbRgzl4vCa4nuYD3Im+9/KzJic5PLPON74nZ4RbyhkwS7hp86W0N6w4pl
-----END CERTIFICATE-----

They are stored in this format. My goal is to identify if there are any duplicates within this file (bundle). Since the alignment of the text within the begin and end tag are different, I was looking for an approach for identifying the dupes.
1. how to get the text between the begin and end tags extract
2. How to identify the dupes by bringing the text from step 1 in uniform format
Last edited by Don Cragun; 07-15-2014 at 02:01 PM.. Reason: Add CODE tags.
# 6  
Old 07-15-2014
What's your OS? (or more specifically, do you have GNU awk?)
# 7  
Old 07-15-2014
Try
Code:
sed -n '/CERTIFICATE/ {s/.*//g;x;s/\n//g;p;d}; H ' file | sort | uniq -c

and tell us the results.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

To Delete the duplicates using Part of File Name

I am using the below script to delete duplicate files but it is not working for directories with more than 10k files "Argument is too long" is getting for ls -t. Tried to replace ls -t with find . -type f \( -iname "*.xml" \) -printf '%T@ %p\n' | sort -rg | sed -r 's/* //' | awk... (8 Replies)
Discussion started by: gold2k8
8 Replies

2. Shell Programming and Scripting

Delete only if duplicates found in each record

Hi, i have another problem. I have been trying to solve it by myself but failed. inputfile ;; ID T08578 NAME T08578 SBASE 30696 EBASE 32083 TYPE P func just test func chronology func cholesterol func null INT 30765-37333 INT 37154-37318 Link 5546 Link 8142 (4 Replies)
Discussion started by: redse171
4 Replies

3. Shell Programming and Scripting

Search for duplicates and delete but remain the first one based on a specific pattern

Hi all, I have been trying to delete duplicates based on a certain pattern but failed to make it works. There are more than 1 pattern which are duplicated but i just want to remove 1 pattern only and remain the rest. I cannot use awk '!x++' inputfile.txt or sed '/pattern/d' or use uniq and sort... (7 Replies)
Discussion started by: redse171
7 Replies

4. Shell Programming and Scripting

delete from line and remove duplicates

My Input.....file1 ABCDE4435 Connected to 107.71.136.122 (SubNetwork=ONRM_RootMo_R SubNetwork=XYVLTN29CRBR99 MeContext=ABCDE4435 ManagedElement=1) ABCDE4478 Connected to 166.208.30.57 (SubNetwork=ONRM_RootMo_R SubNetwork=KLFMTN29CR0R04 MeContext=ABCDE4478 ManagedElement=1) ABCDE4478... (5 Replies)
Discussion started by: pareshkp
5 Replies

5. Shell Programming and Scripting

Fastest way to delete duplicates from a large filelist.....

OK I have two filelists...... The first is formatted like this.... /path/to/the/actual/file/location/filename.jpg and has up to a million records The second list shows filename.jpg where there is more then on instance. and has maybe up to 65,000 records I want to copy files... (4 Replies)
Discussion started by: Bashingaway
4 Replies

6. Shell Programming and Scripting

Delete Duplicates on the basis of two column values.

Hi All, i need ti delete two duplicate processss which are running on the same device type (column 1) and port ID (column 2). here is the sample data p1sc1m1 15517 11325 0 01:00:24 ? 0:00 scagntclsx25octtcp 2967 in3v mvmp01 0 8000 N S 969 750@751@752@ p1sc1m1 15519 11325 0 01:00:24 ? ... (5 Replies)
Discussion started by: neeraj617
5 Replies

7. Shell Programming and Scripting

Delete duplicates via script?

Hello, i have the following problem: there are two folders with a lot of files. Example: FolderA contains AAA, BBB, CCC FolderB contains DDD, EEE, AAA How can i via script identify AAA as duplicate in Folder B and delete it there? So that only DDD and EEE remain, in Folder B? Thank you... (16 Replies)
Discussion started by: Y-T
16 Replies

8. Shell Programming and Scripting

how can I delete duplicates in the log?

I have a log file and I am trying to run a script against it to search for key issues such as invalid users, errors etc. In one part, I grep for session closed and get a lot of the same thing,, ie. root username etc. I want to remove the multiple root and just have it do a count, like wc -l ... (5 Replies)
Discussion started by: taekwondo
5 Replies

9. Shell Programming and Scripting

How can i delete the duplicates based on one column of a line

I have my data something like this (08/03/2009 22:57:42.414)(:) king aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb (08/03/2009 22:57:42.416)(:) John cccccccccccc cccccvssssssssss baaaaa (08/03/2009 22:57:42.417)(:) Michael ddddddd tststststtststts (08/03/2009 22:57:42.425)(:) Ravi... (11 Replies)
Discussion started by: rdhanek
11 Replies

10. Shell Programming and Scripting

An interactive way to delete duplicates

1)I am trying to write a script that works interactively lists duplicated records on certain field/column and asks user to delete one or more. And finally it deletes all the records the used has asked for. I have an idea to store those line numbers in an array, not sure how to do this in... (3 Replies)
Discussion started by: chvs2000
3 Replies
Login or Register to Ask a Question