Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Removing Duplicate Variables : SED?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Removing Duplicate Variables : SED?
# 8  
Old 10-14-2011
Quote:
Originally Posted by Blue Solo
I opened terminal.app and pasted:

Code:
awk '!a[$NF]++' RS='' /Users/user/Desktop/LevelIndices2.mtl 
newmtl m2 
Kd 1.000000 1.000000 1.000000; 
Ka 0 0 0 illum 2 
Ns 64 
d 1.000000 
map_Kd 951E01Other4040.b.        mp

Just copy/paste the awk line. The rest was provided as the illustration of the output given your sample input.
This User Gave Thanks to vgersh99 For This Post:
Blue Solo
# 9  
Old 10-15-2011
I installed nawk and ran this in terminal:
Code:
nawk '!a[$NF]++' RS='' /Users/user/Desktop/LevelIndices2.mtl

It returned every line of text that was in the file LevelIndices2.mtl. Am I supposed to fill in NF or RS='' with something?

By the way thanks for trying to help so far! Smilie

UPDATE:
Now I see what is going on. I ran this:
Code:
nawk '!a[$NF]++' /Users/user/Desktop/LevelIndices2.mtl

And I turned up these results:
Code:
newmtl m2
map_Kd 951E01Other4040.bmp
newmtl m3
newmtl m4
newmtl m5
newmtl m6
newmtl m7
newmtl m8
newmtl m9
map_Kd 952C57w20h200xC6COLORS.bmp
newmtl m10
map_Kd A1C039w20h200x1CCOLORS.bmp
newmtl m11
newmtl m12
newmtl m13
newmtl m14
newmtl m15
map_Kd 946418w20h200x87COLORS.bmp
newmtl m16
newmtl m17

and so on throughout the whole document. So the first newmtl mXXX before a .bmp is the one that needs to be kept (m2, m9, m10,m15...).

-Next step from here is to keep the newmtl mXXX, and the rest are the ones I need to delete.
-Then in another file all mXXX I deleted need to be found and replaced by the one that are kept in this file.

UPDATE 2:
There are a few imperfections; When a .bmp is separating another it doesn't index it.
Example:
Code:
newmtl m2
Kd 1.000000 1.000000 1.000000
Ka 0 0 0
illum 2
Ns 64
d 1.000000
map_Kd image.bmp

newmtl m3
Kd 1.000000 1.000000 1.000000
Ka 0 0 0
illum 2
Ns 64
d 1.000000
map_Kd anotherImage.bmp

newmtl m4
Kd 1.000000 1.000000 1.000000
Ka 0 0 0
illum 2
Ns 64
d 1.000000
map_Kd image.bmp

When the command is run it will show:
newmtl m2
map_Kd image.bmp
newmtl m3
map_Kd anotherImage.bmp
newmtl m4

Instead it should show:
newmtl m2
map_Kd image.bmp
newmtl m4
newmtl m3
map_Kd anotherImage.bmp

This is because there is a separate .bmp in between them.
Last edited by Blue Solo; 10-15-2011 at 06:41 PM.. Reason: Update
# 10  
Old 10-15-2011
Is there a way to order all the newmtl mXXX based on map_Kd XXX.bmp so that they are all grouped together and right next to each other? This should fix the error with:
nawk '!a[$NF]++' /Users/user/Desktop/LevelIndices2.mtl

If it is any help I am attaching the files as .txt; Step1.txt is what should be done first, Step2.txt is what needs to be edited second.
step1.txt (45.3 KB) 
Step2.txt (422.8 KB) 
# 11  
Old 10-15-2011
According to your original post, this may be what you wanted:
Code:
awk '!RS {
  if ($1 != "newmtl") next
  if ($NF in p) {
    d["\\<" $2 "\\>"] = p[$NF] #gawk
    #d[$2] = p[$NF] # too loose
  } else {
    p[$NF] = $2
    print $0 "\n" > "step1.new"
  }
  next
}
{
  for (i in d) gsub(i, d[i])
  print
}' RS='' step1 RS='\n' step2 > step2.new

By the way, the files you attached were DOS files. You may need to convert them into Unix files.
# 12  
Old 10-15-2011
Quote:
Originally Posted by binlib
According to your original post, this may be what you wanted:
Code:
Code

By the way, the files you attached were DOS files. You may need to convert them into Unix files.
Oh wow, thanks! The tip about the DOS to Unix was great.
So that deleted all the duplicates, but it did not rename the duplicates in the step2 to their kept duplicate.

Thanks so far!
Last edited by Blue Solo; 10-16-2011 at 12:43 AM..
# 13  
Old 10-16-2011
I guess you are not using gawk. The problem with the replacement is that you can't just replace m2 with m1, for example, because it will replace m20 with m10. Since your step2 file always has the names as the last field, you can change
Code:
d["\\<" $2 "\\>"] = p[$NF] #gawk
to
d[" " $2 "$" ] = " " p[$NF]

for other awks that don't have \< and \> for beginging/end of word.
This User Gave Thanks to binlib For This Post:
Blue Solo
# 14  
Old 10-16-2011
Quote:
Originally Posted by binlib
I guess you are not using gawk. The problem with the replacement is that you can't just replace m2 with m1, for example, because it will replace m20 with m10. Since your step2 file always has the names as the last field, you can change
Code:
d["\\<" $2 "\\>"] = p[$NF] #gawk
to
d[" " $2 "$" ] = " " p[$NF]

for other awks that don't have \< and \> for beginging/end of word.
Thank you so much! It finally works! Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing duplicate items from an array

Hello Experts, I am trying to write a shell script to find duplicate items in an array, this is what i have tried : #!/bin/bash echo "This is another sample Progg to remove duplicate items from an array" echo "How many number do you want to insert" read n for (( i=0; i<$n; i++ )) do ... (5 Replies)
Discussion started by: mukulverma2408
5 Replies

2. Shell Programming and Scripting

Removing Duplicate Rows in a file

Hello I have a file with contents like this... Part1 Field2 Field3 Field4 (line1) Part2 Field2 Field3 Field4 (line2) Part3 Field2 Field3 Field4 (line3) Part1 Field2 Field3 Field4 (line4) Part4 Field2 Field3 Field4 (line5) Part5 Field2 Field3 Field4 (line6) Part2 Field2 Field3 Field4... (7 Replies)
Discussion started by: ekbaazigar
7 Replies

3. Shell Programming and Scripting

Removing duplicate terms in a file

Hi everybody I have a .txt file that contains some assembly code for optimizing it i need to remove some replicated parts. for example I have:e_li r0,-1 e_li r25,-1 e_lis r25,0000 add r31, r31 ,r0 e_li r28,-1 e_lis r28,0000 add r31, r31 ,r0 e_li r28,-1 ... (3 Replies)
Discussion started by: Behrouzx77
3 Replies

4. Post Here to Contact Site Administrators and Moderators

Removing or Merging some duplicate threads

I have made some threads that were identical and were about the same question :( I've made them in 3 forums , the moderator has moved and merged one of these threads. There is one thread left and it need to be merged or deleted. Is there any way I can delete it or merge it myself ? I have delete... (1 Reply)
Discussion started by: k.a.docpp
1 Replies

5. Shell Programming and Scripting

Removing duplicate records from 2 files

Can anyone help me to removing duplicate records from 2 separate files in UNIX? Please find the sample records for both the files cat Monday.dat 3FAHP0JA1AR319226MOHMED ATEK 966504453742 SAU2010DE 3LNHL2GC6AR636361HEA DEUK CHOI 821057314531 KOR2010LE 3MEHM0JG7AR652083MUTLAB NAL-NAFISAH... (4 Replies)
Discussion started by: zooby
4 Replies

6. Shell Programming and Scripting

Removing Duplicate Lines per Section

Hello, I am in need of removing duplicate lines from within a file per section. File: ABC1 012345 header ABC2 7890-000 ABC3 012345 Header Table ABC4 ABC5 593.0000 587.4800 ABC5 593.5000 587.6580 <= dup need to remove ABC5 593.5000 ... (5 Replies)
Discussion started by: petersf
5 Replies

7. Shell Programming and Scripting

removing the duplicate lines in a file

Hi, I need to concatenate three files in to one destination file.In this if some duplicate data occurs it should be deleted. eg: file1: ----- data1 value1 data2 value2 data3 value3 file2: ----- data1 value1 data4 value4 data5 value5 file3: ----- data1 value1 data4 value4 (3 Replies)
Discussion started by: Sharmila_P
3 Replies

8. Shell Programming and Scripting

removing duplicate blank lines

Hi, how to remove the blank lines from the file only If we have more than one blank line. thanks rameez (8 Replies)
Discussion started by: rameezrajas
8 Replies

9. UNIX for Dummies Questions & Answers

removing duplicate lines from a file

Hi, I am trying to remove duplicate lines from a file. For example the contents of example.txt is: this is a test 2342 this is a test 34343 this is a test 43434 and i want to remove the "this is a test" lines only and end up with the numbers in the file, that is, end up with: 2342... (4 Replies)
Discussion started by: ocelot
4 Replies

10. UNIX for Dummies Questions & Answers

Removing duplicate lines ignore case

hi, I have the following input in file: abc ab a AB b c a C B When I use uniq -u file,the out put file is: abc ab AB c v B C (17 Replies)
Discussion started by: hellsd
17 Replies
Login or Register to Ask a Question