Removing non-unique prefixed lines from a list


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Removing non-unique prefixed lines from a list
# 1  
Old 02-04-2014
Removing non-unique prefixed lines from a list

Not quite sure how to explain what I need to do (or how to title the post!) so will try and show it!

Basically I have a list of 'modules' which takes the form seen below, where there can be a module name, module type and module version (a module may not have each of those and could in theory have more (subversion etc). Also version can be name or number). The list contains an entry for each name; name and type; and name, type version, so for example:

Code:
apache-ant
apache-ant/1.8.4
atlas
atlas/gcc
atlas/gcc/3.8.4
atlas/gcc/3.8.6
binutils
binutils/2.22
binutils/2.23
codeutils/
codeutils/testing
codeutils/working
codeutils/live/1.2

I'm only interested in the final unique entry for each module, so what I want to do is remove the non-unique branches of list, i.e. going from the above list to the list below:

Code:
apache-ant/1.8.4
atlas/gcc/3.8.4
atlas/gcc/3.8.6
binutils/2.22
binutils/2.23
codeutils/testing
codeutils/working
codeutils/live/1.2

I'm not even sure where to start with this, after disregarding rev, uniq etc, though no doubt I'm missing some simple solution!

Any help as always appreciated
# 2  
Old 02-04-2014
Try:
Code:
awk '{print length" "$0}' file | sort -nrk1 | awk '{f=0;for (i in a) if (index (i,$2)!=0) f=1;if (!f) a[$2]=1}END{for (i in a) print i}' | sort

# 3  
Old 02-04-2014
Brilliant, thank you!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Joining broken lines and removing empty lines

Hi - I have req to join broken lines and remove empty lines but should NOT be in one line. It has to be as is line by line. The challenge here is there is no end of line/start of line char. thanks in advance Source:- 2003-04-34024|04-10-2003|Claims|Claim|01-13-2003|Air Bag:Driver;... (7 Replies)
Discussion started by: Jackceasar123
7 Replies

2. UNIX for Dummies Questions & Answers

Removing rows that contain non-unique column entry

Background: I have a file of thousands of potential SSR primers from Batch Primer 3. I can't use primers that will contain the same sequence ID or sequence as another primer. I have some basic shell scripting skills, but not enough to handle this. What you need to know: I need to remove the... (1 Reply)
Discussion started by: msatseqs
1 Replies

3. UNIX for Dummies Questions & Answers

Removing PATTERN from txt without removing lines and general text formatting

Hi Everybody! First post! Totally noobie. I'm using the terminal to read a poorly formatted book. The text file contains, in the middle of paragraphs, hyphenation to split words that are supposed to be on multiple pages. It looks ve -- ry much like this. I was hoping to use grep -v " -- "... (5 Replies)
Discussion started by: AxeHandle
5 Replies

4. Emergency UNIX and Linux Support

Removing ONLY UNIQUE blank spaces with sed?

Hi, I have data that looks similar to this: In which the sentences are written horizontally and the beginning of a sentence is indicated by a 1 in the first column and the number increments until the last item of the sentence. The end of the sentence and the beginning of the next is then indicate... (1 Reply)
Discussion started by: owwow14
1 Replies

5. UNIX for Dummies Questions & Answers

Print unique lines without sort or unique

I would like to print unique lines without sort or unique. Unfortunately the server I am working on does not have sort or unique. I have not been able to contact the administrator of the server to ask him to add it for several weeks. (7 Replies)
Discussion started by: cokedude
7 Replies

6. Shell Programming and Scripting

Transpose lines from individual blocks to unique lines

Hello to all, happy new year 2013! May somebody could help me, is about a very similar problem to the problem I've posted here where the member rdrtx1 and bipinajith helped me a lot. https://www.unix.com/shell-programming-scripting/211147-map-values-blocks-single-line-2.html It is very... (3 Replies)
Discussion started by: Ophiuchus
3 Replies

7. UNIX for Dummies Questions & Answers

Help removing multiple lines from one file with a list from a second file!

I am trying to remove the lines listed in example File A from File B to achieve File C. Both files are much larger than the examples below. (File B has up to 6,000 lines). I have searched the forums and I have not been able to find an answer to this particular question. I tried grep -v -f... (2 Replies)
Discussion started by: pezziza
2 Replies

8. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

9. Shell Programming and Scripting

Removing lines from one list from another list.

Hello, I was wondering if there was an easy way to take lines from a single-column list, and remove them from a second single-column list. For example, I want to remove the contents of list 1 from list 2. How would I do this? Contents of list 1: server1a server2b server3c server4a... (2 Replies)
Discussion started by: LinuxRacr
2 Replies

10. Shell Programming and Scripting

Remove All Lines Between Two Unique Lines

Hi all! Im wondering if its possible to remove all lines between two lines. Im working with a document like this: data1 data2 <Remove> data3 data4 </Remove> data5 data6 I need it to end up like this if that possible: data1 data2 data5 data6 There are multiple instances of... (2 Replies)
Discussion started by: Grizzly
2 Replies
Login or Register to Ask a Question