removal by substring

Thread Tools Search this Thread
# 1  
removal by substring

Hi guys,

I am trying to remove lines that have a duplicate substring from any part in the file. So, for ex:

433043950359.3 5033 305935 2 2dd 5ffgs DOG43453552.A
3443565634 95460 3435 45 23d 56ggh DOG343211
3423895702359 34 66699 9455 2324 DOG43453552.B

This is a very large file, and I dont think
sort -u filename

will work because I need to look for a substring not an entire string.

any ideas?
# 2  
You need to qualify what you mean by "substring". For in your example, all three lines contain "3". And "4". And "D".

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removal of space

Hi , Can any one help me out how to remove space from below line select file_name from dba_data_files where tablespace_name='SYSTEM ----space---- '; i want as : select file_name from dba_data_files where tablespace_name='SYSTEM'; my code i use file=/u01/script/latest/tbs.temp while... (14 Replies)
Discussion started by: Praful Pednekar
14 Replies

2. Web Development

Vbseo removal

first off i want to thank you for such a great site, you helped me narrow down a long search on what was wrong with my forum, i have a large forum motorbicycling DOT com which i had someone remove vbseo before they went under because of all the security problems. anyway i have lots of 404 errors... (13 Replies)
Discussion started by: atcspaul
13 Replies

3. UNIX for Advanced & Expert Users

Duplicate removal

I have an input file of 5GB which contains duplicate records and have to remove duplicate records by retaing first instance of that record . Based on 5 fields the duplicates has to be removed . Kindly request to help me in writing a Unix Script. Thanks Asim (11 Replies)
Discussion started by: duplicate
11 Replies

4. Shell Programming and Scripting

Differential substring removal using coordinates

Hello all, this might be better suited for a bioinformatics forum, but I thought I'd try my luck here as well. I have several tabular text files of DNA sequence reads that appear as such: File_1.txt >H01BA45XW GATTACAGATTCGACATCCAACTGAGGCATT >H02BG78WR CCTTACAGACTGGGCATGAATATTGCATACC... (3 Replies)
Discussion started by: vectorborne5
3 Replies

5. UNIX for Dummies Questions & Answers

removal of space from the end

HI, I need the help from the experts like I have created one file with text like: Code: a b c de f g hi j k l So my question is that i have to write the script in which like in the first sentence it will take only one space after d and remove all the extra space in the end.I dont... (0 Replies)
Discussion started by: bhanudhingra
0 Replies

6. Shell Programming and Scripting

\n removal between two |

I have a multi-line string I'm trying to do some clean-up on. Example: 1|575|67866|"fgnhdgj"|"afhgajh agfajgf ahfjhgfk ahfkhf"|568 2|56|5435|"mayank"|"gupta gdja agdjagf"|677 3|5666|5435|"mayank"|"gupta gdja agdjagf"|677 I need a shell script that replace all \n under " ". (11 Replies)
Discussion started by: mayankgupta18
11 Replies

7. UNIX for Dummies Questions & Answers

semaphore removal

I have one process which creates semaphore with permission 600. Then at some part of the code I have line where this semaphore will be removed. Problem I have is that other processes in my program also try to execute this line of code. Of course since they dont have permission they wont be able to... (1 Reply)
Discussion started by: joker40
1 Replies

8. Homework & Coursework Questions

removal of files?

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: remove all files and only files that the first three characters start with numerals. 2. Relevant commands,... (6 Replies)
Discussion started by: linuxtraining
6 Replies

9. UNIX for Dummies Questions & Answers

Removal of Data

Hi All, (And first up, a Happy New Year to you all! 363 days to go!) I need to make sure that our old Sun Sparcstations and Servers are clear of any Confidential data before I return them... Suggestions? I'm formatting the discs to make sure that there's nothing obvious, but would... (5 Replies)
Discussion started by: geralex
5 Replies

10. UNIX for Dummies Questions & Answers


I am using Unix as my OS on my server and would like to format my hard drive. How do I go about wiping my hard drive or is there a removal tool that I can use? (1 Reply)
Discussion started by: anaconda
1 Replies

Featured Tech Videos