removal by substring

Login or Register to Ask a Question and Join Our Community

removal by substring

Tags

Top Forums UNIX for Dummies Questions & Answers removal by substring

11-23-2011

Registered User

83, 1

Join Date: Oct 2011

Last Activity: 1 June 2014, 10:33 PM EDT

Posts: 83

Thanks Given: 20

Thanked 1 Time in 1 Post

removal by substring

Hi guys,

I am trying to remove lines that have a duplicate substring from any part in the file. So, for ex:

433043950359.3 5033 305935 2 2dd 5ffgs DOG43453552.A
3443565634 95460 3435 45 23d 56ggh DOG343211
3423895702359 34 66699 9455 2324 DOG43453552.B

This is a very large file, and I dont think

Code:

sort -u filename

will work because I need to look for a substring not an entire string.

any ideas?

verse123

View Public Profile for verse123

Find all posts by verse123

11-30-2011

Registered User

313, 60

Join Date: Dec 2010

Last Activity: 7 December 2012, 7:50 PM EST

Location: Albany, NY

Posts: 313

Thanks Given: 15

Thanked 60 Times in 60 Posts

You need to qualify what you mean by "substring". For in your example, all three lines contain "3". And "4". And "D".

m.d.ludwig

View Public Profile for m.d.ludwig

Find all posts by m.d.ludwig

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removal of space

Hi , Can any one help me out how to remove space from below line select file_name from dba_data_files where tablespace_name='SYSTEM ----space---- '; i want as : select file_name from dba_data_files where tablespace_name='SYSTEM'; my code i use file=/u01/script/latest/tbs.temp while...

2. Web Development

Vbseo removal

first off i want to thank you for such a great site, you helped me narrow down a long search on what was wrong with my forum, i have a large forum motorbicycling DOT com which i had someone remove vbseo before they went under because of all the security problems. anyway i have lots of 404 errors...

3. UNIX for Advanced & Expert Users

Duplicate removal

I have an input file of 5GB which contains duplicate records and have to remove duplicate records by retaing first instance of that record . Based on 5 fields the duplicates has to be removed . Kindly request to help me in writing a Unix Script. Thanks Asim

4. Shell Programming and Scripting

Differential substring removal using coordinates

Hello all, this might be better suited for a bioinformatics forum, but I thought I'd try my luck here as well. I have several tabular text files of DNA sequence reads that appear as such: File_1.txt >H01BA45XW GATTACAGATTCGACATCCAACTGAGGCATT >H02BG78WR CCTTACAGACTGGGCATGAATATTGCATACC...

5. OS X (Apple)

vi and special character removal

To the group, when I copy text from a web page that has the below java code , and then do the set list command in the vi editor, I see the $ symbol at the end of each line. I have searched the internet looking for a way to remove this from the file since it will not compile without errors..Please...

6. Shell Programming and Scripting

\n removal between two |

I have a multi-line string I'm trying to do some clean-up on. Example: 1|575|67866|"fgnhdgj"|"afhgajh agfajgf ahfjhgfk ahfkhf"|568 2|56|5435|"mayank"|"gupta gdja agdjagf"|677 3|5666|5435|"mayank"|"gupta gdja agdjagf"|677 I need a shell script that replace all \n under " ".

7. UNIX for Dummies Questions & Answers

semaphore removal

I have one process which creates semaphore with permission 600. Then at some part of the code I have line where this semaphore will be removed. Problem I have is that other processes in my program also try to execute this line of code. Of course since they dont have permission they wont be able to...

8. Homework & Coursework Questions

removal of files?

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: remove all files and only files that the first three characters start with numerals. 2. Relevant commands,...

9. UNIX for Dummies Questions & Answers

Removal of Data

Hi All, (And first up, a Happy New Year to you all! 363 days to go!) I need to make sure that our old Sun Sparcstations and Servers are clear of any Confidential data before I return them... Suggestions? I'm formatting the discs to make sure that there's nothing obvious, but would...

10. UNIX for Dummies Questions & Answers

Removal

I am using Unix as my OS on my server and would like to format my hard drive. How do I go about wiping my hard drive or is there a removal tool that I can use?

Login or Register to Ask a Question