Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Removing Lines Shared by Multiple Files

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Removing Lines Shared by Multiple Files
# 1  
Old 06-18-2012
Removing Lines Shared by Multiple Files
Hey everyone, I have a question about comparing two files. I have two lists of files. The first list, todo.csv, lists a series of compounds my supervisor wants me to perform calculations on. The second list, done.csv, lists a series of compounds that I have already performed calculations on. However, the two lists overlap, and I am interested in reducing redundancy. So my question; is there any way to remove all lines from todo.csv that also occur in done.csv? I suspect awk could be used to do the trick, but I am not really sure how. Any help would be greatly appreciated.
# 2  
Old 06-18-2012
Take a look at two commands - COMM and DIFF
This User Gave Thanks to joeyg For This Post:
Stuart Ness
# 3  
Old 06-18-2012
Ah, thank you very much. I had no idea the comm command existed.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Selecting text on multiple lines, then removing a beginning and end patterns

I have a file similar to the below. I am selecting only the paragraphs with @inlineifset. I am using the following command sed '/@inlineifset/,/^ *$/!d; s/@inlineifset{mrg, @btpar{@//' $flnm >> $ofln This produces @section Correlations between seismograms,,,,}} ... (5 Replies)
Discussion started by: Danette
5 Replies

2. Shell Programming and Scripting

Removing carriage returns from multiple lines in multiple files of different number of columns

Hello Gurus, I have a multiple pipe separated files which have records going over multiple Lines. End of line separator is \n and records going over multiple lines have <CR> as separator. below is example from one file. 1|ABC DEF|100|10 2|PQ RS T|200|20 3| UVWXYZ|300|30 4| GHIJKL|400|40... (7 Replies)
Discussion started by: dJHa
7 Replies

3. Shell Programming and Scripting

Removing multiple lines from input file, if multiple lines match a pattern.

GM, I have an issue at work, which requires a simple solution. But, after multiple attempts, I have not been able to hit on the code needed. I am assuming that sed, awk or even perl could do what I need. I have an application that adds extra blank page feeds, for multiple reports, when... (7 Replies)
Discussion started by: jxfish2
7 Replies

4. Shell Programming and Scripting

Removing carriage return/line feeds on multiple lines

I would like to remove carriage returns/line feeds in a text file, but in a specific cadence: Read first line (Header Line 1), remove cr/lf at the end (replace it with a space ideally); Read the next line (Line of Text 2), leave the cr/lf intact; Read the next line, remove the cr/lf; Read... (14 Replies)
Discussion started by: tomr2012
14 Replies

5. Shell Programming and Scripting

Removing multiple lines but not the next attribute

I'm hitting a brick wall, I have huge ldif files that I'm trying to sanitize and can do it all with SED except one thing. I have a publicKey attribute in binary that can be one line or multiple lines. I'm trying to remove publicKey: and stop at sn (the next attribute). Even with word Wrap... (3 Replies)
Discussion started by: LCol
3 Replies

6. Shell Programming and Scripting

Removing Multiple lines below a keyword using SED?

I have 9,000 + html files. I am using the following to remove the content from a certain line up for i in `ls` do sed '1,569d' $i > $i.bak done This will remove the unwanted formatting keeping the content I need which changes in each HTML file. the problem I have now is that the... (2 Replies)
Discussion started by: deaconf19
2 Replies

7. Shell Programming and Scripting

PERL: removing blank lines from multiple files

Hi Guru's , I have a whole bunch of files in /var/tmp that i need to strip any blank lines from, so ive written the following script to identify the lines (which works perfectly).. but i wanted to know, how can I actually strip the identified lines from the actual source files ?? my... (11 Replies)
Discussion started by: hcclnoodles
11 Replies

8. UNIX for Dummies Questions & Answers

removing multiple lines of text in a file

Hi, I'm trying to remove multiple lines of text based off a series of different words and output it to a new file The document contains a ton of data but i want to delete any line that has the following mx1.rr.biz.com or ns2.ri.biz.com i tried using grep -v filename "mx1.rr.biz.com" >... (3 Replies)
Discussion started by: spartan22
3 Replies

9. Shell Programming and Scripting

Removing end of line to merge multiple lines

I'm sure this will be an easy question for you experts out there, but I have been searching the forum and working on this for a couple hours now and can't get it right. I have a very messy data file that I am trying to tidy up - one of the issues is some records are split into multiple lines: ... (4 Replies)
Discussion started by: tink
4 Replies

10. Shell Programming and Scripting

removing pattern which is spread in multiple lines

I have several huge files wich contains oracle table creation scripts as follows: I would need to remove the pattern colored in red above. Any sed/awk/pearl code will be of much help. Thanks (2 Replies)
Discussion started by: sabyasm
2 Replies
Login or Register to Ask a Question