Delete same lines out of file.


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Delete same lines out of file.
# 1  
Old 06-04-2009
Question Delete same lines out of file.

hello... I have a file with a list of filepaths in it, like so:

delMe.txt (files to be deleted)
Code:
./root/index.php
./root/language/se/home.inc.php
./root/language/pl/home.inc.php

Now here's what I'm trying to do. I want any lines in the above file (delMe.txt) to be deleted out of the original file they came from (below, orig.txt):

orig.txt
Code:
./root/includes/goobermods/Must_Login/index_inc.php
./root/index.php
./root/language/se/home.inc.php
./root/language/nl/home.inc.php
./root/language/fi/home.inc.php
./root/language/en/home.inc.php
./root/language/fr/home.inc.php
./root/language/es/home.inc.php
./root/language/pl/home.inc.php
./root/language/dk/home.inc.php
./root/language/cn/home.inc.php
./root/language/sk/home.inc.php
./root/language/pt/home.inc.php
./root/language/no/home.inc.php
./root/modules/gateway/index.php
./root/test/index.php

I started to do something like this (but think i'm making it complicated)
Code:
#!/bin/bash

lines=`cat delMe.txt | wc -l`

for i in `seq 1 $lines`
do
    sed -n $ip delMe.txt | xargs grep orig.txt...??
    #I HAVE NO IDEA AT THIS POINT ^^
done

thanks a bajillion in advance!!
# 2  
Old 06-04-2009
Code:
fgrep -v -f delMe.txt orig.txt

# 3  
Old 06-04-2009
Error

wait a second - i just realized all of my work is really pretty pointless and might be done in a single line.

what would anybody say is the easiest way to:
given a directory of files and subdirectories w/files, recursively remove an entire line from any file which contains string "this is a string to be removed" somewhere inside it
# 4  
Old 06-05-2009
Error Gumblar Took Over - Remove Strings

Thanks, but I just can't seem to figure out how to use fgrep to do what I want.

I need to remove all malicious lines of crap inserted by Gumblar. My site doesn't use any iframes, and gumblar places everything in as invisible iframes. so I figured I'd grep for all iframes, and than see what else they have in common. I found they had these two URLs in common:
Code:
http://clifedo.net/?click=
http://combinebet.cn:8080/index.php

Here's how I did the search for the files infected with those URLs:
(fyi, its not actually root that's being parsed, its a local copy of the entire public_html file tree)
Code:
#!/bin/bash

grep -ilr "http://combinebet.cn:8080/index.php" ./root/*; echo "
"
grep -ilr "http://clifedo.net/?click=" ./root/*; echo "
"

I redirected that ^ script's output to a file "infected.txt"
it looks like this: (cut off of course)
Code:
...
./root/admin/modules/3rdparty/advanced_image_manager/adv_imgmgr/index.php
./root/admin/modules/3rdparty/Mixable_Quantity_Discounts/index.php
./root/admin/modules/3rdparty/Discounts_By_Quantity/index.php
./root/admin/modules/3rdparty/index.php
./root/admin/modules/3rdparty/Must_Login/index.php
...

I then realised that the last line of the majority of the above found files is where the infection lies. So after saving a copy of the full infected.txt and then removing the file paths listed that didn't have an infection on the last line, I wrote this script.

Code:
#!/bin/bash

#PARAMETERS
fileParsed=infected.txt

lines=`cat $fileParsed | wc -l`

 #DELETES LAST LINE OF EACH FILE IN PARSED LIST
for i in `seq 1 $lines`;
do
    file=`cat $fileParsed | sed -n "$i"p`
    echo "File being edited is: $file" #TEST
    sed -i '$d' "$file"
    echo "the last line of $file is now:"
    cat $file | tail -1; echo "
"
done

Now I'm realising that was silly and inefficient. I should have done something like ^ that, but instead deleting the last line (in hopes that its no where else in the file) the script should delete ANY lines found (in each file it looks at) that have either "malicoiusString1" OR "maliciousString2".

I'm tearing my hair out (with my lack of programming experience) trying to figure out how to do this. I feel like I'm on the write track ^ but, I've just been stuck for a day and a half now.

ANY and all help would be greatly appreciated! Thanks so much in advance.


-----Post Update-----

Okay, this is what I came up with, if ANYone can try and help fill in my blanks/replace my semi-pseudocode, that would be amazing:
Code:
      1 #!/bin/bash
      2 
      3 #PARAMETERS
      4 fileParsed=infected.txt
      5 mal1="http://clifedo.net/?click=" #MALICIOUS STRING
      6 mal2="http://combinebet.cn:8080/index.php" #MALICIOUS STRING
      7 
      8 lines=`cat $fileParsed | wc -l`
      9 
     10 #DELETES LAST LINE OF EACH FILE IN PARSED LIST
     11 #THIS WHOLE FOR LOOP WILL BE WRITTEN TWICE, ONCE FOR EACH MAL# VARIABLE.
     12 for i in `seq 1 $lines`;
     13 do      
     14         file=`cat $fileParsed | sed -n "$i"p`
     15         echo "File being searched is: $file" #TEST
     16         num=`grep -inlr "mal1" $file`
     17         num= 1st DELIMITER of $num
     18         if $num IS DIGIT
     19         then    
     20                 #CAN ANYONE TELL ME IF THIS LINE IS/ISN'T THE BEST APPROACH?
     21                 sed -i "\$num"d' "$file"
     22         else    
     23                 echo "this file no longer has $mal1"
     24         fi
     25 done


Last edited by jzacsh; 06-05-2009 at 01:10 PM..
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Delete Some Lines from File

Hi, I have a txt document having a format like this: DATA1 | DATA2 | DATA3 | 23-JAN-20 23:41:34 DATA1 | DATA2 | DATA3 | 23-JAN-20 23:41:32 DATA1 | DATA2 | DATA3 | 23-JAN-20 23:41:30 ... DATA1 | DATA2 | DATA3 | 23-JAN-20 22:35:31 DATA1 | DATA2 | DATA3 | 23-JAN-20 22:30:34 DATA1 | DATA2 |... (1 Reply)
Discussion started by: gc_sw
1 Replies

2. Shell Programming and Scripting

Delete 40 lines after every 24 lines from a file

Hello, I have file of more than 10000 lines. I want to delete 40 lines after every 20 lines. e.g from a huge file, i want to delete line no from 34 - 74, then 94 - 134 etc and so on. Please let me know how i can do it. Best regards, (11 Replies)
Discussion started by: nehashine
11 Replies

3. UNIX for Dummies Questions & Answers

Delete lines in a file

Hi This is a sample of my data file. ##field PH01000000 1 4869017 #PH01000000G0240 WWW278545G0240 P.he_model_v1.0 erine 119238 121805 . - . ID=PH01000000G0240;Description="zinc finger, C3HC4 type domain containing protein, expressed"... (7 Replies)
Discussion started by: sonia102
7 Replies

4. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

5. UNIX for Dummies Questions & Answers

How get only required lines & delete the rest of the lines in file

Hiiii I have a file which contains huge data as a.dat: PDE 1990 1 9 18 51 28.90 24.7500 95.2800 118.0 6.1 0.0 BURMA event name: 010990D time shift: 7.3000 half duration: 5.0000 latitude: 24.4200 longitude: 94.9500 depth: 129.6000 Mrr: ... (7 Replies)
Discussion started by: reva
7 Replies

6. Shell Programming and Scripting

How to delete lines in a file that have duplicates or derive the lines that aper once

Input: a b b c d d I need: a c I know how to get this (the lines that have duplicates) : b d sort file | uniq -d But i need opossite of this. I have searched the forum and other places as well, but have found solution for everything except this variant of the problem. (3 Replies)
Discussion started by: necroman08
3 Replies

7. Shell Programming and Scripting

delete n last lines of a file

Hello!!! how can I delete the last n lines of a file??? Thanks (7 Replies)
Discussion started by: ncatdesigner
7 Replies

8. Shell Programming and Scripting

delete the lines from file

i have two files & want to delete the lines from 2nd file which matches with 1st file (2 Replies)
Discussion started by: sameersam
2 Replies

9. Shell Programming and Scripting

delete lines from a file.

I have a file which has about 500K records and I need to delete about 50 records from the file. I know line numbers and am using sed '13456,13457,......d' filename > new file. It does not seem to be working. Any help will greatly appreciated. (5 Replies)
Discussion started by: oracle8
5 Replies

10. UNIX for Dummies Questions & Answers

delete all lines in file

how can i delete all lines in file by using "vi" ? (6 Replies)
Discussion started by: strok
6 Replies
Login or Register to Ask a Question