Search and remove the lines


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Search and remove the lines
# 1  
Old 07-19-2013
Search and remove the lines

Hallo Team,

Hope you are having a wonderful Friday. Here goes i am searching for a pattern and after finding the lines which contain this pattern i want to remove/delete them. This is my code:

Code:
 grep Originating BW*2013*|grep -v "ACCOUNT NOT FOUND"|grep -v "Unknown called number"|grep -v "Y,Unknown called on-net party"|grep -v "|Y,Unknown calling on-net party"|grep -v "Y,ACCOUNT IN RECALC."|grep "|Y,Unclear scenario:On-net to Off-net PGW, On-net to Off-net Partner IP:BSO,PGWO"

My output is like below and i would like to delete the lines or files:

Code:
BW-CDR-20130711180000-2-AC162DB137C8-037434-000.csv_rejected_4213307-20130718193659.log:0072782780AC162DB137C820130711155727.3020+020000,MRPRICE,Normal,+27313671590,,Originating,+27313671590,Public,0218369379,20130711155727.302,0+020000,Yes,20130711155758.255,20130711155803.894,016,VoIP,,0218369379,interlat,to,0218369379,,remote,10.0.108.11:5060,BW175727304110713-2083830430@196.35.130.5,G729/8000/1,10.0.12.58,4864d0100007f13ce000a4f4b00027100@127.0.0.1,,,,mrp_headoffice-cco,,,,,,,,,,y,public,,832627663:2227,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0313671590@isvoip.net,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,34.717,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Network,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,|Y,Unclear scenario:On-net to Off-net PGW, On-net to Off-net Partner IP:BSO,PGWO
BW-CDR-20130712074500-2-AC162DB137C8-037599-000.csv_rejected_4214106-20130718193659.log:0072880124AC162DB137C820130712053952.2820+020000,NORTON,Normal,+27110170600,,Originating,+27110170600,Anonymous,0218026894,20130712053952.282,0+020000,Yes,20130712054019.351,20130712054019.415,016,VoIP,,0218026894,interlat,to,0218026894,,local,10.0.108.11:5060,BW073952284120713166667830@196.35.130.5,G729/8000/1,10.0.4.117,B27FAE28-E9EC11E2-B056A9B6-23934ADB@196.38.51.242,,,,NORTON-HO,,,,,,,,,,y,public,,833851331:2448,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0110170600@isvoip.net,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,26.123,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Network,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,|Y,Unclear scenario:On-net to Off-net PGW, On-net to Off-net Partner IP:BSO,PGWO
BW-CDR-20130712084500-2-AC162DB137C8-037611-000.csv_rejected_4214495-20130718193659.log:0073014403AC162DB137C820130712064057.4190+020000,NORTON,Normal,+27873510673,,Originating,+27873510673,Public,0319041547,20130712064057.419,0+020000,Yes,20130712064102.933,20130712064106.378,016,VoIP,,0319041547,interlat,to,0319041547,,local,10.0.108.11:5060,BW084057422120713-197885586@196.35.130.5,G729/8000/1,10.0.4.117,B35406D4-E9F611E2-ABA8E293-3FE9594E@192.168.35.2,,,,NORTON-JUNE,,,,,,,,,,y,public,,834288043:143,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0873510673@isvoip.net,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,3.445,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Network,,,,,,,,,,,,,,,,+27873510673,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,|Y,Unclear scenario:On-net to Off-net PGW, On-net to Off-net Partner IP:BSO,PGWO
BW-CDR-20130712084500-2-AC162DB137C8-037611-000.csv_rejected_4214495-20130718193659.log:0073023480AC162DB137C820130712064304.9150+020000,RCS,Normal,+27215974210,,Originating,+27215974210,Anonymous,0153075587,20130712064304.915,0+020000,Yes,20130712064321.138,20130712064328.386,016,VoIP,,0153075587,interlat,to,0153075587,,local,10.0.108.11:5060,BW0843049181207131443732226@196.35.130.5,G729/8000/1,10.0.8.70,4C4196DA-E9F811E2-8D55E8C5-EFB54917@196.14.62.229,,,,RCS-CPT,,,,,,,,,,y,public,,834009203:4927,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0215974210@isvoip.net,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,9.518,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Network,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,|Y,Unclear scenario:On-net to Off-net PGW, On-net to Off-net Partner IP:BSO,PGWO
BW-CDR-20130712085000-2-AC162DB137C8-037612-000.csv_rejected_4214161-20130718193659.log:0073036207AC162DB137C820130712064336.1780+020000,VELOCITY,Normal,+27313278050,,Originating,+27313278050,Anonymous,0129918581,20130712064336.178,0+020000,Yes,20130712064509.255,20130712064643.114,016,VoIP,,0129918581,interlat,to,0129918581,,local,10.0.108.11:5060,BW0843361801207131128716458@196.35.130.5,G729/8000/1,10.0.12.18,298B8C35-EA0211E2-A4B9A37C-B09F2E77@196.35.99.82,,,,VELOCITY_UMH,,,,,,,,,,y,public,,834030275:820,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0313278050@isvoip.net,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,185.546,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Network,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,|Y,Unclear scenario:On-net to Off-net PGW, On-net to Off-net Partner IP:BSO,PGWO
BW-CDR-20130712085500-2-AC162DB137C8-037613-000.csv_rejected_4213301-20130718193659.log:0073051501AC162DB137C820130712064903.3170+020000,SAICOM-CORP,Normal,+27873516000,,Originating,+27873516000,Anonymous,0357870631,20130712064903.317,0+020000,Yes,20130712064925.379,20130712065037.397,016,VoIP,,0357870631,interlat,to,0357870631,,local,10.0.108.11:5060,BW084903320120713506949740@196.35.130.5,G729/8000/1,10.0.24.50,04D109C9-FEFB-4CCC-9282-8FBCB41BA855,,,,CORPORATE,,,,,,,,,,y,public,,834488621:4,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0873516000@isvoip.net,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,72.018,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Network,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,|Y,Unclear scenario:On-net to Off-net PGW, On-net to Off-net Partner IP:BSO,PGWO
BW-CDR-20130712085500-2-AC162DB137C8-037613-000.csv_rejected_4213301-20130718193659.log:0073065650AC162DB137C820130712065223.3030+020000,SEESA,Normal,+27219109256,,Originating,+27219109256,Public,0214214016,20130712065223.303,0+020000,Yes,20130712065340.993,20130712065404.457,016,VoIP,,0214214016,local,to,0214214016,,local,10.0.108.11:5060,BW0852233061207131522387561@196.35.130.5,G729/8000/1,10.0.8.97,89650630519337@192.168.3.20,,,,SEESA-CPT,,,,,,,,,,y,public,,834541151:0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0219109256@isvoip.net,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,98.267,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Network,,,,,,,,,,,,,,,,+27219109256,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,|Y,Unclear scenario:On-net to Off-net PGW, On-net to Off-net Partner IP:BSO,PGWO
BW-CDR-20130712090500-2-AC162DB137C8-037615-000.csv_rejected_4216427-20130718195021.log:0073096717AC162DB137C820130712065912.3740+020000,AVUSA,Normal,+27114714072,,Originating,+27114714072,Anonymous,0110352300,20130712065912.374,0+020000,Yes,20130712065916.303,20130712070141.409,016,VoIP,,0110352300,private,to,0110352300,,local,10.0.108.11:5060,BW085912376120713378639972@196.35.130.5,G729/8000/1,10.0.24.157,34C7D79D-E9F811E2-9821C520-E97C0089@172.17.84.2,,,,AVUSA-INDUSTRIA,,,,,,,,,,y,public,,834620763:1,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0114714072@isvoip.net,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,147.752,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Network,,,,,,,,,,,,,,,,+27114714072,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,|Y,Unclear scenario:On-net to Off-net PGW, On-net to Off-net Partner IP:BSO,PGWO
BW-CDR-20130712090500-2-AC162DB137C8-037615-000.csv_rejected_4216669-20130718195021.log:0073106024AC162DB137C820130712070250.8060+020000,MURROB-VOICE,Normal,+27114561000,,Originating,+27114561000,Anonymous,0115932184,20130712070250.806,0+020000,Yes,20130712070252.011,20130712070356.550,016,VoIP,,0115932184,local,to,0115932184,,remote,10.0.108.11:5060,BW0902508091207131728834809@196.35.130.5,G729/8000/1,10.0.0.102,f398e80-1df1aa1a-234d8c-a1086e0a@10.110.8.161,,,,DRC,,,,,,,,,,y,public,,833824631:1598,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0114561000@isvoip.net,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,64.538,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Network,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,|Y,Unclear scenario:On-net to Off-net PGW, On-net to Off-net Partner IP:BSO,PGWO
BW-CDR-20130712091500-2-AC162DB137C8-037617-000.csv_rejected_4215097-20130718193659.log:0073140848AC162DB137C820130712065558.6470+020000,FUEL-RTT,Normal,+27115521219,,Originating,+27115521219,Public,0538321273,20130712065558.647,0+020000,Yes,20130712065607.980,20130712071215.187,016,VoIP,,0538321273,private,to,0538321273,,local,10.0.108.20:5060,BW085558650120713-2132867107@196.35.130.5,G729/8000/1,10.0.4.160,7a191f540c17c53359075e850163b863@10.98.20.4:5060,,,,KEMPTONPARK-DC,,,,,,,,,,y,public,,834586903:0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0115521219@isvoip.net,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,970.546,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Network,,,,,,,,,,,,,,,,+27115521219,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,|Y,Unclear scenario:On-net to Off-net PGW, On-net to Off-net Partner IP:BSO,PGWO
-bash-3.2$ grep Originating BW*2013*|grep -v "ACCOUNT NOT FOUND"|grep -v "Unknown called number"|grep -v "Y,Unknown called on-net party"|grep -v "|Y,Unknown calling on-net party"|grep -v "Y,ACCOUNT IN RECALC."|grep "|Y,Unclear scenario:On-net to Off-net PGW, On-net to Off-net Partner IP:BSO,PGWO"|head
-bash-3.2$

Thanking you in advance.
# 2  
Old 07-19-2013
Not clear.
Can u give us your input file and what output you expect?
# 3  
Old 07-19-2013
So, could you put all the unwanted patterns in a file, similar to this:-
Code:
ACCOUNT NOT FOUND
Unknown called number
Y,Unknown called on-net party
Y,ACCOUNT IN RECALC.

If so, you can then:-
Code:
grep -vf exclude_patterns_file   data_file|grep "|Y,Unclear scenario:On-net to Off-net PGW, On-net to Off-net Partner IP:BSO,PGWO"

....but they seems awfully long search patterns.


Is this the right sort of approach? Can you reduce and neaten the patterns and not include/exclude the wrong things?

Eg. to have a pattern that looks for abc or abd, you can use something more like:-
Code:
grep "ab[cd]" file


Just as a matter of security, you're not publishing customer contact details here are you? If so, please sanitise them. I wouldn't like the data to be misused.


I hope that this helps, but if I've missed the point, please clarify what you actually need with some sanitised sample input from a single file and your expected output.


Thanks,

Robin
Liverpool/Blackburn
UK
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to remove lines that do not start with digit and combine line or lines

I have been searching and trying to come up with an awk that will perform the following on a converted text file (original is a pdf). 1. Since the first two lines are (begin with) text they are removed 2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies

2. Shell Programming and Scripting

Remove lines that are subsets of other lines in File

Hello everyone, Although it seems easy, I've been stuck with this problem for a moment now and I can't figure out a way to get it done. My problem is the following: I have a file where each line is a sequence of IP addresses, example : 10.0.0.1 10.0.0.2 10.0.0.5 10.0.0.1 10.0.0.2... (5 Replies)
Discussion started by: MisterJellyBean
5 Replies

3. Shell Programming and Scripting

Search pattern on logfile and search for day/dates and skip duplicate lines if any

Hi, I've written a script to search for an Oracle ORA- error on a log file, print that line and the .trc file associated with it as well as the dateline of when I assumed the error occured. In most it is the first dateline previous to the error. Unfortunately, this is not a fool proof script.... (2 Replies)
Discussion started by: newbie_01
2 Replies

4. Shell Programming and Scripting

Two files, remove lines from second based on lines in first

I have two files, a keepout.txt and a database.csv. They're unsorted, but could be sorted. keepout: user1 buser3 anuser19 notheruser27 database: user1,2343,"information about",field,blah,34 user2,4231,"mo info",etc,stuff,43 notheruser27,4344,"hiya",thing,more thing,423... (4 Replies)
Discussion started by: esoffron
4 Replies

5. UNIX for Dummies Questions & Answers

Want to remove all lines but not latest 50 lines from a file

Hi, I have a huge file which has Lacs of lines. File system got full. I want your guys help to suggest me a solution so that I can remove all lines from that file but not last 50,000 lines. I want solution which can remove lines from existing file so that I can have some space left with. (28 Replies)
Discussion started by: prashant2507198
28 Replies

6. Shell Programming and Scripting

How to search multiple patterns and remove lines from a file?

Hi, I have a file content as below. Table : PAYR Displayed fields: 15 of 15 Fixed columns: 4 List width 0999... (4 Replies)
Discussion started by: shirdi
4 Replies

7. Shell Programming and Scripting

search and replace, when found, delete multiple lines, add new set of lines?

hey guys, I tried searching but most 'search and replace' questions are related to one liners. Say I have a file to be replaced that has the following: $ cat testing.txt TESTING AAA BBB CCC DDD EEE FFF GGG HHH ENDTESTING This is the input file: (3 Replies)
Discussion started by: DeuceLee
3 Replies

8. Shell Programming and Scripting

remove blank lines and merge lines in shell

Hi, I'm not a expert in shell programming, so i've come here to take help from u gurus. I'm trying to tailor a csv file that i got to make it work for the LOAD FROM command. I've a datatable csv of the below format - --in file format xx,xx,xx ,xx , , , , ,,xx, xxxx,, ,, xxx,... (11 Replies)
Discussion started by: dvah
11 Replies

9. Shell Programming and Scripting

Search and Remove Lines within File

Hello, I've searched through the scripting section but could not find what I need. I need to search for empty sections within a file and remove them. Here is an example file: Title 123 A B C D E Title 098 Title 567 Z Y (4 Replies)
Discussion started by: leepet01
4 Replies

10. Shell Programming and Scripting

different take on common ?: search for two strings and remove lines between them

Thank you for assisting, I've got a partial solution just needs a tweak. Hulk-BASH$ cat somefile.txt oh there is some stuff here some more stuff here START_LABEL stuff I want more stuff I want END_LABEL other stuff here too and even more stuff here too Hulk-BASH$ Hulk-BASH$ sed... (8 Replies)
Discussion started by: laser
8 Replies
Login or Register to Ask a Question