Delete lines containing key words dynamically


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Delete lines containing key words dynamically
# 1  
Old 05-15-2017
Delete lines containing key words dynamically

Hi Frens,

I have a requirement where I need to delete lines having key words and am using the below command to do that

Code:
sed '/UNIX/d' inputfile > output

But now I have one more requirement where in there will be one reference file which has the ID's to be deleted from the master file.

Eg:

Master File

Code:
ID,NAME
1,XX
2,YY
3,ZZ
4,DD

Reference File

Code:
ID
2
3

So my output file should be

Code:
ID,NAME
1,XX
4,DD

It would be helpful if you can let me know how we can acheive it dynamically.

Last edited by Don Cragun; 05-16-2017 at 02:26 AM.. Reason: Add CODE tags.
# 2  
Old 05-15-2017
Hi.

Unclear to me what is meant by dynamically. Here is a solution with grep:
Code:
#!/usr/bin/env bash

# @(#) s1       Demonstrate delete lines with reference file, grep.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C grep

FILE=${1-data1}
E=expected-output.txt

pl " Input data file and reference file data2:"
head data[12]

pl " Expected output:"
cat $E

pl " Results:"
grep -vf data2 data1 |
tee f1

pl " Verify results if possible:"
C=$HOME/bin/pass-fail
[ -f $C ] && $C || ( pe; pe " Results cannot be verified." ) >&2

exit 0

producing:
Code:
$ ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.7 (jessie) 
bash GNU bash 4.3.30
grep (GNU grep) 2.20

-----
 Input data file and reference file data2:
==> data1 <==
1,XX
2,YY
3,ZZ
4,DD

==> data2 <==
2
3

-----
 Expected output:
1,XX
4,DD

-----
 Results:
1,XX
4,DD

-----
 Verify results if possible:

-----
 Comparison of 2 created lines with 2 lines of desired results:
 Succeeded -- files (computed) f1 and (standard) expected-output.txt have same content.

See man grep for details.

Best wishes ... cheers, drl
This User Gave Thanks to drl For This Post:
# 3  
Old 05-16-2017
Thanks drl for your time. It works and once again thanks for your step by step explanation.
# 4  
Old 05-16-2017
Hi weknowd,
Note that according to post #1 in this thread, the desired output included the header that was present in both input files, while the output produced by:
Code:
grep -vf file2 file1

deleted the header line. Note also that if the ID values are of varying lengths and one of the IDs in file2 to be removed from the master file (file1) also appears as a substring of another ID or appears in the NAME field, the suggested code may remove additional lines that have IDs that are not included in file2. For example, if file1 contained:
Code:
ID,NAME
123,Jane Doe
2,John Doe
312,Jack Smith
421,Jim Taylor
567,Fred Zahn

and file2 contained:
Code:
ID
2
3

the given code will not only remove the header line, a line with ID 2, and a line with ID 3; it will also remove lines with IDs 12, 21-29, 32, 42, ... and lines with IDs 13, 23, 31, 33-39, 43, ....; i.e., the results would only be:
Code:
456,Fred Zahn

not:
Code:
ID,NAME
123,Jane Doe
312,Jack Smith
421,Jim Taylor
567,Fred Zahn

With your problem statement, we have no idea whether or not the code drl suggested works for real data you may need to process or just works for the sample data you provided.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

How to block some key words in my url for apache config?

Hi Folks, I am running a website and that needs to be tightened with security in terms of hacking... Whereas, In my URL, when i click on certain links the entire link as contains some words like below:/control_panel /controlpanel /admin /cms Whereas, i need to block those words in apache... (1 Reply)
Discussion started by: gsiva
1 Replies

2. UNIX for Beginners Questions & Answers

Non-root script used search and list specific key words

Hy there all. Im new here. Olso new to terminal & bash, but it seams that for me it's much easyer to undarsatnd scripts than an actual programming language as c or anyother languare for that matter. S-o here is one og my home works s-o to speak. Write a shell script which: -only works as a... (1 Reply)
Discussion started by: Crisso2Face
1 Replies

3. Shell Programming and Scripting

Using key words or symbols to determine parser

There are multiple conditions in which the user can input data. 1. > 2. del 3. ins 4. delins The user input is saved in a specific directory using the code below: Code: cd 'C:' C:/Users/cmccabe/Desktop/Python27/python.exe C:/Users/cmccabe/Desktop/Python27/run_batch_job.py... (12 Replies)
Discussion started by: cmccabe
12 Replies

4. Shell Programming and Scripting

Shell Script @ Find a key word and If the key word matches then replace next 7 lines only

Hi All, I have a XML file which is looks like as below. <<please see the attachment >> <?xml version="1.0" encoding="UTF-8"?> <esites> <esite> <name>XXX.com</name> <storeId>10001</storeId> <module> ... (4 Replies)
Discussion started by: Rajeev_hbk
4 Replies

5. UNIX for Dummies Questions & Answers

Delete lines according to a key words in that line

HI, I have a file A like this: c 1 length 14432 width 3434 temp 34 c 2 length 3343 width 0923 height 9383 hm 902 temp34 c 3 length 938 height 982 hm 9292 temp 23 ... (2 Replies)
Discussion started by: the_simpsons
2 Replies

6. UNIX for Dummies Questions & Answers

Filter logs for key words and create an index

well, im a total novice of unix commands and shell scripting, i never made anything. But yesterday in my work i was working with a lot of log files and i was wondering if there is a command for doing something like this: i have a bunch of files with text like this: blablabla errorcode1... (11 Replies)
Discussion started by: matius_88
11 Replies

7. Shell Programming and Scripting

Extract key words and print their values

Input file (HTTP request log file): GET... (2 Replies)
Discussion started by: buptwy
2 Replies

8. Shell Programming and Scripting

grep - Extracting multiple key words from stdout

Hello. From command line, the command zypper info nxclient return a bloc of data : linux local # zypper info nxclient Loading repository data... Reading installed packages... Information for package nxclient: Repository: zypper_local Name: nxclient Version: 3.5.0-7 Arch: x86_64... (7 Replies)
Discussion started by: jcdole
7 Replies

9. Shell Programming and Scripting

SED - delete words between two possible words

Hi all, I want to make an script using sed that removes everything between 'begin' (including the line that has it) and 'end1' or 'end2', not removing this line. Let me paste an 2 examples: anything before any string begin few lines of content end1 anything after anything before any... (4 Replies)
Discussion started by: meuser
4 Replies

10. Shell Programming and Scripting

Delete lines that contain 3 or more words?

How can I delete lines that contain 3 or more words? I have a file, old.txt, that has multi-word phrases in it and I want to remove the lines with 3 words or more and send the output to new.txt. I've tried the following using sed but it doesn't seem to work: sed '/(\b\w+\b){3,}/d' old.txt >... (5 Replies)
Discussion started by: revax
5 Replies
Login or Register to Ask a Question