Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Delete lines containing key words dynamically

Shell Programming and Scripting


Closed    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 05-15-2017   -   Original Discussion by weknowd
weknowd weknowd is offline
Registered User
 
Join Date: Jan 2010
Last Activity: 15 May 2017, 11:16 PM EDT
Posts: 42
Thanks: 8
Thanked 0 Times in 0 Posts
Delete lines containing key words dynamically

Hi Frens,

I have a requirement where I need to delete lines having key words and am using the below command to do that


Code:
sed '/UNIX/d' inputfile > output

But now I have one more requirement where in there will be one reference file which has the ID's to be deleted from the master file.

Eg:

Master File


Code:
ID,NAME
1,XX
2,YY
3,ZZ
4,DD

Reference File


Code:
ID
2
3

So my output file should be


Code:
ID,NAME
1,XX
4,DD

It would be helpful if you can let me know how we can acheive it dynamically.

Last edited by Don Cragun; 05-16-2017 at 02:26 AM.. Reason: Add CODE tags.
Sponsored Links
    #2  
Old Unix and Linux 05-15-2017   -   Original Discussion by weknowd
drl's Unix or Linux Image
drl drl is offline Forum Advisor  
Registered Voter
 
Join Date: Apr 2007
Last Activity: 21 November 2017, 8:03 PM EST
Location: Saint Paul, MN USA / BSD, CentOS, Debian, OS X, Solaris
Posts: 2,206
Thanks: 241
Thanked 413 Times in 357 Posts
Hi.

Unclear to me what is meant by dynamically. Here is a solution with grep:

Code:
#!/usr/bin/env bash

# @(#) s1       Demonstrate delete lines with reference file, grep.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C grep

FILE=${1-data1}
E=expected-output.txt

pl " Input data file and reference file data2:"
head data[12]

pl " Expected output:"
cat $E

pl " Results:"
grep -vf data2 data1 |
tee f1

pl " Verify results if possible:"
C=$HOME/bin/pass-fail
[ -f $C ] && $C || ( pe; pe " Results cannot be verified." ) >&2

exit 0

producing:

Code:
$ ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.7 (jessie) 
bash GNU bash 4.3.30
grep (GNU grep) 2.20

-----
 Input data file and reference file data2:
==> data1 <==
1,XX
2,YY
3,ZZ
4,DD

==> data2 <==
2
3

-----
 Expected output:
1,XX
4,DD

-----
 Results:
1,XX
4,DD

-----
 Verify results if possible:

-----
 Comparison of 2 created lines with 2 lines of desired results:
 Succeeded -- files (computed) f1 and (standard) expected-output.txt have same content.

See man grep for details.

Best wishes ... cheers, drl
The Following User Says Thank You to drl For This Useful Post:
weknowd (05-16-2017)
Sponsored Links
    #3  
Old Unix and Linux 05-16-2017   -   Original Discussion by weknowd
weknowd weknowd is offline
Registered User
 
Join Date: Jan 2010
Last Activity: 15 May 2017, 11:16 PM EDT
Posts: 42
Thanks: 8
Thanked 0 Times in 0 Posts
Thanks drl for your time. It works and once again thanks for your step by step explanation.
    #4  
Old Unix and Linux 05-16-2017   -   Original Discussion by weknowd
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 22 November 2017, 3:49 AM EST
Location: San Jose, CA, USA
Posts: 10,673
Thanks: 572
Thanked 3,737 Times in 3,189 Posts
Hi weknowd,
Note that according to post #1 in this thread, the desired output included the header that was present in both input files, while the output produced by:

Code:
grep -vf file2 file1

deleted the header line. Note also that if the ID values are of varying lengths and one of the IDs in file2 to be removed from the master file (file1) also appears as a substring of another ID or appears in the NAME field, the suggested code may remove additional lines that have IDs that are not included in file2. For example, if file1 contained:

Code:
ID,NAME
123,Jane Doe
2,John Doe
312,Jack Smith
421,Jim Taylor
567,Fred Zahn

and file2 contained:

Code:
ID
2
3

the given code will not only remove the header line, a line with ID 2, and a line with ID 3; it will also remove lines with IDs 12, 21-29, 32, 42, ... and lines with IDs 13, 23, 31, 33-39, 43, ....; i.e., the results would only be:

Code:
456,Fred Zahn

not:

Code:
ID,NAME
123,Jane Doe
312,Jack Smith
421,Jim Taylor
567,Fred Zahn

With your problem statement, we have no idea whether or not the code drl suggested works for real data you may need to process or just works for the sample data you provided.
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Delete lines according to a key words in that line the_simpsons UNIX for Dummies Questions & Answers 2 09-09-2013 09:43 AM
appending lines to a file dynamically Ananthdoss Shell Programming and Scripting 2 12-09-2011 08:41 AM
deleting few lines from a file dynamically vivek d r Shell Programming and Scripting 4 12-07-2011 05:46 AM
SED - delete words between two possible words meuser Shell Programming and Scripting 4 04-12-2011 08:42 AM
Delete lines that contain 3 or more words? revax Shell Programming and Scripting 5 12-11-2007 06:33 PM



All times are GMT -4. The time now is 06:57 AM.