Script to remove same content from other file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to remove same content from other file
# 8  
Old 09-09-2013
Quote:
Originally Posted by Jotne
Do you run your test with file you posted in post #1, or real data.
If it does not work on real data, only test data, do post real data.
tested with a similar test data.

Code:
bash-3.00# grep -vf a.out b.out > c.out
grep: illegal option -- f
Usage: grep -hblcnsviw pattern file . . .
bash-3.00# awk 'FNR==NR{a[$0]++;next} !($0 in a) ' a.out b.out >c.out
awk: syntax error near line 1
awk: bailing out near line 1
bash-3.00# more a.out
a
b
c
d
bash-3.00# more b.out
a
a
a
b
b
c
d
d
e
f

Quote:
Originally Posted by MadeInGermany
In Solaris 10 you must use the Posix variants /usr/xpg4/bin/grep and /usr/xpg4/bin/awk. The ones in /usr/bin/ are from old Unix SysV 4.0.
Especially /usr/bin/awk is a link to /usr/bin/oawk (already AT&T said it's old and provided /usr/bin/nawk).
The previous awk code rewritten for oawk:
Code:
awk '(FILENAME=="-") {a[$0]++; next} (a[$0]==0)' - <a.out b.out >c.out

oh i see...

yup this one works perfect

Code:
bash-3.00# awk '(FILENAME=="-") {a[$0]++; next} (a[$0]==0)' - <a.out b.out >c.out
bash-3.00# more c.out
e
f

---------- Post updated at 05:42 PM ---------- Previous update was at 04:51 PM ----------

Quote:
Originally Posted by Mr_47
tested with a similar test data.

Code:
bash-3.00# grep -vf a.out b.out > c.out
grep: illegal option -- f
Usage: grep -hblcnsviw pattern file . . .
bash-3.00# awk 'FNR==NR{a[$0]++;next} !($0 in a) ' a.out b.out >c.out
awk: syntax error near line 1
awk: bailing out near line 1
bash-3.00# more a.out
a
b
c
d
bash-3.00# more b.out
a
a
a
b
b
c
d
d
e
f



oh i see...

yup this one works perfect

Code:
bash-3.00# awk '(FILENAME=="-") {a[$0]++; next} (a[$0]==0)' - <a.out b.out >c.out
bash-3.00# more c.out
e
f


however this is still much faster with 1.5mil data, result with both codes is still the same, already double checked it.

Code:
fgrep -vxf a.out b.out > c.out


Last edited by Mr_47; 09-09-2013 at 08:07 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to remove exisiting file content from a file and have to append new file content?

hi all, i had the below script x=`cat input.txt |wc -1` awk 'NR>1 && NR<'$x' ' input.txt > output.txt by using above script i am able to remove the head and tail part from the input file and able to append the output to the output.txt but if i run it for second time the output is... (2 Replies)
Discussion started by: hemanthsaikumar
2 Replies

2. Shell Programming and Scripting

Remove the duplicate content in a file

Here is the contents of test.txt Dependencies Resolved Changes in packages about to be updated: ChangeLog for: 1:perl-Archive-Extract-0.38-131.el6_4.x86_64, - Resolves: #915692 - CVE-2013-1667 (DoS in rehashing code) Dependencies Resolved Changes in packages about to be updated: ... (5 Replies)
Discussion started by: ashokvpp
5 Replies

3. Shell Programming and Scripting

Shell script to monitor new file in a directory and mail the file content

Hi I am looking for a help in designing a bash script on linux which can do below:- 1) Look in a specific directory for any new files 2) Mail the content of the new file Appreciate any help Regards Neha (5 Replies)
Discussion started by: neha0785
5 Replies

4. Shell Programming and Scripting

Remove some content from a verilog file

Hi, I have a verilog file which looks like module xyz (x, y, z, a, b, c); input x; input y; input z; output a; output b; output c; initial begin ... end always ... ... endmodule (7 Replies)
Discussion started by: matbrow
7 Replies

5. Shell Programming and Scripting

Remove the file content based on the Header of the file

Hi All, I want to remove the content based on the header information . Please find the example below. File1.txt Name|Last|First|Location|DepId|Depname|DepLoc naga|rr|tion|hyd|1|wer|opr Nava|ra|tin|gen|2|wera|opra I have to search for the DepId and remove the data from the... (5 Replies)
Discussion started by: i150371485
5 Replies

6. Shell Programming and Scripting

Shell script to remove some content in a file

How can I remove all data that contain domain e.g zzgh@something.com, sdd@something.com.my and gg@something.my in one file? so that i only have data without the domain in the file. Here is the file structure "test.out" more test.out 1 zzztop@b.com 1 zzzulll 1 zzzullll@s.com.my ... (4 Replies)
Discussion started by: Mr_47
4 Replies

7. Shell Programming and Scripting

Help with remove duplicate content and only keep the first content detail

Input data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_21 SSA data_19 TYUEC data_14 TYUE data_15 SSA data_32 PEOCV . . Desired Output data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_19 TYUEC (9 Replies)
Discussion started by: patrick87
9 Replies

8. Shell Programming and Scripting

Remove specific content in a file

Hi, I have a file called fl_list consists of files i have to archive. I want to create a exception parm called except_parm, so if it finds the directory it will not archive these files and remove from fl_list. $ cat fl_list /apps/dev/ihub/ready/IA003B/IA003B_Deal_Header_yyyymmdd_hhmmss.txt... (1 Reply)
Discussion started by: k9cheung
1 Replies

9. Shell Programming and Scripting

remove space from file content

i am a bit new to shell scripting i have a file containing xxxx xx xx but i want to output the content as xxxxxxxx. thus removing the space. any idea how i can do this (4 Replies)
Discussion started by: blackzinga
4 Replies

10. Shell Programming and Scripting

Remove content from file

hey all, I have a file with records in following format 8-29-2006 13:01:45|ABC|45 8-29-2006 14:23:12|DEF|21 8-30-2006 00:04:57|ABC|34 I want to remove all yesterday records. Can anyone show me how? Thanks! (10 Replies)
Discussion started by: mpang_
10 Replies
Login or Register to Ask a Question