Visit Our UNIX and Linux User Community

Top Forums Shell Programming and Scripting Script to remove same content from other file Post 302851473 by Mr_47 on Monday 9th of September 2013 05:42:00 AM
Old 09-09-2013
Quote:
Originally Posted by Jotne
Do you run your test with file you posted in post #1, or real data.
If it does not work on real data, only test data, do post real data.
tested with a similar test data.

Code:
bash-3.00# grep -vf a.out b.out > c.out
grep: illegal option -- f
Usage: grep -hblcnsviw pattern file . . .
bash-3.00# awk 'FNR==NR{a[$0]++;next} !($0 in a) ' a.out b.out >c.out
awk: syntax error near line 1
awk: bailing out near line 1
bash-3.00# more a.out
a
b
c
d
bash-3.00# more b.out
a
a
a
b
b
c
d
d
e
f

Quote:
Originally Posted by MadeInGermany
In Solaris 10 you must use the Posix variants /usr/xpg4/bin/grep and /usr/xpg4/bin/awk. The ones in /usr/bin/ are from old Unix SysV 4.0.
Especially /usr/bin/awk is a link to /usr/bin/oawk (already AT&T said it's old and provided /usr/bin/nawk).
The previous awk code rewritten for oawk:
Code:
awk '(FILENAME=="-") {a[$0]++; next} (a[$0]==0)' - <a.out b.out >c.out

oh i see...

yup this one works perfect

Code:
bash-3.00# awk '(FILENAME=="-") {a[$0]++; next} (a[$0]==0)' - <a.out b.out >c.out
bash-3.00# more c.out
e
f

---------- Post updated at 05:42 PM ---------- Previous update was at 04:51 PM ----------

Quote:
Originally Posted by Mr_47
tested with a similar test data.

Code:
bash-3.00# grep -vf a.out b.out > c.out
grep: illegal option -- f
Usage: grep -hblcnsviw pattern file . . .
bash-3.00# awk 'FNR==NR{a[$0]++;next} !($0 in a) ' a.out b.out >c.out
awk: syntax error near line 1
awk: bailing out near line 1
bash-3.00# more a.out
a
b
c
d
bash-3.00# more b.out
a
a
a
b
b
c
d
d
e
f



oh i see...

yup this one works perfect

Code:
bash-3.00# awk '(FILENAME=="-") {a[$0]++; next} (a[$0]==0)' - <a.out b.out >c.out
bash-3.00# more c.out
e
f


however this is still much faster with 1.5mil data, result with both codes is still the same, already double checked it.

Code:
fgrep -vxf a.out b.out > c.out


Last edited by Mr_47; 09-09-2013 at 07:07 AM..
 
Test Your Knowledge in Computers #674
Difficulty: Medium
Wi-Fi does not use any parts of the IEEE 803 protocol.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove content from file

hey all, I have a file with records in following format 8-29-2006 13:01:45|ABC|45 8-29-2006 14:23:12|DEF|21 8-30-2006 00:04:57|ABC|34 I want to remove all yesterday records. Can anyone show me how? Thanks! (10 Replies)
Discussion started by: mpang_
10 Replies

2. Shell Programming and Scripting

remove space from file content

i am a bit new to shell scripting i have a file containing xxxx xx xx but i want to output the content as xxxxxxxx. thus removing the space. any idea how i can do this (4 Replies)
Discussion started by: blackzinga
4 Replies

3. Shell Programming and Scripting

Remove specific content in a file

Hi, I have a file called fl_list consists of files i have to archive. I want to create a exception parm called except_parm, so if it finds the directory it will not archive these files and remove from fl_list. $ cat fl_list /apps/dev/ihub/ready/IA003B/IA003B_Deal_Header_yyyymmdd_hhmmss.txt... (1 Reply)
Discussion started by: k9cheung
1 Replies

4. Shell Programming and Scripting

Help with remove duplicate content and only keep the first content detail

Input data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_21 SSA data_19 TYUEC data_14 TYUE data_15 SSA data_32 PEOCV . . Desired Output data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_19 TYUEC (9 Replies)
Discussion started by: patrick87
9 Replies

5. Shell Programming and Scripting

Shell script to remove some content in a file

How can I remove all data that contain domain e.g zzgh@something.com, sdd@something.com.my and gg@something.my in one file? so that i only have data without the domain in the file. Here is the file structure "test.out" more test.out 1 zzztop@b.com 1 zzzulll 1 zzzullll@s.com.my ... (4 Replies)
Discussion started by: Mr_47
4 Replies

6. Shell Programming and Scripting

Remove the file content based on the Header of the file

Hi All, I want to remove the content based on the header information . Please find the example below. File1.txt Name|Last|First|Location|DepId|Depname|DepLoc naga|rr|tion|hyd|1|wer|opr Nava|ra|tin|gen|2|wera|opra I have to search for the DepId and remove the data from the... (5 Replies)
Discussion started by: i150371485
5 Replies

7. Shell Programming and Scripting

Remove some content from a verilog file

Hi, I have a verilog file which looks like module xyz (x, y, z, a, b, c); input x; input y; input z; output a; output b; output c; initial begin ... end always ... ... endmodule (7 Replies)
Discussion started by: matbrow
7 Replies

8. Shell Programming and Scripting

Shell script to monitor new file in a directory and mail the file content

Hi I am looking for a help in designing a bash script on linux which can do below:- 1) Look in a specific directory for any new files 2) Mail the content of the new file Appreciate any help Regards Neha (5 Replies)
Discussion started by: neha0785
5 Replies

9. Shell Programming and Scripting

Remove the duplicate content in a file

Here is the contents of test.txt Dependencies Resolved Changes in packages about to be updated: ChangeLog for: 1:perl-Archive-Extract-0.38-131.el6_4.x86_64, - Resolves: #915692 - CVE-2013-1667 (DoS in rehashing code) Dependencies Resolved Changes in packages about to be updated: ... (5 Replies)
Discussion started by: ashokvpp
5 Replies

10. Shell Programming and Scripting

How to remove exisiting file content from a file and have to append new file content?

hi all, i had the below script x=`cat input.txt |wc -1` awk 'NR>1 && NR<'$x' ' input.txt > output.txt by using above script i am able to remove the head and tail part from the input file and able to append the output to the output.txt but if i run it for second time the output is... (2 Replies)
Discussion started by: hemanthsaikumar
2 Replies

Featured Tech Videos

All times are GMT -4. The time now is 11:51 AM.
Unix & Linux Forums Content Copyright 1993-2020. All Rights Reserved.
Privacy Policy