Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Script to remove same content from other file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to remove same content from other file
# 1  
Old 09-06-2013
Script to remove same content from other file
Hi/ Hello all Guru here,

I am trying to create script to remove same content from other file, already tested few idea and found that in unix it is limited to sort and uniq. There is many script for removing duplicate content however to delete all same content is non. Need your help and guide . thx

so here the situation

1. a.out a file with content that we need to delete in other file
Code:
more a.out
a
b
c
d


2. b.out a file that

Code:
more b.out
a
a
a
b
b
c
d
d
e
f


3. the result should be like c.out file

Code:
more c.out
e
f


note: the data is around 1.5 million list.

---------- Post updated at 03:55 PM ---------- Previous update was at 03:53 PM ----------

tested and not working and most only to delete duplication and im lost

Code:
sed -e "s/Text_1/TextA/" -e "s/Text1/TextB/" <your_file.txt>your_file_new.txt

cat deleteme.txt deleteme.txt masterlist.txt | sort | uniq -u > newmasterlist.txt

diff file-a file-b --new-line-format="" --old-line-format="%L" --unchanged-line-format="" > file-a

# 2  
Old 09-06-2013
Try

Code:
grep -vf a.out b.out > c.out

# 3  
Old 09-06-2013
Using awk
Code:
awk 'FNR==NR{a[$0]++;next} !($0 in a) ' a.out b.out >c.out
cat c.out
e
f

# 4  
Old 09-06-2013
Quote:
Originally Posted by pamu
Try

Code:
grep -vf a.out b.out > c.out

As I understand the requirement it should be
Code:
fgrep -vxf a.out b.out > c.out

No RE match, and full line match.
The difference is visible if b.out has a line
Code:
fa

This User Gave Thanks to MadeInGermany For This Post:
Mr_47
# 5  
Old 09-09-2013
Quote:
Originally Posted by pamu
Try

Code:
grep -vf a.out b.out > c.out

Quote:
Originally Posted by Jotne
Using awk
Code:
awk 'FNR==NR{a[$0]++;next} !($0 in a) ' a.out b.out >c.out
cat c.out
e
f

this is not working, Im using solaris 10 64bit ( unix ) btw

Code:
bash-3.00# grep -vf a.out b.out > c.out
grep: illegal option -- f
Usage: grep -hblcnsviw pattern file . . .
bash-3.00# awk 'FNR==NR{a[$0]++;next} !($0 in a) ' a.out b.out >c.out
awk: syntax error near line 1
awk: bailing out near line 1


Quote:
Originally Posted by MadeInGermany
As I understand the requirement it should be
Code:
fgrep -vxf a.out b.out > c.out

No RE match, and full line match.
The difference is visible if b.out has a line
Code:
fa

this is perfectly working!!! thx a million mate. also as checked all the similar entries was removed in c.out.

Code:
bash-3.00# fgrep -vxf a.out b.out > c.out
bash-3.00# more c.out|wc -l
 1481733

# 6  
Old 09-09-2013
Do you run your test with file you posted in post #1, or real data.
If it does not work on real data, only test data, do post real data.
# 7  
Old 09-09-2013
In Solaris 10 you must use the Posix variants /usr/xpg4/bin/grep and /usr/xpg4/bin/awk. The ones in /usr/bin/ are from old Unix SysV 4.0.
Especially /usr/bin/awk is a link to /usr/bin/oawk (already AT&T said it's old and provided /usr/bin/nawk).
The previous awk code rewritten for oawk:
Code:
awk '(FILENAME=="-") {a[$0]++; next} (a[$0]==0)' - <a.out b.out >c.out

Last edited by MadeInGermany; 09-09-2013 at 06:41 AM..
This User Gave Thanks to MadeInGermany For This Post:
Mr_47
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to remove exisiting file content from a file and have to append new file content?

hi all, i had the below script x=`cat input.txt |wc -1` awk 'NR>1 && NR<'$x' ' input.txt > output.txt by using above script i am able to remove the head and tail part from the input file and able to append the output to the output.txt but if i run it for second time the output is... (2 Replies)
Discussion started by: hemanthsaikumar
2 Replies

2. Shell Programming and Scripting

Remove the duplicate content in a file

Here is the contents of test.txt Dependencies Resolved Changes in packages about to be updated: ChangeLog for: 1:perl-Archive-Extract-0.38-131.el6_4.x86_64, - Resolves: #915692 - CVE-2013-1667 (DoS in rehashing code) Dependencies Resolved Changes in packages about to be updated: ... (5 Replies)
Discussion started by: ashokvpp
5 Replies

3. Shell Programming and Scripting

Shell script to monitor new file in a directory and mail the file content

Hi I am looking for a help in designing a bash script on linux which can do below:- 1) Look in a specific directory for any new files 2) Mail the content of the new file Appreciate any help Regards Neha (5 Replies)
Discussion started by: neha0785
5 Replies

4. Shell Programming and Scripting

Remove some content from a verilog file

Hi, I have a verilog file which looks like module xyz (x, y, z, a, b, c); input x; input y; input z; output a; output b; output c; initial begin ... end always ... ... endmodule (7 Replies)
Discussion started by: matbrow
7 Replies

5. Shell Programming and Scripting

Remove the file content based on the Header of the file

Hi All, I want to remove the content based on the header information . Please find the example below. File1.txt Name|Last|First|Location|DepId|Depname|DepLoc naga|rr|tion|hyd|1|wer|opr Nava|ra|tin|gen|2|wera|opra I have to search for the DepId and remove the data from the... (5 Replies)
Discussion started by: i150371485
5 Replies

6. Shell Programming and Scripting

Shell script to remove some content in a file

How can I remove all data that contain domain e.g zzgh@something.com, sdd@something.com.my and gg@something.my in one file? so that i only have data without the domain in the file. Here is the file structure "test.out" more test.out 1 zzztop@b.com 1 zzzulll 1 zzzullll@s.com.my ... (4 Replies)
Discussion started by: Mr_47
4 Replies

7. Shell Programming and Scripting

Help with remove duplicate content and only keep the first content detail

Input data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_21 SSA data_19 TYUEC data_14 TYUE data_15 SSA data_32 PEOCV . . Desired Output data_10 SSA data_2 TYUE data_3 PEOCV data_6 SSAT data_19 TYUEC (9 Replies)
Discussion started by: patrick87
9 Replies

8. Shell Programming and Scripting

Remove specific content in a file

Hi, I have a file called fl_list consists of files i have to archive. I want to create a exception parm called except_parm, so if it finds the directory it will not archive these files and remove from fl_list. $ cat fl_list /apps/dev/ihub/ready/IA003B/IA003B_Deal_Header_yyyymmdd_hhmmss.txt... (1 Reply)
Discussion started by: k9cheung
1 Replies

9. Shell Programming and Scripting

remove space from file content

i am a bit new to shell scripting i have a file containing xxxx xx xx but i want to output the content as xxxxxxxx. thus removing the space. any idea how i can do this (4 Replies)
Discussion started by: blackzinga
4 Replies

10. Shell Programming and Scripting

Remove content from file

hey all, I have a file with records in following format 8-29-2006 13:01:45|ABC|45 8-29-2006 14:23:12|DEF|21 8-30-2006 00:04:57|ABC|34 I want to remove all yesterday records. Can anyone show me how? Thanks! (10 Replies)
Discussion started by: mpang_
10 Replies
Login or Register to Ask a Question