Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to avoid duplication within 2 files?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to avoid duplication within 2 files?
# 1  
Old 10-15-2008
How to avoid duplication within 2 files?
Hi all,

Actually 2 files are there - file1, file2.
file1 contains --->
london
mosco
america
russia
mosco

file2 contains -->

europe
india
japan
mosco
england
london

Question is I want to print all the city names without duplication cities in those files.
for ex,
london
mosco
america
russia
europe
india
japan
england

plz, tell me which cmd I can use?

Thnx,
Balan Smilie
# 2  
Old 10-15-2008

If you don't mind the order being re-arranged:

Code:
sort -u file1 file2

If you want to preserve the order:

Code:
awk '!x[$0]++' file1 file2

# 3  
Old 10-15-2008
duplication
........and without duplication cities in those files.
Code:
sort -u file1 file2 | uniq

Last edited by ./hari.sh; 10-15-2008 at 01:20 PM.. Reason: added complete command
# 4  
Old 10-15-2008
Thanx guys...
# 5  
Old 10-15-2008
Quote:
Originally Posted by ./hari.sh
........and without duplication cities in those files.
Code:
sort -u file1 file2 | uniq

Whats the need for uniq ? Thats what -u does
# 6  
Old 10-16-2008
ok.....
# 7  
Old 06-06-2009
Quote:
Originally Posted by cfajohnson
If you don't mind the order being re-arranged:

Code:
sort -u file1 file2

If you want to preserve the order:

Code:
awk '!x[$0]++' file1 file2


The first one is okay for me ..but when I am trying the second one I am getting

x[$0]++': Event not found .

Could it be because of permission restrictions on me ?>
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. AIX

How to avoid errors when moving files in a bff?

I am building a bff using mkinstallp. My template file is : Package Name: svr_exForum Package VRMF: 7.2 Update: N Fileset Fileset Name: svr_exForum.rte Fileset VRMF: 7.2 USRLIBLPPFiles Pre-installation Script: /lppdir/lpp/exForum/F_pre_i ... (2 Replies)
Discussion started by: kevinl
2 Replies

2. Linux

De-Duplication Problem

Hi all, I download and install lessfs for deduplication, I copy files in /SharedFiles directory and lessfs work right and not store again copy files, but, when i delete all files in /SharedFiles , not return free space to total space, files not show in /SharedFiles , but not copy new files in... (3 Replies)
Discussion started by: saeedha
3 Replies

3. Shell Programming and Scripting

Avoid locking between two files:

Hi frnz, I have requirement as follows.. There are two Modules say A and B...n both have few set of files..we have one script which read these files and load into database(vertica) table..while running script smtimes i encounter an error like one table is getting loaded from A module file... (7 Replies)
Discussion started by: gnnsprapa
7 Replies

4. Ubuntu

Avoid typing long filenames in terminal, shortcut for selecting files?

You probably know the answer to this, because I know it exists. I have super long filenames with md5 hashes and I sucks to type the whole hash in the console. Because... just because :P What is the shortcut for selecting a file in the current directory? Like you get a sort of loop through the... (1 Reply)
Discussion started by: hellfire1
1 Replies

5. Shell Programming and Scripting

How to Avoid intermediate files when pipe does nt work

problem with piping one output to another.Would like to avoid the intermediate file creation.The piping does nt work on places where files have been created and goes in an endless loop. sed -e "s/^\.\///g" $LINE1| sed -e "s/_\(\)/kkk\1/g" > $file1 tr -s '_' ' ' < $file1| \ sort -n -k... (1 Reply)
Discussion started by: w020637
1 Replies

6. Shell Programming and Scripting

Avoid files being archived

hi all, i want to write a shell script which can automatically touch my all files within a folder in an interval of 90 days ...so that i can avoid them being archived. I don't want to manually touch the all files instead i want an automated shell script to do this. Thanks in advance, Om (3 Replies)
Discussion started by: koti
3 Replies

7. Programming

What is the proper way to combine C++ files (with g++) to avoid link (ld) errors?

Problem background: gcc v 4.1 2 .cpp files, 2 .h files Files: main.cpp a.cpp a.h b.h Organization: main.cpp includes a.h (because it calls a.cpp code) a.cpp includes a.h a.h includes b.h (because a class in a.h uses a b.h class) There is no inheritance between a.h or b.h or any of... (1 Reply)
Discussion started by: johnqsmith
1 Replies

8. Ubuntu

Avoid creating temporary files on editing a file in Ubuntu

Hi, My ubuntu flavor always create temporary files having filename followed by ~ on editing. For eg: if I am editing a file called "sip.c", automatically a temporary (bkup) file is getting created with the name "sip.c~". How to avoid this file creation? (7 Replies)
Discussion started by: royalibrahim
7 Replies

9. Windows & DOS: Issues & Discussions

File Duplication

hi all how to find the file duplication in a windows 2000 server as usual replies are sincerely appreciated. thanks raguram R (3 Replies)
Discussion started by: raguramtgr
3 Replies
Login or Register to Ask a Question