Sponsored Content
Top Forums Shell Programming and Scripting Merge files without duplicates Post 302793431 by pravfraz on Friday 12th of April 2013 10:50:12 AM
Old 04-12-2013
Merge files without duplicates

Hi all,


In a directory of many files, I need to merge only files which do not have identical lines and also the resulatant merge file should not be more than 50000 lines. Basically I need to cover up all text files in that directory and turn them to Merge files.txt with 50000 lines each

Ex:
Code:
a.txt   b.txt c.txt  
A         C      F
B         D      G
C         E      H

In the above case a.txt and c.txt alone should merge , as a.txt and b.txt has value 'C' in common.
Can anyone help quickly?

Last edited by Scrutinizer; 04-12-2013 at 05:48 PM.. Reason: code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Find duplicates from multuple files with 2 diff types of files

I need to compare 2 diff type of files and find out the duplicate after comparing each types of files: Type 1 file name is like: file1.abc (the extension abc could any 3 characters but I can narrow it down or hardcode for 10/15 combinations). The other file is file1.bcd01abc (the extension... (2 Replies)
Discussion started by: ricky007
2 Replies

2. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies

3. Shell Programming and Scripting

Merge Two Tables with duplicates in first table

Hi.. File 1: 1 aa rep 1 dd rep 1 kk rep 2 bb sad 2 ss sad 3 ee dam File 2 1 apple fruit 2 mango tree 3 lilly flower output: 1 aaple fruit aa,dd,kk rep (7 Replies)
Discussion started by: empyrean
7 Replies

4. Shell Programming and Scripting

how to check the duplicates files contents

Hi all , am using unix aix .. how to check the duplicates files contents and moving to someother folders ex .. i have files called qqq cat qqq 1 2 3 i have another file qqq02 cat qqq02 1 2 3 here qqq02 has having same contents as qqq file.. i need to check the duplicates files... (1 Reply)
Discussion started by: Venkatesh1
1 Replies

5. UNIX for Advanced & Expert Users

How to find duplicates contents in a files by comparing other files?

Hi Guys , we have one directory ...in that directory all files will be set on each day.. files must have header ,contents ,footer.. i wants to compare the header,contents,footer ..if its same means display an error message as 'files contents same' (7 Replies)
Discussion started by: Venkatesh1
7 Replies

6. UNIX for Dummies Questions & Answers

Remove duplicates lines in a files

I have a file called FILE cat FILE 11/11/2012 11/11/2012 12/11/2012 15/11/2012 need to remove the duplicates dates ( ie 11/11/2012 is present two times i need remove one duplicates date ) Need outputs like this 11/11/2012 12/11/2012 15/11/2012 I have tried using awk... (8 Replies)
Discussion started by: Venkatesh1
8 Replies

7. Shell Programming and Scripting

Checking in a directory how many files are present and basing on that merge all the files

Hi, My requirement is,there is a directory location like: :camp/current/ In this location there can be different flat files that are generated in a single day with same header and the data will be different, differentiated by timestamp, so i need to verify how many files are generated... (10 Replies)
Discussion started by: srikanth_sagi
10 Replies

8. Shell Programming and Scripting

Merging few files into one, duplicates are removed

Hello, I have few file such as below: abc.txt def.txt ghi.txt jkl.txt n.txt I would like to merge all these files together into one file. At the same time, any duplicates will be removed. (5 Replies)
Discussion started by: alegnagrp
5 Replies

9. Shell Programming and Scripting

Find duplicates in column 1 and merge their lines (awk?)

Hi, I have a file (sorted by sort) with 8 tab delimited columns. The first column contains duplicated fields and I need to merge all these identical lines. My input file: comp100002 aaa bbb ccc ddd eee fff ggg comp100003 aba aba aba aba aba aba aba comp100003 fff fff fff fff fff fff fff... (5 Replies)
Discussion started by: falcox
5 Replies

10. Shell Programming and Scripting

Help with merge and remove duplicates

Hi all, I need some help to remove duplicates from a file before merging. I have got 2 files: file1 has data in format 4300 23456 4301 2357 the 4 byte values on the right hand side is uniq, and are not repeated anywhere in the file file 2 has data in same format but is not in... (10 Replies)
Discussion started by: roy121
10 Replies
bntext(5)							File Formats Manual							 bntext(5)

NAME
bnmotd.txt, bnnews.txt bnissue.txt - messages for the Unix Battle.net daemon DESCRIPTION
The file bnmotd.txt contains text displayed by bnetd(1), when users first log into the server. The file bnnews.txt contains text displayed when the user uses the /news chat command. The files consist of raw text with printf-style formatting escapes. Each line of a file can contain a type formatter from the following list: %B Use the broadcast attribute (???). %C Execute the line as if the user entered it as a command. %E Use the error attribute (red). %I Use the info attribute (yellow). This is the same was %W. %M Normal chat message (white). This will appear as if the user said it. %T Emote chat message (???). This will appear as if the user said it. %W Use the warning attribute (yellow). This is the same was %I. Within a line, any of the following format formatters may be used: %% Expand to a literal percent sign (%). %a Expand to the number of accounts on the server. %c Expand to the number of channels on the server. This includes all permanent and current temporary channels. %g Expand to the number of games on the server. This includes both public and private (passworded) games. %h Expand to the hostname of the server (as returned by gethostname(2)). %i Expand to this user's account ID number, formatted with a leading pound (#) sign and leading zeros. %l Expand to this user's current chat name which is usually the same %r Expand to the IP of the remote machine (the client). %t Expand to four character client tag. %u Expand to the number of users logged into the server. %v Expand to the version number of the server. SEE ALSO
bnetd(1) AUTHOR
Ross Combs (ross@bnetd.org) 2 August, 2001 bntext(5)
All times are GMT -4. The time now is 08:53 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy