Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Compare multiple files, identify common records and combine unique values into one file
Top Forums Shell Programming and Scripting Compare multiple files, identify common records and combine unique values into one file Post 302809497 by nashton on Monday 20th of May 2013 02:47:50 AM
Old 05-20-2013
Compare multiple files, identify common records and combine unique values into one file
Good morning all,

I have a problem that is one step beyond a standard awk compare.

I would like to compare three files which have several thousand records against a fourth file. All of them have a value in each row that is identical, and one value in each of those rows which may be duplicated in the tree files vis a vis the fourth

What I want to see is:

1) The number of records that is unique in each of three (not in any of the others),
2) The number of records that is not unique in each of three,
3) the number of records in the fourth that is NOT in any of the other three;
4) An output file with the full row of each unique record across all the files

These are all text files.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Combine multiple files with common string into one new file.

I need to compile a large amount of data with a common string from individual text files throughout many directories. An example data file is below. I want to search for the following string, "cc_sectors_1" and combine all the data from each file which contains this string, into one new... (2 Replies)
Discussion started by: GradStudent2010
2 Replies

2. Shell Programming and Scripting

Compare multiple files and print unique lines

Hi friends, I have multiple files. For now, let's say I have two of the following style cat 1.txt cat 2.txt output.txt Please note that my files are not sorted and in the output file I need another extra column that says the file from which it is coming. I have more than 100... (19 Replies)
Discussion started by: jacobs.smith
19 Replies

3. Shell Programming and Scripting

Compare values in two files. For matching rows print corresponding values from File 1 in File2.

- I have two files (File 1 and File 2) and the contents of the files are mentioned below. - I am trying to compare the values of Column1 of File1 with Column1 of File2. If a match is found, print the corresponding value from Column2 of File1 in Column5 of File2. - I tried to modify and use... (10 Replies)
Discussion started by: Santoshbn
10 Replies

4. Shell Programming and Scripting

Parsing common values across multiple files

Hi All, I have multiple (5+) text files with single columns and I would like to grep the common values across all the text files and parse it to a new file. All the values are numerical. Please let me know how to do it using awk. (6 Replies)
Discussion started by: Lucky Ali
6 Replies

5. Shell Programming and Scripting

Combine multiple unique lines from event log text file into one line, use PERL or AWK?

I can't decide if I should use AWK or PERL after pouring over these forums for hours today I decided I'd post something and see if I couldn't get some advice. I've got a text file full of hundreds of events in this format: Record Number : 1 Records in Seq : ... (3 Replies)
Discussion started by: Mayday22
3 Replies

6. Shell Programming and Scripting

Compare multiple files, and extract items that are common to ALL files only

I have this code awk 'NR==FNR{a=$1;next} a' file1 file2 which does what I need it to do, but for only two files. I want to make it so that I can have multiple files (for example 30) and the code will return only the items that are in every single one of those files and ignore the ones... (7 Replies)
Discussion started by: castrojc
7 Replies

7. Shell Programming and Scripting

Compare columns of multiple files and print those unique string from File1 in an output file.

Hi, I have multiple files that each contain one column of strings: File1: 123abc 456def 789ghi File2: 123abc 456def 891jkl File3: 234mno 123abc 456def In total I have 25 of these type of file. (5 Replies)
Discussion started by: owwow14
5 Replies

8. Shell Programming and Scripting

Get both common and missing values from multiple files

Hi, I have 5 files with two columns. I need to merge all the 5 files based on column 1. If any of them are missing then corresponding 2nd column should be substituted by missing value. I know hoe to do this for 2 files. but how can I implement for 5 files. I tried this based on 5 files but it... (2 Replies)
Discussion started by: Diya123
2 Replies

9. Shell Programming and Scripting

Count Unique values from multiple lists of files

Looking for a little help here. I have 1000's of text files within a multiple folders. YYYY/ /MM /1000's Files Eg. 2014/01/1000 files 2014/02/1237 files 2014/03/1400 files There are folders for each year and each month, and within each monthly folder there are... (4 Replies)
Discussion started by: whegra
4 Replies

10. Shell Programming and Scripting

How to identify varying unique fields values from a text file in UNIX?

Hi, I have a huge unsorted text file. We wanted to identify the unique field values in a line and consider those fields as a primary key for a table in upstream system. Basically, the process or script should fetch the values from each line that are unique compared to the rest of the lines in... (13 Replies)
Discussion started by: manikandan23
13 Replies

LEARN ABOUT DEBIAN

struct::compare

Compare(3pm)						User Contributed Perl Documentation					      Compare(3pm)

NAME

       Struct::Compare - Recursive diff for perl structures.

SYNOPSIS

	   use Struct::Compare;
	   my $is_different = compare($ref1, $ref2);

DESCRIPTION

       Compares two values of any type and structure and returns true if they are the same. It does a deep comparison of the structures, so a hash
       of a hash of a whatever will be compared correctly.

       This is especially useful for writing unit tests for your modules!

PUBLIC FUNCTIONS

       o   $bool = compare($var1, $var2)

	   Recursively compares $var1 to $var2, returning false if either structure is different than the other at any point. If both are
	   undefined, it returns true as well, because that is considered equal.

BUGS
/NEEDED ENHANCEMENTS
       o   blessed references

	   compare currently does not deal with blessed references. I need to look into how to deal with this.

LICENSE

       (The MIT License)

       Copyright (c) 2001 Ryan Davis, Zen Spider Software

       Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the
       "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish,
       distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to
       the following conditions:

       The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

       THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
       MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
       CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
       SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

AUTHOR

       Ryan Davis <ryand-cmp@zenspider.com> Zen Spider Software <http://www.zenspider.com/ZSS/>

perl v5.12.4							    2011-07-04							      Compare(3pm)
All times are GMT -4. The time now is 04:19 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy