Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Merge 4 bim files by keeping only the overlapping variants (unique rs values ) Post 303042746 by nezabudka on Sunday 5th of January 2020 03:49:45 PM
Old 01-05-2020
Maybe these are file names x2014 x2016 x2017 x2018 ?
Five hundred thousand rows for an array in AWK is a drop in the bucket
Code:
awk '!T[$2]++' x201[4678] > ONE_FILE

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Need to find only unique values for a given tag across the files

Need to find only unique values for a given tag across the files: For eg: Test1: <Tag1>aaa</Tag1> <Tag2>bbb</Tag2> <Tag3>ccc</Tag3> Test2: <Tag1>aaa</Tag1> <Tag2>ddd</Tag2> <Tag3>eee</Tag3> Test3: <Tag1>aaa</Tag1> <Tag2>ddd</Tag2> <Tag3>eee</Tag3> Test4: (8 Replies)
Discussion started by: sudheshnaiyer
8 Replies

2. Shell Programming and Scripting

comparing 2 text files to get unique values??

Hi all, I have got a problem while comparing 2 text files and the result should contains the unique values(Non repeatable). For eg: file1.txt 1 2 3 4 file2.txt 2 3 So after comaping the above 2 files I should get only 1 and 4 as the output. Pls help me out. (7 Replies)
Discussion started by: smarty86
7 Replies

3. Shell Programming and Scripting

merge files with same row values

Hi everyone, I'm just wondering how could I using awk language merge two files by comparison of one their row. I mean, I have one file like this: file#1: 21/07/2009 11:45:00 100.0000000 27.2727280 21/07/2009 11:50:00 75.9856644 25.2492676 21/07/2009 11:55:00 51.9713287 23.2258072... (4 Replies)
Discussion started by: tonet
4 Replies

4. Shell Programming and Scripting

sort split merge -u unique

Hi, this is about sorting a very large file (like 10 gb) to keep lines with unique entries across SOME of the columns. The line originally looked like this: sort -u -k2,2 -k3,3n -k4,4n -k5,5n -k6,6n file_unsorted > file_sorted please note the -u flag. The problem is that this single... (4 Replies)
Discussion started by: jbr950
4 Replies

5. UNIX for Dummies Questions & Answers

How to count specific columns and merge with unique ones?

Hi. I am not sure the title gives an optimal description of what I want to do. I have several text files that contain data in many columns. All the files are organized the same way, but the data in the columns might differ. I want to count the number of times data occur in specific columns,... (0 Replies)
Discussion started by: JamesT
0 Replies

6. UNIX for Dummies Questions & Answers

Merge two files with non-overlapping identities

Hi All, I wish to merge two files: file1: with header rsSNP-ID Chromosome Chr-Pos rs171 1 175261679 rs242 1 20869461 rs538 1 6160958 file2: without header disease:AAT deficiency:M0525101 rs1243168 20109307 1 disease:AAT deficiency:M0525101 rs4900229 20109307 1... (3 Replies)
Discussion started by: luoruicd
3 Replies

7. Shell Programming and Scripting

Compare multiple files, identify common records and combine unique values into one file

Good morning all, I have a problem that is one step beyond a standard awk compare. I would like to compare three files which have several thousand records against a fourth file. All of them have a value in each row that is identical, and one value in each of those rows which may be duplicated... (1 Reply)
Discussion started by: nashton
1 Replies

8. Shell Programming and Scripting

Identify the overlapping and non overlapping regions

file1 chr pos1 pos2 pos3 pos4 1)chr1 1000 2000 3000 4000 2)chr1 1380 1480 6800 7800 3)chr1 6700 7700 1200 2200 4)chr2 8500 9500 5670 6670 file2 chr pos1 pos2 pos3 pos4 1)chr2 8500 9500 5000 6000 2)chr1 6700 7700 1200 2200 3)chr1 1380 1480 6700 7700 4)chr1 1000 2000 4900 5900 I... (2 Replies)
Discussion started by: data_miner
2 Replies

9. Shell Programming and Scripting

Count Unique values from multiple lists of files

Looking for a little help here. I have 1000's of text files within a multiple folders. YYYY/ /MM /1000's Files Eg. 2014/01/1000 files 2014/02/1237 files 2014/03/1400 files There are folders for each year and each month, and within each monthly folder there are... (4 Replies)
Discussion started by: whegra
4 Replies

10. Shell Programming and Scripting

How to merge two files with unique values matching.?

I have one script as below: #!/bin/ksh Outputfile1="/home/OutputFile1.xls" Outputfile2="/home/OutputFile2.xls" InputFile1="/home/InputFile1.sql" InputFile2="/home/InputFile2.sql" echo "Select hobby, class, subject, sports, rollNumber from Student_Table" >> InputFile1 echo "Select rollNumber... (3 Replies)
Discussion started by: Sharma331
3 Replies
Hash::Merge::Simple(3pm)				User Contributed Perl Documentation				  Hash::Merge::Simple(3pm)

NAME
Hash::Merge::Simple - Recursively merge two or more hashes, simply VERSION
version 0.051 SYNOPSIS
use Hash::Merge::Simple qw/ merge /; my $a = { a => 1 }; my $b = { a => 100, b => 2}; # Merge with righthand hash taking precedence my $c = merge $a, $b; # $c is { a => 100, b => 2 } ... Note: a => 100 has overridden => 1 # Also, merge will take care to recursively merge any subordinate hashes found my $a = { a => 1, c => 3, d => { i => 2 }, r => {} }; my $b = { b => 2, a => 100, d => { l => 4 } }; my $c = merge $a, $b; # $c is { a => 100, b => 2, c => 3, d => { i => 2, l => 4 }, r => {} } # You can also merge more than two hashes at the same time # The precedence increases from left to right (the rightmost has the most precedence) my $everything = merge $this, $that, $mine, $yours, $kitchen_sink, ...; DESCRIPTION
Hash::Merge::Simple will recursively merge two or more hashes and return the result as a new hash reference. The merge function will descend and merge hashes that exist under the same node in both the left and right hash, but doesn't attempt to combine arrays, objects, scalars, or anything else. The rightmost hash also takes precedence, replacing whatever was in the left hash if a conflict occurs. This code was pretty much taken straight from Catalyst::Utils, and modified to handle more than 2 hashes at the same time. USAGE
Hash::Merge::Simple->merge( <hash1>, <hash2>, <hash3>, ..., <hashN> ) Hash::Merge::Simple::merge( <hash1>, <hash2>, <hash3>, ..., <hashN> ) Merge <hash1> through <hashN>, with the nth-most (rightmost) hash taking precedence. Returns a new hash reference representing the merge. NOTE: The code does not currently check for cycles, so infinite loops are possible: my $a = {}; $a->{b} = $a; merge $a, $a; NOTE: If you want to avoid giving/receiving side effects with the merged result, use "clone_merge" or "dclone_merge" An example of this problem (thanks Uri): my $left = { a => { b => 2 } } ; my $right = { c => 4 } ; my $result = merge( $left, $right ) ; $left->{a}{b} = 3 ; $left->{a}{d} = 5 ; # $result->{a}{b} == 3 ! # $result->{a}{d} == 5 ! Hash::Merge::Simple->clone_merge( <hash1>, <hash2>, <hash3>, ..., <hashN> ) Hash::Merge::Simple::clone_merge( <hash1>, <hash2>, <hash3>, ..., <hashN> ) Perform a merge, clone the merge, and return the result This is useful in cases where you need to ensure that the result can be tweaked without fear of giving/receiving any side effects This method will use Clone to do the cloning Hash::Merge::Simple->dclone_merge( <hash1>, <hash2>, <hash3>, ..., <hashN> ) Hash::Merge::Simple::dclone_merge( <hash1>, <hash2>, <hash3>, ..., <hashN> ) Perform a merge, clone the merge, and return the result This is useful in cases where you need to ensure that the result can be tweaked without fear of giving/receiving any side effects This method will use Storable (dclone) to do the cloning SEE ALSO
Hash::Merge Catalyst::Utils Clone Storable ACKNOWLEDGEMENTS
This code was pretty much taken directly from Catalyst::Utils: Sebastian Riedel "sri@cpan.org" Yuval Kogman "nothingmuch@woobling.org" AUTHOR
Robert Krimen <robertkrimen@gmail.com> COPYRIGHT AND LICENSE
This software is copyright (c) 2010 by Robert Krimen. This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself. perl v5.10.1 2010-12-07 Hash::Merge::Simple(3pm)
All times are GMT -4. The time now is 01:58 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy