Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Merge 4 bim files by keeping only the overlapping variants (unique rs values ) Post 303042724 by fondan on Saturday 4th of January 2020 11:17:26 AM
Old 01-04-2020
Merge 4 bim files by keeping only the overlapping variants (unique rs values )

Dear community, I am facing a problem and I kindly ask your help:


I have 4 different data sets consisted from 3 different types of array.



On each file, column 1 is chromosome position, column 2 is SNP id etc... Lets say I have the following (bim) datasets:


x2014:
Code:
1       rs3094315       0       752566  G       A
1       rs3131972       0       752721  G       A

....more 550.000


x2016:
Code:
0       200610-10       0       0       G       A
0       200610-108      0       0       G       A

...


x2017
Code:
0       200610-10       0       0       G       A
0       200610-108      0       0       G       A

...



x2018:
Code:
0       200610-10       0       0       G       A
0       200610-108      0       0       G       A

.....more 550K rows




How can I merge all files together, without having any duplicate values based on the 2nd column (rs_id)?

Last edited by vbe; 01-04-2020 at 12:40 PM.. Reason: code tage please
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Need to find only unique values for a given tag across the files

Need to find only unique values for a given tag across the files: For eg: Test1: <Tag1>aaa</Tag1> <Tag2>bbb</Tag2> <Tag3>ccc</Tag3> Test2: <Tag1>aaa</Tag1> <Tag2>ddd</Tag2> <Tag3>eee</Tag3> Test3: <Tag1>aaa</Tag1> <Tag2>ddd</Tag2> <Tag3>eee</Tag3> Test4: (8 Replies)
Discussion started by: sudheshnaiyer
8 Replies

2. Shell Programming and Scripting

comparing 2 text files to get unique values??

Hi all, I have got a problem while comparing 2 text files and the result should contains the unique values(Non repeatable). For eg: file1.txt 1 2 3 4 file2.txt 2 3 So after comaping the above 2 files I should get only 1 and 4 as the output. Pls help me out. (7 Replies)
Discussion started by: smarty86
7 Replies

3. Shell Programming and Scripting

merge files with same row values

Hi everyone, I'm just wondering how could I using awk language merge two files by comparison of one their row. I mean, I have one file like this: file#1: 21/07/2009 11:45:00 100.0000000 27.2727280 21/07/2009 11:50:00 75.9856644 25.2492676 21/07/2009 11:55:00 51.9713287 23.2258072... (4 Replies)
Discussion started by: tonet
4 Replies

4. Shell Programming and Scripting

sort split merge -u unique

Hi, this is about sorting a very large file (like 10 gb) to keep lines with unique entries across SOME of the columns. The line originally looked like this: sort -u -k2,2 -k3,3n -k4,4n -k5,5n -k6,6n file_unsorted > file_sorted please note the -u flag. The problem is that this single... (4 Replies)
Discussion started by: jbr950
4 Replies

5. UNIX for Dummies Questions & Answers

How to count specific columns and merge with unique ones?

Hi. I am not sure the title gives an optimal description of what I want to do. I have several text files that contain data in many columns. All the files are organized the same way, but the data in the columns might differ. I want to count the number of times data occur in specific columns,... (0 Replies)
Discussion started by: JamesT
0 Replies

6. UNIX for Dummies Questions & Answers

Merge two files with non-overlapping identities

Hi All, I wish to merge two files: file1: with header rsSNP-ID Chromosome Chr-Pos rs171 1 175261679 rs242 1 20869461 rs538 1 6160958 file2: without header disease:AAT deficiency:M0525101 rs1243168 20109307 1 disease:AAT deficiency:M0525101 rs4900229 20109307 1... (3 Replies)
Discussion started by: luoruicd
3 Replies

7. Shell Programming and Scripting

Compare multiple files, identify common records and combine unique values into one file

Good morning all, I have a problem that is one step beyond a standard awk compare. I would like to compare three files which have several thousand records against a fourth file. All of them have a value in each row that is identical, and one value in each of those rows which may be duplicated... (1 Reply)
Discussion started by: nashton
1 Replies

8. Shell Programming and Scripting

Identify the overlapping and non overlapping regions

file1 chr pos1 pos2 pos3 pos4 1)chr1 1000 2000 3000 4000 2)chr1 1380 1480 6800 7800 3)chr1 6700 7700 1200 2200 4)chr2 8500 9500 5670 6670 file2 chr pos1 pos2 pos3 pos4 1)chr2 8500 9500 5000 6000 2)chr1 6700 7700 1200 2200 3)chr1 1380 1480 6700 7700 4)chr1 1000 2000 4900 5900 I... (2 Replies)
Discussion started by: data_miner
2 Replies

9. Shell Programming and Scripting

Count Unique values from multiple lists of files

Looking for a little help here. I have 1000's of text files within a multiple folders. YYYY/ /MM /1000's Files Eg. 2014/01/1000 files 2014/02/1237 files 2014/03/1400 files There are folders for each year and each month, and within each monthly folder there are... (4 Replies)
Discussion started by: whegra
4 Replies

10. Shell Programming and Scripting

How to merge two files with unique values matching.?

I have one script as below: #!/bin/ksh Outputfile1="/home/OutputFile1.xls" Outputfile2="/home/OutputFile2.xls" InputFile1="/home/InputFile1.sql" InputFile2="/home/InputFile2.sql" echo "Select hobby, class, subject, sports, rollNumber from Student_Table" >> InputFile1 echo "Select rollNumber... (3 Replies)
Discussion started by: Sharma331
3 Replies
Bio::PopGen::HtSNP(3pm) 				User Contributed Perl Documentation				   Bio::PopGen::HtSNP(3pm)

NAME
Bio::PopGen::HtSNP.pm- Select htSNP from a haplotype set SYNOPSIS
use Bio::PopGen::HtSNP; my $obj = Bio::PopGen::HtSNP->new($hap,$snp,$pop); DESCRIPTION
Select the minimal set of SNP that contains the full information about the haplotype without redundancies. Take as input the followin values: - the haplotype block (array of array). - the snp id (array). - family information and frequency (array of array). The final haplotype is generated in a numerical format and the SNP's sets can be retrieve from the module. considerations: - If you force to include a family with indetermination, the SNP's with indetermination will be removed from the analysis, so consider before to place your data set what do you really want to do. - If two families have the same information (identical haplotype), one of them will be removed and the removed files will be stored classify as removed. - Only are accepted for calculation A, C, G, T and - (as deletion) and their combinations. Any other value as n or ? will be considered as degenerations due to lack of information. RATIONALE On a haplotype set is expected that some of the SNP and their variations contribute in the same way to the haplotype. Eliminating redundancies will produce a minimal set of SNP's that can be used as input for a taging selection process. On the process SNP's with the same variation are clustered on the same group. The idea is that because the tagging haplotype process is exponential. All redundant information we could eliminate on the tagging process will help to find a quick result. CONSTRUCTORS my $obj = Bio::PopGen::HtSNP->new (-haplotype_block => @haplotype_patterns, -snp_ids => @snp_ids, -pattern_freq => @pattern_name_and_freq); where $hap, $snp and $pop are in the format: my $hap = [ 'acgt', 'agtc', 'cgtc' ]; # haplotype patterns' id my $snp = [qw/s1 s2 s3 s4/]; # snps' Id's my $pop = [ [qw/ uno 0.20/], [qw/ dos 0.20/], [qw/ tres 0.15/], ]; # haplotype_pattern_id Frequency OBJECT METHODS See Below for more detailed summaries. DETAILS
How the process is working with one example Let's begin with one general example of the code. Input haplotype: acgtcca-t cggtagtgc cccccgtgc cgctcgtgc The first thing to to is to split the haplotype into characters. a c g t c c a - t c g g t a g t g c c c c c c g t g c c g c t c g t g c Now we have to convert the haplotype to Upercase. This will produce the same SNP if we have input a or A. A C G T C C A - T C G G T A G T G C C C C C C G T G C C G C T C G T G C The program admit as values any combination of ACTG and - (deletions). The haplotype is converted to number, considering the first variation as zero and the alternate value as 1 (see expanded description below). 0 0 0 0 0 0 0 0 0 1 1 0 0 1 1 1 1 1 1 0 1 1 0 1 1 1 1 1 1 1 0 0 1 1 1 1 Once we have the haplotype converted to numbers we have to generate the snp type information for the haplotype. SNP code = SUM ( value * multiplicity ^ position ); where: SUM is the sum of the values for the SNP value is the SNP number code (0 [generally for the mayor allele], 1 [for the minor allele]. position is the position on the block. For this example the code is: 0 0 0 0 0 0 0 0 0 1 1 0 0 1 1 1 1 1 1 0 1 1 0 1 1 1 1 1 1 1 0 0 1 1 1 1 ------------------------------------------------------------------ 14 10 12 4 2 14 14 14 14 14 = 0*2^0 + 1*2^1 + 1*2^2 + 1*2^3 12 = 0*2^0 + 1*2^1 + 0*2^2 + 1*2^3 .... Once we have the families classify. We will take just the SNP's not redundant. 14 10 12 4 2 This information will be passed to the tag module is you want to tag the htSNP. Whatever it happens to one SNPs of a class will happen to a SNP of the same class. Therefore you don't need to scan redundancies Working with fuzzy data. This module is designed to work with fuzzy data. As the source of the haplotype is diverse. The program assume that some haplotypes can be generated using different values. If there is any indetermination (? or n) or any other degenerated value or invalid. The program will take away This SNP and will leave that for a further analysis. On a complex situation: a c g t ? c a c t a c g t ? c a - t c g ? t a g ? g c c a c t c g t g c c g c t c g t g c c g g t a g ? g c a c ? t ? c a c t On this haplotype everything is happening. We have a multialelic variance. We have indeterminations. We have deletions and we have even one SNP which is not a real SNP. The buiding process will be the same on this situation. Convert the haplotype to uppercase. A C G T ? C A C T A C G T ? C A - T C G ? T A G ? G C C A C T C G T G C C G C T C G T G C C G G T A G ? G C A C ? T ? C A C T All columns that present indeterminations will be removed from the analysis on this Step. hapotype after remove columns: A C T C C T A C T C - T C G T G G C C A T G G C C G T G G C C G T G G C A C T C C T All changes made on the haplotype matrix, will be also made on the SNP list. snp_id_1 snp_id_2 snp_id_4 snp_id_6 snp_id_8 snp_id_9 now the SNP that is not one SNP will be removed from the analysis. SNP with Id snp_id_4 (the one with all T's). because of the removing. Some of the families will become the same and will be clustered. A posteriori analysis will diference these families. but because of the indetermination can not be distinguish. A C C C T A C C - T C G G G C C A G G C C G G G C C G G G C A C C C T The result of the mergering will go like: A C C C T A C C - T C G G G C C A G G C Once again the changes made on the families and we merge the frequency (to be implemented) Before to convert the haplotype into numbers we consider how many variations we have on the set. On this case the variations are 3. The control code will use on this situation base three as mutiplicity 0 0 0 0 0 0 0 0 1 0 1 1 1 2 1 1 2 1 2 1 ----------------------------------- 36 63 36 75 36 And the minimal set for this combination is 0 0 0 0 0 1 1 1 2 1 2 2 NOTE: this second example is a remote example an on normal conditions. This conditions makes no sense, but as the haplotypes, can come from many sources we have to be ready for all kind of combinations. FEEDBACK
Mailing Lists User feedback is an integral part of the evolution of this and other Bioperl modules. Send your comments and suggestions preferably to the Bioperl mailing list. Your participation is much appreciated. bioperl-l@bioperl.org - General discussion http://bioperl.org/wiki/Mailing_lists - About the mailing lists Support Please direct usage questions or support issues to the mailing list: bioperl-l@bioperl.org rather than to the module maintainer directly. Many experienced and reponsive experts will be able look at the problem and quickly address it. Please include a thorough description of the problem with code and data examples if at all possible. Reporting Bugs Report bugs to the Bioperl bug tracking system to help us keep track of the bugs and their resolution. Bug reports can be submitted via the web: https://redmine.open-bio.org/projects/bioperl/ AUTHOR - Pedro M. Gomez-Fabre Email pgf18872-at-gsk-dot-com APPENDIX
The rest of the documentation details each of the object methods. Internal methods are usually preceded with a _ new Title : new Function: constructor of the class. Usage : $obj-> Bio::PopGen::HtSNP->new(-haplotype_block -snp_ids -pattern_freq) Returns : self hash Args : input haplotype (array of array) snp_ids (array) pop_freq (array of array) Status : public haplotype_block Title : haplotype_block Usage : my $haplotype_block = $HtSNP->haplotype_block(); Function: Get the haplotype block for a haplotype tagging selection Returns : reference of array Args : reference of array with haplotype pattern snp_ids Title : snp_ids Usage : my $snp_ids = $HtSNP->$snp_ids(); Function: Get the ids for a haplotype tagging selection Returns : reference of array Args : reference of array with SNP ids pattern_freq Title : pattern_freq Usage : my $pattern_freq = $HtSNP->pattern_freq(); Function: Get the pattern id and frequency for a haplotype tagging selection Returns : reference of array Args : reference of array with SNP ids _check_input Title : _check_input Usage : _check_input($self) Function: check for errors on the input Returns : self hash Args : self Status : internal _haplotype_length_error Title : _haplotype_length_error Usage : _haplotype_length_error($self) Function: check if the haplotype length is the same that the one on the SNP id list. If not break and exit Returns : self hash Args : self Status : internal _population_error Title : _population_error Usage : _population_error($self) Function: use input_block and pop_freq test if the number of elements match. If doesn't break and quit. Returns : self hash Args : self Status : internal _do_it Title : _do_it Usage : _do_it($self) Function: Process the input generating the results. Returns : self hash Args : self Status : internal input_block Title : input_block Usage : $obj->input_block() Function: returns input block Returns : reference to array of array Args : none Status : public hap_length Title : hap_length Usage : $obj->hap_length() Function: get numbers of SNP on the haplotype Returns : scalar Args : none Status : public pop_freq Title : pop_freq Usage : $obj->pop_freq() Function: returns population frequency Returns : reference to array Args : none Status : public deg_snp Title : deg_snp Usage : $obj->deg_snp() Function: returns snp_removes due to indetermination on their values Returns : reference to array Args : none Status : public snp_type Title : snp_type Usage : $obj->snp_type() Function: returns hash with SNP type Returns : reference to hash Args : none Status : public silent_snp Title : silent_snp Usage : $obj->silent_snp() Function: some SNP's are silent (not contibuting to the haplotype) and are not considering for this analysis Returns : reference to a array Args : none Status : public useful_snp Title : useful_snp Usage : $obj->useful_snp() Function: returns list of SNP's that are can be used as htSNP. Some of them can produce the same information. But this is not considered here. Returns : reference to a array Args : none Status : public ht_type Title : ht_type Usage : $obj->ht_type() Function: every useful SNP has a numeric code dependending of its value and position. For a better description see description of the module. Returns : reference to a array Args : none Status : public ht_set Title : ht_set Usage : $obj->ht_set() Function: returns the minimal haplotype in numerical format. This haplotype contains the maximal information about the haplotype variations but with no redundancies. It's the minimal set that describes the haplotype. Returns : reference to an array of arrays Args : none Status : public snp_type_code Title : snp_type_code Usage : $obj->snp_type_code() Function: returns the numeric code of the SNPs that need to be tagged that correspond to the SNP's considered in ht_set. Returns : reference to an array Args : none Status : public snp_and_code Title : snp_and_code Usage : $obj->snp_and_code() Function: Returns the full list of SNP's and the code associate to them. If the SNP belongs to the group useful_snp it keep this code. If the SNP is silent the code is 0. And if the SNP is degenerated the code is -1. Returns : reference to an array of array Args : none Status : public deg_pattern Title : deg_pattern Usage : $obj->deg_pattern() Function: Returns the a list with the degenerated haplotype. Sometimes due to degeneration some haplotypes looks the same and if we don't remove them it won't find any tag. Returns : reference to a hash of array Args : none Status : public split_hap Title : split_hap Usage : $obj->split_hap() Function: simple representation of the haplotype base by base Same information that input haplotype but base based. Returns : reference to an array of array Args : none Status : public _split_haplo Title : _split_haplo Usage : _split_haplo($self) Function: Take a haplotype and split it into bases Returns : self Args : none Status : internal _to_upper_case Title : _to_upper_case Usage : _to_upper_case() Function: make SNP or in-dels Upper case Returns : self Args : an AoA ref Status : private _remove_deg Title : _remove_deg Usage : _remove_deg() Function: when have a indetermination or strange value this SNP is removed Returns : haplotype family set and degeneration list Args : ref to an AoA and a ref to an array Status : internal _rem_silent_snp Title : _rem_silent_snp Usage : _rem_silent_snp() Function: there is the remote possibilty that one SNP won't be a real SNP on this situation we have to remove this SNP, otherwise the program won't find any tag Returns : nonthing Args : ref to an AoA and a ref to an array Status : internal _find_silent_snps Title : _find_silent_snps Usage : Function: list of snps that are not SNPs. All values for that SNPs on the set is the same one. Look stupid but can happend and if this happend you will not find any tag Returns : nothing Args : Status : _find_indet Title : _find_indet Usage : Function: find column (SNP) with invalid or degenerated values and store this values into the second parameter supplied. Returns : nothing Args : ref to AoA and ref to an array Status : internal _remove_col Title : _remove_col Usage : Function: remove columns contained on the second array from the first arr Returns : nothing Args : array of array reference and array reference Status : internal _remove_snp_id Title : _remove_snp_id Usage : Function: remove columns contained on the second array from the first arr Returns : nothing Args : array of array reference and array reference Status : internal _find_deg_pattern Title : _find_deg_pattern Usage : Function: create a list with the degenerated patterns Returns : @array Args : a ref to AoA Status : public _keep_these_patterns Title : _keep_these_patterns Usage : Function: this is a basic approach, take a LoL and a list, keep just the columns included on the list Returns : nothing Args : an AoA and an array Status : public compare_arrays Title : compare_arrays Usage : Function: take two arrays and compare their values Returns : 1 if the two values are the same 0 if the values are different Args : an AoA and an array Status : public _convert_to_numbers Title : _convert_to_numbers Usage : _convert_to_numbers() Function: tranform the haplotype into numbers. before to do that we have to consider the variation on the set. Returns : nonthing Args : ref to an AoA and a ref to an array Status : internal _snp_type_code Title : _snp_type_code Usage : Function: we have to create the snp type code for each version. The way the snp type is created is the following: we take the number value for every SNP and do the following calculation let be a SNP set as follow: 0 0 1 1 1 2 and multiplicity 3 on this case the situation is: sum (value * multiplicity ^ position) for each SNP 0 * 3 ^ 0 + 1 * 3 ^ 1 + 1 * 3 ^ 2 = 12 0 * 3 ^ 0 + 1 * 3 ^ 1 + 2 * 3 ^ 2 = 21 Returns : nothing Args : $self Status : private _alleles_number Title : _alleles_number Usage : Function: calculate the max number of alleles for a haplotype and if the number. For each SNP the number is stored and the max number of alleles for a SNP on the set is returned Returns : max number of alleles (a scalar storing a number) Args : ref to AoA Status : public _htSNP Title : _htSNP Usage : _htSNP() Function: calculate the minimal set that contains all information of the haplotype. Returns : nonthing Args : ref to an AoA and a ref to an array Status : internal _snp_and_code_summary Title : _snp_and_code_summary Usage : _snp_and_code_summary() Function: compile on a list all SNP and the code for each. This information can be also obtained combining snp_type and snp_type_code but on these results the information about the rest of SNP's are not compiled as table. 0 will be silent SNPs -1 are degenerated SNPs and the rest of positive values are the code for useful SNP Returns : nonthing Args : ref to an AoA and a ref to an array Status : internal perl v5.14.2 2012-03-02 Bio::PopGen::HtSNP(3pm)
All times are GMT -4. The time now is 08:59 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy