Sponsored Content
Top Forums Shell Programming and Scripting Help with merge data based on similarity Post 302582954 by Skrynesaver on Monday 19th of December 2011 06:33:55 AM
Old 12-19-2011
As you're using perl already, why not read the reference file into a hash keyed on the non-numeric field in each record and then use that to create the new records as you read the input file?
Code:
$ perl -Mstrict -e '
my %index;
open (my $ref,"<","reference.dat");
while (<$ref>){
   chomp;
   my $record=$_;
   my @record=split(/\s+/,$record);
   for my $field (@record){
      if ($field !~ /^\d+$/){
         $index{$field} = $record;
      }
   }
}
close $ref;
open(my $in, "<","input.dat");
while(<$in>){
   chomp;
   my @fields=split /\s+/,$_;
   print "$_ $index{$fields[1]}\n";
}'

This User Gave Thanks to Skrynesaver For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merge files based on key

Hi Friends, Can any one help me with merging these file based on two columns : File1: A|123|99|SAMS B|456|95|GEORGE D|789|85|HOVARD File2: S|123|99|NANcY|6357 S|123|99|GREGRO|83748 A|456|95|HARRY|827|somers S|456|95|ANTONY|546841|RUDOLPH|7263 B|456|95|SMITH|827|BOISE STATE|834... (3 Replies)
Discussion started by: sbasetty
3 Replies

2. Shell Programming and Scripting

merge based on common, awk help

All, $ cat x.txt z 11 az x 12 ax y 13 ay $ cat y.txt ay TT ax NN Output required: y 13 ay TT x 12 ax NN (3 Replies)
Discussion started by: jkl_jkl
3 Replies

3. Shell Programming and Scripting

Merge Two Files based on First column

Hi, I need to join two files based on first column of both files.If first column of first file matches with the first column of second file, then the lines should be merged together and go for next line to check. It is something like: File one: 110001 abc efd 110002 fgh dfg 110003 ... (10 Replies)
Discussion started by: apjneeraj
10 Replies

4. Shell Programming and Scripting

Extract data based on match against one column data from a long list data

My input file: data_5 Ali 422 2.00E-45 102/253 140/253 24 data_3 Abu 202 60.00E-45 12/23 140/23 28 data_1 Ahmad 256 7.00E-45 120/235 140/235 22 data_4 Aman 365 8.00E-45 15/65 140/65 20 data_10 Jones 869 9.00E-45 65/253 140/253 18... (12 Replies)
Discussion started by: patrick87
12 Replies

5. Shell Programming and Scripting

Merge two file data together based on specific pattern match

My input: File_1: 2000_t g1110.b1 abb.1 2001_t g1111.b1 abb.2 abb.2 g1112.b1 abb.3 2002_t . . File_2: 2000_t Ali england 135 abb.1 Zoe british 150 2001_t Ali england 305 g1111.b1 Lucy russia 126 (6 Replies)
Discussion started by: patrick87
6 Replies

6. Shell Programming and Scripting

Help with sort list of file based on similarity

Input file (long list of input file): s_1_1_AABCD.txt s_1_1_ABADA.txt s_1_1_DSCBA.txt s_1_1_DSCCA.txt s_1_1_EATTG.txt s_1_1_FADSD.txt s_1_1_TGACC.txt s_1_1_TTAGG.txt s_1_2_AABCD.txt s_1_2_ABADA.txt s_1_2_DSCBA.txt s_1_2_DSCCA.txt s_1_2_EATTG.txt s_1_2_FADSD.txt ... (1 Reply)
Discussion started by: perl_beginner
1 Replies

7. Shell Programming and Scripting

Need to merge lines based on pattern

Hi, I have a requirement to merge multiple lines based on search pattern. The search criteria is : it will search for CONSTRAINT and when it found CONSTRAINT, it will merge all lines to 1 line till it founds blank line. For Example: CREATE TABLE "AMS_DISTRIBUTOR_XREF" ( "SOURCE"... (5 Replies)
Discussion started by: satyaatcgi
5 Replies

8. Shell Programming and Scripting

Merge files based on columns

011111123444 1234 1 20000 011111123444 1235 1 30000 011111123446 1234 3 40000 011111123447 1234 4 50000 011111123448 1234 3 50000 File2: 011111123444,Rsttponrfgtrgtrkrfrgtrgrer 011111123446,Rsttponrfgtrgtr 011111123447,Rsttponrfgtrguii 011111123448,Rsttponrfgtrgtjiiu I have 2 files... (4 Replies)
Discussion started by: vinus
4 Replies

9. Shell Programming and Scripting

Merge lines based on match

I am trying to merge two lines to one based on some matching condition. The file is as follows: Matches filter: 'request ', timestamp, <HTTPFlow request=<GET: Matches filter: 'request ', timestamp, <HTTPFlow request=<GET: Matches filter: 'request ', timestamp, <HTTPFlow ... (8 Replies)
Discussion started by: jamie_123
8 Replies

10. Shell Programming and Scripting

Merge row based on replicates ID

Dear All, I was wondering if you may help me with an issue. I would like to merge row based on column 1. input file: b1 ggg b2 fff NA NA hhh NA NA NA NA NA a1 xxx a2 yyy NA NA zzz NA NA NA NA NA a1 xxx NA NA a3 ttt NA ggg NA NA NA NA output file: b1 ggg b2 fff NA NA hhh NA NA NA NA NA... (5 Replies)
Discussion started by: giuliangiuseppe
5 Replies
DBS_UPDATE(1p)						User Contributed Perl Documentation					    DBS_UPDATE(1p)

NAME
dbs_update - Update SQL Databases DESCRIPTION
dbs_update is an utility to update SQL databases from text files. FORMAT OF THE TEXT FILES dbs_update assumes that each line of the input contains a data record and that the field within the records are separated by tabulators. You can tell dbs_update about the input format with the --format option. The first field of the data record is used as table specification. These consists of the table name and optionally the index of starting column, separated by a dot. Alternatively dbs_update can read the column names from the first line of input (see the -h/--headline option). These can even be aliases for the real column names (see the -m/--map option). COMMAND LINE PARAMETERS
Required command line parameters are the DBI driver ("Pg" for Postgres or "mysql" for MySQL) and the database name. The third parameter is optionally and specifies the database user and/or the host where the database resides ("racke", "racke@linuxia.de" or "@linuxia.de"). OPTIONS
--cleanse Removes all records which remain unaffected from the update process. The same result as deleting all records from the table first and then running dbs_update, but the table is not empty in the meantime. -c COLUMN,COLUMN,..., --columns=COLUMN,COLUMN,... Update only the table columns given by the COLUMN parameters. To exclude columns from the update prepend "!" or "^" to the parameters. --rows=ROW,ROW,... Update only the input rows given by the ROW parameters. The first row is 1 where headlines doesn't count. To exclude rows from the update prepend "!" or "^" to the parameters. -f FILE, --file=FILE Reads records from file FILE instead of from standard input. --format=FORMAT[SEPCHAR] Assumes FORMAT as format for the input. Only CSV can be specified for now, default is TAB. The default field separator for CSV is a comma, you may change this by appending the separator to the format. -h, --headline Reads the column names from the first line of the input instead of dedicting them from the database layout. Requires the -t/--table option. -k COUNT, -k KEY,KEY,..., --keys=COUNT, --keys=KEY,KEY,... Specifies the keys for the table(s) either as the number of columns used as keys or by specifying them explicitly as comma separated argu- ments to the option. This is used for the detection of existing records. -m ALIASDEF, --map=ALIASDEF Maps the names found in the first line of input to the actual column names in the database. The alias and the column name are separated with "=" signs and the different entries are separated by ";" signs, e.g. "Art-No.=code;Short Description=shortdescr'". --map-filter=FILTER Applies a filter to the column names read from the input file. Currently there is only the "lc" filter available. --match-sql=FIELD:{STATEMENT} Updates only records where the value of the column FIELD is in the result set of the SQL statement STATEMENT, e.g. "category:{select dis- tinct name from categories}". -o, --update-only Updates existing database entries only, stops if it detects new ones. -r ROUTINE, --routine=ROUTINE Applies ROUTINE to any data record. ROUTINE must be a subroutine. dbs_update passes the table name and a hash reference to this subrou- tine. The keys of the hash are the column names and the values are the corresponding field values. If the return value of ROUTINE is not a truth value, the data record will be skipped. "sub {my ($table, $valref) = @_; unless (defined $$valref{country} && $$valref{country} !~ /S/) { $$valref{country} = "Germany"; } 1; }" --skipbadlines Lines not matching the assumed format are ignored. Without this option, dbs_update simply stops. -t TABLE, --table=TABLE Uses TABLE as table name for all records instead of the first field name. AUTHOR
Stefan Hornburg (Racke), racke@linuxia.de SEE ALSO
perl(1), DBIx::Easy(3) perl v5.8.8 2007-02-01 DBS_UPDATE(1p)
All times are GMT -4. The time now is 05:46 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy