Sponsored Content
Top Forums UNIX for Advanced & Expert Users Solution for the Massive Comparison Operation Post 302428591 by jim mcnamara on Thursday 10th of June 2010 10:08:46 AM
Old 06-10-2010
We have a similar problem. Are you running diff? That would take forever.

Use something that has associative (hashed) arrays like awk or perl. Assuming you have several files, and an "old" one and a "new" one, that should take less than an hour.
You can search here for examples of both types of code on how to find file differences.

You need a lot of virtual memory, we run on a Solaris 9 sparc v440 with 32GB of memory.
We complete comparing 1.5GB (250K lines) files in about 5 minutes. We do them 12 at a time: 6 old vs 6 new.

I hope this is what you were asking....
This User Gave Thanks to jim mcnamara For This Post:
 

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Looking for AWK Solution for column comparison in a single file

- I am looking for different kind of awk solution which I don't think is mentioned before in these forums. Number of rows in the file are fixed Their are two columns in file1.txt 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 I am looking for 3... (1 Reply)
Discussion started by: softwarekids23
1 Replies

2. Shell Programming and Scripting

Column operation : cosne and sine operation

I have a txt file with several columns and i want to peform an operation on two columns and output it to a new txt file . file.txt 900.00000 1 1 1 500.00000 500.00000 100000.000 4 4 1.45257346E-07 899.10834 ... (4 Replies)
Discussion started by: shashi792
4 Replies

3. Homework & Coursework Questions

having massive trouble with 5 questions about egrep!

Hi all! I need help to do a few things with a .txt file using egrep. 1. I need to list all sequences where the vowel letters 'a, e, i, o, u' occur in that order, possibly separated by characters other than a, e, i, o, u; consisting of one or more complete words, possibly including punctuation. ... (1 Reply)
Discussion started by: dindiqotu
1 Replies

4. Shell Programming and Scripting

Massive Copy With Base Directory

I have a script that I am using to copy around 40-70k files to a NFS NAS. I have posted my code below in hopes that someone can help me figure out a faster way of achieving this. At the end of the script i need to have all the files in the list, copied over to the nas with source directory... (8 Replies)
Discussion started by: nitrobass24
8 Replies

5. Shell Programming and Scripting

Massive ftp

friends good morning FTP works perfect but I have a doubt if I want to transport 10 files, I imagine that I should not open 10 connections as I can transfer more than 1 file? ftp -n <<!EOF open caburga user ephfact ephfact cd /users/efactura/docONE/entrada bin mput EPH`date... (16 Replies)
Discussion started by: tricampeon81
16 Replies
DBIx::Class::Manual::QuickStart(3)			User Contributed Perl Documentation			DBIx::Class::Manual::QuickStart(3)

NAME
DBIx::Class::Manual::QuickStart - up and running with DBIC in 10 minutes DESCRIPTION
This document shows the minimum amount of code to make you a productive DBIC user. It requires you to be familiar with just the basics of database programming (what database tables, rows and columns are) and the basics of Perl object-oriented programming (calling methods on an object instance). It also helps if you already know a bit of SQL and how to connect to a database through DBI. Follow along with the example database shipping with this distribution, see directory examples/Schema. This database is also used through- out the rest of the documentation. Preparation First, install DBIx::Class like you do with any other CPAN distribution. See <http://www.cpan.org/modules/INSTALL.html> and perlmodinstall. Then open the distribution in your shell and change to the subdirectory mentioned earlier, the next command will download and unpack it: $ perl -mCPAN -e'CPAN::Shell->look("DBIx::Class")' DBIx-Class$ cd examples/Schema Inspect the database: DBIx-Class/examples/Schema$ sqlite3 db/example.db .dump You can also use a GUI database browser such as SQLite Manager <https://addons.mozilla.org/firefox/addon/sqlite-manager>. Have a look at the schema classes files in the subdirectory MyApp. The "MyApp::Schema" class is the entry point for loading the other classes and interacting with the database through DBIC and the "Result" classes correspond to the tables in the database. DBIx::Class::Manual::Example shows how to write all that Perl code. That is almost never necessary, though. Instead use dbicdump (part of the distribution DBIx::Class::Schema::Loader) to automatically create schema classes files from an existing database. The chapter "Resetting the database" below shows an example invocation. Connecting to the database A schema object represents the database. use MyApp::Schema qw(); my $schema = MyApp::Schema->connect('dbi:SQLite:db/example.db'); The first four arguments are the same as for "connect" in DBI. Working with data Almost all actions go through a resultset object. Adding data Via intermediate result objects: my $artist_ma = $schema->resultset('Artist')->create({ name => 'Massive Attack', }); my $cd_mezz = $artist_ma->create_related(cds => { title => 'Mezzanine', }); for ('Angel', 'Teardrop') { $cd_mezz->create_related(tracks => { title => $_ }); } Via relation accessors: $schema->resultset('Artist')->create({ name => 'Metallica', cds => [ { title => q{Kill 'Em All}, tracks => [ { title => 'Jump in the Fire' }, { title => 'Whiplash' }, ], }, { title => 'ReLoad', tracks => [ { title => 'The Memory Remains' }, { title => 'The Unforgiven II' }, { title => 'Fuel' }, ], }, ], }); Columns that are not named are filled with default values. The value "undef" acts as a "NULL" in the database. See the chapter "Introspecting the schema classes" below to find out where the non-obvious source name strings such as "Artist" and accessors such as "cds" and "tracks" come from. Set the environment variable "DBI_TRACE='1|SQL'" to see the generated queries. Retrieving data Set up a condition. my $artists_starting_with_m = $schema->resultset('Artist')->search( { name => { like => 'M%' } } ); Iterate over result objects of class "MyApp::Schema::Result::Artist". Result objects represent a row and automatically get accessors for their column names. for my $artist ($artists_starting_with_m->all) { say $artist->name; } Changing data Change the release year of all CDs titled ReLoad. $schema->resultset('Cd')->search( { title => 'ReLoad', } )->update_all( { year => 1997, } ); Removing data Removes all tracks titled Fuel regardless of which CD the belong to. $schema->resultset('Track')->search( { title => 'Fuel', } )->delete_all; Introspecting the schema classes This is useful for getting a feel for the naming of things in a REPL or during explorative programming. From the root to the details: $schema->sources; # returns qw(Cd Track Artist) $schema->source('Cd')->columns; # returns qw(cdid artist title year) $schema->source('Cd')->relationships; # returns qw(artist tracks) From a detail to the root: $some_result->result_source; # returns appropriate source $some_resultset->result_source; $some_resultsource->schema; # returns appropriate schema Resetting the database # delete database file DBIx-Class/examples/Schema$ rm -f db/example.db # create database and set up tables from definition DBIx-Class/examples/Schema$ sqlite3 db/example.db < db/example.sql # fill them with data DBIx-Class/examples/Schema$ perl ./insertdb.pl # delete the schema classes files DBIx-Class/examples/Schema$ rm -rf MyApp # recreate schema classes files from database file DBIx-Class/examples/Schema$ dbicdump -o dump_directory=. MyApp::Schema dbi:SQLite:db/example.db Where to go next If you want to exercise what you learned with a more complicated schema, load Northwind <http://code.google.com/p/northwindextended/> into your database. If you want to transfer your existing SQL knowledge, read DBIx::Class::Manual::SQLHackers. Continue with DBIx::Class::Tutorial and "WHERE TO START READING" in DBIx::Class. perl v5.18.2 2014-01-05 DBIx::Class::Manual::QuickStart(3)
All times are GMT -4. The time now is 06:22 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy