Sponsored Content
Top Forums Shell Programming and Scripting Matching 10 Million file records with 10 Million in other file Post 302655427 by vguleria on Wednesday 13th of June 2012 07:09:06 AM
Old 06-13-2012
The OS is linux, it's a one time job(occasionally). these are offline files and not being updated. Need to make a process for future requirements.

Its not in DB.. actually these are application log files.
The size of files are 1.5G approx. Right now only thinking of the best way/approach to complete the task...
Had tried using perl hashes(didn't work), i guess keeping that much data in memory is not possible... hence algorithm has to be really efficient here.Smilie


Sample files:
Input.txt
Code:
20.04.2012 11.08.44;RECV;APPNAME@HOSTNAME06:11496059192;processed;Location;contact;status;email_id;2
20.04.2012 11.08.44;RECV;APPNAME@HOSTNAME06:11496059168;processed;Location;contact;status;email_id;1
20.04.2012 11.08.44;RECV;APPNAME@HOSTNAME06:11496059220;processed;Location;contact;status;email_id;2

Status.txt
Code:
APPNAME@HOSTNAME06:11496059192;SUCCESS
APPNAME@HOSTNAME06:11496059224;SUCCESS
APPNAME@HOSTNAME06:11496059168;FAILURE
APPNAME@HOSTNAME06:11496059220;FAILURE
APPNAME@HOSTNAME06:11496059193;SUCCESS

need to update the status field in input.txt with the status(success/failure) in status.txt

Last edited by Franklin52; 06-13-2012 at 08:47 AM.. Reason: Please use code tags for data and code samples
 

7 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract data from large file 80+ million records

Hello, I have got one file with more than 120+ million records(35 GB in size). I have to extract some relevant data from file based on some parameter and generate other output file. What will be the besat and fastest way to extract the ne file. sample file format :--... (2 Replies)
Discussion started by: learner16s
2 Replies

2. Shell Programming and Scripting

sort a file which has 3.7 million records

hi, I'm trying to sort a file which has 3.7 million records an gettign the following error...any help is appreciated... sort: Write error while merging. Thanks (6 Replies)
Discussion started by: greenworld
6 Replies

3. What is on Your Mind?

Pick a Number Between 0 and 20 for 1 Million Bits

Here is an easy game! I wrote a number between 0 and 20 (that can include 0 and 20) on a piece of paper. I am staring at it now, imagining the number so you can read my mind ;) Reply once, and only once, with a number from 0 to 20 and the first person to guess it wins 1,000,000 Bits. ... (24 Replies)
Discussion started by: Neo
24 Replies

4. Shell Programming and Scripting

Tail 86000 lines from 1.2 million line file?

I have a log file that is about 1.2 million lines long and about 300MB. we need a way to clean up this file and only keep the last few thousand lines. if i use tail command we run our of memory as the file is too big. I do have a key word to match on. example, we want to keep every line... (8 Replies)
Discussion started by: robsonde
8 Replies

5. UNIX for Dummies Questions & Answers

Pls. help with script to remove million files

Hi, one of the server, log directory was never cleaned up. We have so many files. I want to remove all the files that starts with dfr* but I get error message when I use the *. rm qfr* bash: /usr/bin/rm: Arg list too long I am trying to write this script but not working. ... (4 Replies)
Discussion started by: samnyc
4 Replies

6. UNIX for Dummies Questions & Answers

Deleting a million of files ..

Hi, Which way is faster rm -rf /path/ or find / -name -exec rm {} \; and why? (7 Replies)
Discussion started by: cain82
7 Replies

7. UNIX for Dummies Questions & Answers

Add 1 million columns

Hi, here is my problem: I've got a file with 6 columns (file1): a b c d e f a b c d e f a b c d e f a b c d e f I need to add 1 million columns to this file, each column needs to be a zero. Here is how the result file (file2) should look like (for the sake of the example, I've only... (7 Replies)
Discussion started by: zajtat
7 Replies
lb_register(3ncs)														 lb_register(3ncs)

Name
       lb_register - register an object and an interface with the Location Broker

Syntax
       #include <idl/c/lb.h>

       void lb_$register(object, obj_type, obj_interface, flags, annotation,			location, location_length, entry, status)
       uuid_$t *object;
       uuid_$t *obj_type;
       uuid_$t *obj_interface;
       lb_$server_flag_t flags;
       unsigned char annotation[64];
       socket_$addr_t *location;
       unsigned long location_length;
       lb_$entry_t *entry;
       status_$t *status;

Arguments
       object		   The UUID of the object being registered.

       obj_type 	   The UUID of the type of the object being registered.

       obj_interface	   The UUID of the interface being registered.

       flags		   Must  be  either  lb_$server_flag_local  (specifying registration with only the LLB at the local host) or 0 (specifying
			   registration with both the LLB and the GLB).

       annotation	   A character array used only for informational purposes.  This field can contain a textual description of the object and
			   the interface.  For proper display by the tool, the annotation should be terminated by a null character.

       location 	   The socket address of the server that exports the interface to the object.

       location_length	   The length, in bytes, of the socket address specified by the location field.

       entry		   A copy of the entry that was entered in the Location Broker database.

       status		   The completion status.  If the completion status returned in is equal to status_$ok , then the routine that supplied it
			   was successful.

Description
       The routine registers with the Location Broker an interface to an object and the location of a server that exports  that  interface.   This
       routine replaces any existing entry in the Location Broker database that matches object, obj_type, obj_interface, and both the address fam-
       ily and host in location; if no such entry exists, the routine adds a new entry to the database.

       If the flags parameter is the entry is registered only in the LLB database at the host where the  call  is  issued.   Otherwise,  the  flag
       should be 0 to register with both the LLB and the GLB databases.

Examples
       The following statement registers the interface to the object identified by :
       lb_$register (&bank_id, &bank_$uuid, &bank_$if_spec.id, 0,
			  BankName, &saddr, slen, &entry, &status);

Diagnostics
       This section lists status codes for errors returned by this routine in

       lb_$database_invalid
			   The format of the Location Broker database is out of date.  The database may have been created by an old version of the
			   Location Broker; in this case, delete the out-of-date database and reregister any entries that it contained.   The  LLB
			   or  GLB that was accessed may be running out-of-date software; in this case, update all Location Brokers to the current
			   software version.

       lb_$database_busy   The Location Broker database is currently in use in an incompatible manner.

       lb_$update_failed   The Location Broker was unable to register the entry.

       lb_$cant_access	   The Location Broker cannot access the database.  Among the possible reasons:

			   1. The database does not exist, and the Location Broker cannot create it.

			   2. The database exists, but the Location Broker cannot access it.

			   3. The GLB entry table is full.

       lb_$server_unavailable
			   The Location Broker Client Agent cannot reach the requested GLB or LLB.  A communications failure occurred or the  bro-
			   ker was not running.

Files
See Also
       intro(3ncs), lb_unregister(3ncs)

																 lb_register(3ncs)
All times are GMT -4. The time now is 10:08 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy