Sponsored Content
Top Forums Shell Programming and Scripting Matching 10 Million file records with 10 Million in other file Post 302655427 by vguleria on Wednesday 13th of June 2012 07:09:06 AM
Old 06-13-2012
The OS is linux, it's a one time job(occasionally). these are offline files and not being updated. Need to make a process for future requirements.

Its not in DB.. actually these are application log files.
The size of files are 1.5G approx. Right now only thinking of the best way/approach to complete the task...
Had tried using perl hashes(didn't work), i guess keeping that much data in memory is not possible... hence algorithm has to be really efficient here.Smilie


Sample files:
Input.txt
Code:
20.04.2012 11.08.44;RECV;APPNAME@HOSTNAME06:11496059192;processed;Location;contact;status;email_id;2
20.04.2012 11.08.44;RECV;APPNAME@HOSTNAME06:11496059168;processed;Location;contact;status;email_id;1
20.04.2012 11.08.44;RECV;APPNAME@HOSTNAME06:11496059220;processed;Location;contact;status;email_id;2

Status.txt
Code:
APPNAME@HOSTNAME06:11496059192;SUCCESS
APPNAME@HOSTNAME06:11496059224;SUCCESS
APPNAME@HOSTNAME06:11496059168;FAILURE
APPNAME@HOSTNAME06:11496059220;FAILURE
APPNAME@HOSTNAME06:11496059193;SUCCESS

need to update the status field in input.txt with the status(success/failure) in status.txt

Last edited by Franklin52; 06-13-2012 at 08:47 AM.. Reason: Please use code tags for data and code samples
 

7 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract data from large file 80+ million records

Hello, I have got one file with more than 120+ million records(35 GB in size). I have to extract some relevant data from file based on some parameter and generate other output file. What will be the besat and fastest way to extract the ne file. sample file format :--... (2 Replies)
Discussion started by: learner16s
2 Replies

2. Shell Programming and Scripting

sort a file which has 3.7 million records

hi, I'm trying to sort a file which has 3.7 million records an gettign the following error...any help is appreciated... sort: Write error while merging. Thanks (6 Replies)
Discussion started by: greenworld
6 Replies

3. What is on Your Mind?

Pick a Number Between 0 and 20 for 1 Million Bits

Here is an easy game! I wrote a number between 0 and 20 (that can include 0 and 20) on a piece of paper. I am staring at it now, imagining the number so you can read my mind ;) Reply once, and only once, with a number from 0 to 20 and the first person to guess it wins 1,000,000 Bits. ... (24 Replies)
Discussion started by: Neo
24 Replies

4. Shell Programming and Scripting

Tail 86000 lines from 1.2 million line file?

I have a log file that is about 1.2 million lines long and about 300MB. we need a way to clean up this file and only keep the last few thousand lines. if i use tail command we run our of memory as the file is too big. I do have a key word to match on. example, we want to keep every line... (8 Replies)
Discussion started by: robsonde
8 Replies

5. UNIX for Dummies Questions & Answers

Pls. help with script to remove million files

Hi, one of the server, log directory was never cleaned up. We have so many files. I want to remove all the files that starts with dfr* but I get error message when I use the *. rm qfr* bash: /usr/bin/rm: Arg list too long I am trying to write this script but not working. ... (4 Replies)
Discussion started by: samnyc
4 Replies

6. UNIX for Dummies Questions & Answers

Deleting a million of files ..

Hi, Which way is faster rm -rf /path/ or find / -name -exec rm {} \; and why? (7 Replies)
Discussion started by: cain82
7 Replies

7. UNIX for Dummies Questions & Answers

Add 1 million columns

Hi, here is my problem: I've got a file with 6 columns (file1): a b c d e f a b c d e f a b c d e f a b c d e f I need to add 1 million columns to this file, each column needs to be a zero. Here is how the result file (file2) should look like (for the sake of the example, I've only... (7 Replies)
Discussion started by: zajtat
7 Replies
lb_unregister(3ncs)													       lb_unregister(3ncs)

Name
       lb_unregister - remove an entry from the Location Broker database

Syntax
       #include <idl/c/lb.h>

       void lb_$unregister(entry, status)
       lb_$entry_t *entry;
       status_$t *status;

Arguments
       entry  The entry being removed from the Location Broker database.

       status The completion status.  If the completion status returned in is equal to status_$ok , then the routine that supplied it was success-
	      ful.

Description
       The routine removes from the Location Broker database the entry that matches entry.  The  value	of  entry  should  be  identical  to  that
       returned  by  the routine when the database entry was created.  However, does not compare all of the fields in entry, the annotation field,
       and the port number in the saddr field.

       This routine removes the entry from the LLB database on the local host (the host that issues the routine).  If the flags field of entry	is
       equal  to  0,  it  removes  the entry from the GLB database.  If the flags field is equal to lb_$server_flag_local, it deletes only the LLB
       entry.

Examples
       The following statement unregisters the entry specified by which was obtained from a previous routine:
       lb_$unregister (&BankEntry, &status);

Diagnostics
       This section lists status codes for errors returned by this routine in

       lb_$database_invalid
			   The format of the Location Broker database is out of date.  The database may have been created by an old version of the
			   Location  Broker;  in this case, delete the out-of-date database and reregister any entries that it contained.  The LLB
			   or GLB that was accessed may be running out-of-date software; in this case, update all Location Brokers to the  current
			   software version.

       lb_$database_busy   The Location Broker database is currently in use in an incompatible manner.

       lb_$not_registered  The	Location  Broker  does	not  have  any	entries  that match the criteria specified in the unregister routine.  The
			   requested object, type, interface, or combination thereof is not registered in the specified database.

       lb_$update_failed   The Location Broker was unable to register or unregister the entry.

       lb_$cant_access	   The Location Broker cannot access the database.  Among the possible reasons:

			   1. The database does not exist.

			   2. The database exists, but the Location Broker cannot access it.

       lb_$server_unavailable
			   The Location Broker Client Agent cannot reach the requested GLB or LLB.  A communications failure occurred or the  bro-
			   ker was not running.

Files
See Also
       intro(3ncs), lb_register(3ncs)

															       lb_unregister(3ncs)
All times are GMT -4. The time now is 02:34 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy