Sponsored Content
Top Forums Shell Programming and Scripting Perl: Need help comparing huge files Post 302671613 by Corona688 on Friday 13th of July 2012 06:43:26 PM
Old 07-13-2012
If you sort your data, however, you can use the comm utility, which does not need to completely load either file into memory. Since the lines are in sorted order, it can tell when a line and when a line is skipped by whether the next line is greater or less or equal...

sort should be smart enough to process in blocks and not run out of memory. Be sure you have enough /tmp/ space, or redirect it to use another folder for temporary files where you have the room. See man sort for details.

Code:
$ sort data1 > data1-s
$ sort data2 > data2-s
$ comm -2 -3 data1-s data2-s > only-data1
$ comm -1 -3 data1-s data2-s > only-data2
$ cat only-data1

1233
4444
7777

$ cat only-data2

1244
9898
9999

$

Note that it might be possible to run comm once to get both sets of data, if only I knew what your data looks like -- which I still don't, after asking several times...
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

comparing Huge Files - Performance is very bad

Hi All, Can you please help me in resolving the following problem? My requirement is like this: 1) I have two files YESTERDAY_FILE and TODAY_FILE. Each one is having nearly two million data. 2) I need to check each record of TODAY_FILE in YESTERDAY_FILE. If exists we can skip that by... (5 Replies)
Discussion started by: madhukalyan
5 Replies

2. Shell Programming and Scripting

Comparing two huge files

Hi, I have two files file A and File B. File A is a error file and File B is source file. In the error file. First line is the actual error and second line gives the information about the record (client ID) that throws error. I need to compare the first field (which doesnt start with '//') of... (11 Replies)
Discussion started by: kmkbuddy_1983
11 Replies

3. Shell Programming and Scripting

Perl script error to split huge data one by one.

Below is my perl script: #!/usr/bin/perl open(FILE,"$ARGV") or die "$!"; @DATA = <FILE>; close FILE; $join = join("",@DATA); @array = split( ">",$join); for($i=0;$i<=scalar(@array);$i++){ system ("/home/bin/./program_name_count_length MULTI_sequence_DATA_FILE -d... (5 Replies)
Discussion started by: patrick87
5 Replies

4. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

5. Shell Programming and Scripting

Problem running Perl Script with huge data files

Hello Everyone, I have a perl script that reads two types of data files (txt and XML). These data files are huge and large in number. I am using something like this : foreach my $t (@text) { open TEXT, $t or die "Cannot open $t for reading: $!\n"; while(my $line=<TEXT>){ ... (4 Replies)
Discussion started by: ad23
4 Replies

6. Shell Programming and Scripting

Comparing two huge files on field basis.

Hi all, I have two large files and i want a field by field comparison for each record in it. All fields are tab seperated. file1: Email SELVAKUMAR RAMACHANDRAN Email SHILPA SAHU Web NIYATI SONI Web NIYATI SONI Email VIINII DOSHI Web RAJNISH KUMAR Web ... (4 Replies)
Discussion started by: Suman Singh
4 Replies

7. Shell Programming and Scripting

Comparing 2 huge text files

I have this 2 files: k5login sanwar@systems.nyfix.com jjamnik@systems.nyfix.com nisha@SYSTEMS.NYFIX.COM rdpena@SYSTEMS.NYFIX.COM service/backups-ora@SYSTEMS.NYFIX.COM ivanr@SYSTEMS.NYFIX.COM nasapova@SYSTEMS.NYFIX.COM tpulay@SYSTEMS.NYFIX.COM rsueno@SYSTEMS.NYFIX.COM... (11 Replies)
Discussion started by: linuxgeek
11 Replies

8. Shell Programming and Scripting

Perl: Comparing to two files and displaying the differences

Hi, I'm new to perl and i have to write a perl script that will compare to log/txt files and display the differences. Unfortunately I'm not allowed to use any complied binaries or applications like diff or comm. So far i've across a code like this: use strict; use warnings; my $list1;... (2 Replies)
Discussion started by: dont_be_hasty
2 Replies

9. Shell Programming and Scripting

Removing Dupes from huge file- awk/perl/uniq

Hi, I have the following command in place nawk -F, '!a++' file > file.uniq It has been working perfectly as per requirements, by removing duplicates by taking into consideration only first 3 fields. Recently it has started giving below error: bash-3.2$ nawk -F, '!a++'... (17 Replies)
Discussion started by: makn
17 Replies

10. Shell Programming and Scripting

Need help in comparing two files using shell or Perl

I have these two file that I am trying to compare using shell arrays. I need to find out the changed or the missing enteries from File2. For example. The line "f nsd1" in file2 is different from file1 and the line "g nsd6" is missing from file2. I dont want to use "for loop" because my files... (2 Replies)
Discussion started by: sags007_99
2 Replies
openipmicmd(1)						     An IPMI Command Interface						    openipmicmd(1)

NAME
openipmicmd - An IPMI Command Interface SYNOPSIS
openipmicmd [-k entry-to-execute] <connection parms> DESCRIPTION
The openipmicmd program allows a user to execute direct IPMI commands. It can work with direct interface with the OpenIPMI driver or with IPMI LAN interfaces. OPTIONS
-k entry-to-execute Execute a single command an exit. <connection parms> The parameters for the connection depend on the connection type. These are all described in openipmi_conparms (7) COMMANDS
Once up, you can execute commands in the user interface. Note that commands and responses are asynchronous, you issue a command and the interface returns immediately. When the response comes back, it will be dumped on your console. That's a little strange looking, but IPMI is asynchronous underneath. Note that the -k option is synchronous, it will wait for the response or a timeout before returning. 0f lun netfn cmd [data1 [data2 ...]] Send a command to the BMC you are connected to. channel [ipmb] IPMB-addr lun netfn [seq] cmd [data1 [data2 ...]] Send a command to a device on the IPMB bus. The "ipmb" string is optional. The seq must be provided if the netfn is a response (an odd number). It must be the same sequence number that came in on the command in the "Command Sequence = seq" part of the command. channel 00 IPMB-addr lun netfn cmd [data1 [data2 ...]] Send a broadcast command to a device on the IPMB bus. channel lan handle remote-swid local-swid lun netfn cmd [data1 [data ...]] Send a command to a device over a LAN channel. Note that this not the same as a LAN connection. This sends a message through a local BMC to a remote system that is hooked up with a LAN connection. test_lat count command Executes the given commands (one of the previous commands) count times and gives the average time per command to execute. Note that "count" is hexadecimal. help Display some help. regcmd netfn cmd Register to receive the given command. If the driver receives an external command, it will print it out. This only works with sys- tem interface connections, it will not work on LAN connections. unregcmd netfn cmd Remove a command registration. SEE ALSO
ipmi_ui(1), ipmilan(8), openipmi_conparms(7) KNOWN PROBLEMS
The asynchronous nature of the program can be annoying. AUTHOR
Corey Minyard <cminyard@mvista.org> OpenIPMI 05/13/03 openipmicmd(1)
All times are GMT -4. The time now is 07:43 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy