07-13-2012
Perl's only limitation is the amount of memory in your system.
If these 205 million records are 4 or 5 bytes each like you've shown, that might amount to close to a gig of memory. If they're much larger, they probably won't fit into memory on a 32-bit system and other approaches would need to be tried, such as sorting them, so you can tell when a record's absent without having to load every possible record into memory at once...
If your records aren't as you've shown, then nothing we've written for you is likely to work at all anyway. We need to see what you're really dealing with.
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hi All,
Can you please help me in resolving the following problem?
My requirement is like this:
1) I have two files YESTERDAY_FILE and TODAY_FILE. Each one is having nearly two million data.
2) I need to check each record of TODAY_FILE in YESTERDAY_FILE. If exists we can skip that by... (5 Replies)
Discussion started by: madhukalyan
5 Replies
2. Shell Programming and Scripting
Hi,
I have two files file A and File B. File A is a error file and File B is source file. In the error file. First line is the actual error and second line gives the information about the record (client ID) that throws error. I need to compare the first field (which doesnt start with '//') of... (11 Replies)
Discussion started by: kmkbuddy_1983
11 Replies
3. Shell Programming and Scripting
Below is my perl script:
#!/usr/bin/perl
open(FILE,"$ARGV") or die "$!";
@DATA = <FILE>;
close FILE;
$join = join("",@DATA);
@array = split( ">",$join);
for($i=0;$i<=scalar(@array);$i++){
system ("/home/bin/./program_name_count_length MULTI_sequence_DATA_FILE -d... (5 Replies)
Discussion started by: patrick87
5 Replies
4. Shell Programming and Scripting
Hi, all:
I've got two folders, say, "folder1" and "folder2".
Under each, there are thousands of files.
It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command.
However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies
5. Shell Programming and Scripting
Hello Everyone,
I have a perl script that reads two types of data files (txt and XML). These data files are huge and large in number. I am using something like this :
foreach my $t (@text)
{
open TEXT, $t or die "Cannot open $t for reading: $!\n";
while(my $line=<TEXT>){
... (4 Replies)
Discussion started by: ad23
4 Replies
6. Shell Programming and Scripting
Hi all,
I have two large files and i want a field by field comparison for each record in it.
All fields are tab seperated.
file1:
Email SELVAKUMAR RAMACHANDRAN
Email SHILPA SAHU
Web NIYATI SONI
Web NIYATI SONI
Email VIINII DOSHI
Web RAJNISH KUMAR
Web ... (4 Replies)
Discussion started by: Suman Singh
4 Replies
7. Shell Programming and Scripting
I have this 2 files:
k5login
sanwar@systems.nyfix.com
jjamnik@systems.nyfix.com
nisha@SYSTEMS.NYFIX.COM
rdpena@SYSTEMS.NYFIX.COM
service/backups-ora@SYSTEMS.NYFIX.COM
ivanr@SYSTEMS.NYFIX.COM
nasapova@SYSTEMS.NYFIX.COM
tpulay@SYSTEMS.NYFIX.COM
rsueno@SYSTEMS.NYFIX.COM... (11 Replies)
Discussion started by: linuxgeek
11 Replies
8. Shell Programming and Scripting
Hi,
I'm new to perl and i have to write a perl script that will compare to log/txt files and display the differences. Unfortunately I'm not allowed to use any complied binaries or applications like diff or comm.
So far i've across a code like this:
use strict;
use warnings;
my $list1;... (2 Replies)
Discussion started by: dont_be_hasty
2 Replies
9. Shell Programming and Scripting
Hi,
I have the following command in place
nawk -F, '!a++' file > file.uniq
It has been working perfectly as per requirements, by removing duplicates by taking into consideration only first 3 fields. Recently it has started giving below error:
bash-3.2$ nawk -F, '!a++'... (17 Replies)
Discussion started by: makn
17 Replies
10. Shell Programming and Scripting
I have these two file that I am trying to compare using shell arrays. I need to find out the changed or the missing
enteries from File2. For example. The line "f nsd1" in file2 is different from file1 and the line "g nsd6" is missing
from file2.
I dont want to use "for loop" because my files... (2 Replies)
Discussion started by: sags007_99
2 Replies
LEARN ABOUT HPUX
audit_memory_usage
audit_memory_usage(5) File Formats Manual audit_memory_usage(5)
NAME
audit_memory_usage - percentage of physical memory that can be used by audit subsystem
VALUES
Failsafe
Default
Allowed values
Recommended values
DESCRIPTION
The tunable defines the percentage of physical memory that can be used by audit records. If the audit record memory usage exceeds the
limit, the audit subsystem blocks the system call until memory usage comes down inside the limit.
Who Is Expected to Change This Tunable?
Administrator with proper privileges can change the value of
Restrictions on Changing
This tunable value can be changed any time, as this is a dynamic tunable; but its value can not be lowered below the memory currently used
by the audit subsystem for records.
When Should the Value of This Tunable Be Raised?
When the system to be audited is critical and generates many records.
What are the Side Effects of Raising the Value?
If the system is generating great number of records, and the value is raised, then it can create a lot of unflushed memory and might slow
down the system.
When Should the Value of This Tunable Be Lowered?
When the audit subsystem is not generating many records.
What are the Side Effects of Lowering the Value?
If many audit records are generated, then the auditing subsystem might block the system call.
What Other Tunable Values Should Be Changed at the Same Time?
None.
WARNINGS
All HP-UX kernel tunable parameters are release specific. This parameter may be removed or have its meaning changed in future releases of
HP-UX.
Installation of optional kernel software, from HP or other vendors, may cause changes to tunable parameter values. After installation,
some tunable parameters may no longer be at the default or recommended values. For information about the effects of installation on tun-
able values, consult the documentation for the kernel software being installed. For information about optional kernel software that was
factory installed on your system, see at
AUTHOR
was developed by HP.
SEE ALSO
audit(5).
Tunable Kernel Parameters audit_memory_usage(5)