Sponsored Content
Top Forums UNIX for Dummies Questions & Answers comparing Huge Files - Performance is very bad Post 302092588 by ghostdog74 on Tuesday 10th of October 2006 10:58:27 PM
Old 10-10-2006
Python alternative:
Code:
#!/usr/bin/python
deltafile = open("delta.txt","a")
yfile = open("yester_file.txt") #open yesterday file
tfile = open("today_file.txt") #open today file

for i in xrange(0,2000000): #loop 2million records
        yesterline = yfile.readline().strip() #strip newline
        todayline = tfile.readline().strip()
        y_primary , y_2nd, y_3rd , y_4th = yesterline.split("|")
        t_primary, t_2nd, t_3rd, t_4th = todayline.split("|")
        if y_primary == t_primary:
                if y_4th != t_4th:         
                        print >> deltafile , "C|%s|%s|%s|%s" %( t_primary , t_2nd ,"U" , t_4th)
        else:
                print >> deltafile, "A|%s|%s|%s|%s" %( t_primary , t_2nd, t_3rd, t_4th )
                print >> deltafile, "D|%s|%s|%s|%s" %( y_primary , y_2nd, "D", y_4th)

deltafile.close() #close output file

Output:
/home > python test.py
C|aaa|xxxxxxxxxxxxxxxxxxxxxxxxx|U|vvvvvvvvvvvvvvvvvvv
C|bbb|xxxxxxxxxxxxxxxxxxxxxxxxx|U|kkkkkkkkkkkkkkkk
A|ddd|xxxxxxxxxxxxxxxxxxxxxxxxx|I|zzzzzzzzzzzzzzzzz
D|ccc|xxxxxxxxxxxxxxxxxxxxxxxxx|D|bbbbbbbbbbbbbbb
 

9 More Discussions You Might Find Interesting

1. AIX

Bad performance when log in with putty

Hello guys! I'm n00b in AIX and I'm sticked in a problem. (my English is poor enough, but I hope you can understand me :P). So.. I'm trying to connect to an AIX machine with putty, and .. 'using username xxx' appears after 2 sec (OK), but 'xxx@ip's password' appears after 1:15 min. After... (6 Replies)
Discussion started by: combat2k
6 Replies

2. Shell Programming and Scripting

Comparing two huge files

Hi, I have two files file A and File B. File A is a error file and File B is source file. In the error file. First line is the actual error and second line gives the information about the record (client ID) that throws error. I need to compare the first field (which doesnt start with '//') of... (11 Replies)
Discussion started by: kmkbuddy_1983
11 Replies

3. HP-UX

Bad performance but Low CPU loading?

There might be some problem with my server, because every morning at 7, it's performance become bad with no DB extra deadlock. But I just couldn't figure it out. Please give me some advise, thanks a lot... According to the CPU performace chart, Daily CPU loading Maximum: 42 %, Average:36%. ... (8 Replies)
Discussion started by: GreenShery
8 Replies

4. Shell Programming and Scripting

Comparing two huge files on field basis.

Hi all, I have two large files and i want a field by field comparison for each record in it. All fields are tab seperated. file1: Email SELVAKUMAR RAMACHANDRAN Email SHILPA SAHU Web NIYATI SONI Web NIYATI SONI Email VIINII DOSHI Web RAJNISH KUMAR Web ... (4 Replies)
Discussion started by: Suman Singh
4 Replies

5. Shell Programming and Scripting

Comparing 2 huge text files

I have this 2 files: k5login sanwar@systems.nyfix.com jjamnik@systems.nyfix.com nisha@SYSTEMS.NYFIX.COM rdpena@SYSTEMS.NYFIX.COM service/backups-ora@SYSTEMS.NYFIX.COM ivanr@SYSTEMS.NYFIX.COM nasapova@SYSTEMS.NYFIX.COM tpulay@SYSTEMS.NYFIX.COM rsueno@SYSTEMS.NYFIX.COM... (11 Replies)
Discussion started by: linuxgeek
11 Replies

6. Solaris

Performance (iops) becomes bad, what is the reason?

I have written a virtual HBA driver named "xmp_vhba". A scsi disk is attached on it. as shown below: xmp_vhba, instance #0 disk, instance #11 But the performance became very bad when we read/write the scsi disk using the vdbench(a read/write io tool). What is the reason? ... (7 Replies)
Discussion started by: ForgetChen
7 Replies

7. HP-UX

Performance issue with 'grep' command for huge file size

I have 2 files; one file (say, details.txt) contains the details of employees and another file (say, emp.txt) has some selected employee names. I am extracting employee details from details.txt by using emp.txt and the corresponding code is: while read line do emp_name=`echo $line` grep -e... (7 Replies)
Discussion started by: arb_1984
7 Replies

8. Shell Programming and Scripting

Perl: Need help comparing huge files

What do i need to do have the below perl program load 205 million record files into the hash. It currently works on smaller files, but not working on huge files. Any idea what i need to do to modify to make it work with huge files: #!/usr/bin/perl $ot1=$ARGV; $ot2=$ARGV; open(mfileot1,... (12 Replies)
Discussion started by: mrn6430
12 Replies

9. UNIX for Advanced & Expert Users

Performance problem with removing duplicates in a huge file (50+ GB)

I'm trying to remove duplicate data from an input file with unsorted data which is of size >50GB and write the unique records to a new file. I'm trying and already tried out a variety of options posted in similar threads/forums. But no luck so far.. Any suggestions please ? Thanks !! (9 Replies)
Discussion started by: Kannan K
9 Replies
BADSECT(8)						    BSD System Manager's Manual 						BADSECT(8)

NAME
badsect -- create files to contain bad sectors SYNOPSIS
badsect bbdir sector ... DESCRIPTION
badsect makes a file to contain a bad sector. Normally, bad sectors are made inaccessible by the standard formatter, which provides a for- warding table for bad sectors to the driver; see bad144(8) for details. If a driver supports the bad blocking standard it is much preferable to use that method to isolate bad blocks, since the bad block forwarding makes the pack appear perfect, and such packs can then be copied with dd(1). The technique used by this program is also less general than bad block forwarding, as badsect can't make amends for bad blocks in the i-list of file systems or in swap areas. On some disks, adding a sector which is suddenly bad to the bad sector table currently requires the running of the standard DEC formatter. Thus to deal with a newly bad block or on disks where the drivers do not support the bad-blocking standard badsect may be used to good effect. badsect is used on a quiet file system in the following way: First mount the file system, and change to its root directory. Make a directory BAD there. Run badsect giving as argument the BAD directory followed by all the bad sectors you wish to add. The sector numbers must be relative to the beginning of the file system, but this is not hard as the system reports relative sector numbers in its console error mes- sages. Then change back to the root directory, unmount the file system and run fsck(8) on the file system. The bad sectors should show up in two files or in the bad sector files and the free list. Have fsck(8) remove files containing the offending bad sectors, but do not have it remove the BAD/nnnnn files. This will leave the bad sectors in only the BAD files. badsect works by giving the specified sector numbers in a mknod(2) system call, creating an illegal file whose first block address is the block containing bad sector and whose name is the bad sector number. When it is discovered by fsck(8) it will ask ``HOLD BAD BLOCK ?'' A positive response will cause fsck(8) to convert the inode to a regular file containing the bad block. DIAGNOSTICS
badsect refuses to attach a block that resides in a critical area or is out of range of the file system. A warning is issued if the block is already in use. SEE ALSO
bad144(8), fsck(8) HISTORY
The badsect command appeared in 4.1BSD. BUGS
If more than one of the sectors in a file system fragment are bad, you should specify only one of them to badsect, as the blocks in the bad sector files actually cover all the sectors in a file system fragment. BSD
June 5, 1993 BSD
All times are GMT -4. The time now is 04:49 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy