Sponsored Content
Top Forums Programming Suitable data structure large number of heterogeneous records Post 302506468 by shoaibjameel123 on Monday 21st of March 2011 01:07:53 AM
Old 03-21-2011
First of all my sincere apologies. I should have put in the comments there. Secondly, your code does seem to do the trick. It sorts in less than a second on my 1.3 million data. This means there's something wrong with my code. I was quite confident about my code as it gave perfect results on a small dataset but don't kow why it failed on such a huge data. I guess the sorting problem is solved now. Thanks again!
 

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract data from large file 80+ million records

Hello, I have got one file with more than 120+ million records(35 GB in size). I have to extract some relevant data from file based on some parameter and generate other output file. What will be the besat and fastest way to extract the ne file. sample file format :--... (2 Replies)
Discussion started by: learner16s
2 Replies

2. Shell Programming and Scripting

How to Pick Random records from a large file

Hi, I have a huge file say with 2000000 records. The file has 42 fields. I would like to pick randomly 1000 records from this huge file. Can anyone help me how to do this? (1 Reply)
Discussion started by: ajithshankar@ho
1 Replies

3. Shell Programming and Scripting

Find line number of bad data in large file

Hi Forum. I was trying to search the following scenario on the forum but was not able to. Let's say that I have a very large file that has some bad data in it (for ex: 0.0015 in the 12th column) and I would like to find the line number and remove that particular line. What's the easiest... (3 Replies)
Discussion started by: pchang
3 Replies

4. Shell Programming and Scripting

awk - splitting 1 large file into multiple based on same key records

Hello gurus, I am new to "awk" and trying to break a large file having 4 million records into several output files each having half million but at the same time I want to keep the similar key records in the same output file, not to exist accross the files. e.g. my data is like: Row_Num,... (6 Replies)
Discussion started by: kam66
6 Replies

5. Shell Programming and Scripting

AWK print number of records, divide this number

I would like to print the number of records of 2 files, and divide the two numbers awk '{print NR}' file1 > output1 awk '{print NR}' file2 > output2 paste output1 output2 > output awl '{print $1/$2}' output > output_2 is there a faster way? (8 Replies)
Discussion started by: programmerc
8 Replies

6. Shell Programming and Scripting

Compare two files with different number of records and output only the Extra records from file1

Hi Freinds , I have 2 files . File 1 |nag|HYd|1|Che |esw|Gun|2|hyd |pra|bhe|3|hyd |omu|hei|4|bnsj |uer|oeri|5|uery File 2 |nag|HYd|1|Che |esw|Gun|2|hyd |uer|oi|3|uery output : (9 Replies)
Discussion started by: i150371485
9 Replies

7. Shell Programming and Scripting

Split a large file in n records and skip a particular record

Hello All, I have a large file, more than 50,000 lines, and I want to split it in even 5000 records. Which I can do using sed '1d;$d;' <filename> | awk 'NR%5000==1{x="F"++i;}{print > x}'Now I need to add one more condition that is not to break the file at 5000th record if the 5000th record... (20 Replies)
Discussion started by: ibmtech
20 Replies

8. Shell Programming and Scripting

Quick way to select many records from a large file

I have a file, named records.txt, containing large number of records, around 0.5 million records in format below: 28433005 1 1 3 2 2 2 2 2 2 2 2 2 2 2 28433004 0 2 3 2 2 2 2 2 2 1 2 2 2 2 ... Another file is a key file, named key.txt, which is the list of some numbers in the first column of... (5 Replies)
Discussion started by: zenongz
5 Replies
UNPACK(3PVM)							  PVM Version 3.4						      UNPACK(3PVM)

NAME
pvm_unpack - Unpack the active message buffer into arrays of prescribed data type. SYNOPSIS
C int info = pvm_unpackf( const char *fmt, ... ) int info = pvm_upkbyte( char *xp, int nitem, int stride) int info = pvm_upkcplx( float *cp, int nitem, int stride) int info = pvm_upkdcplx( double *zp, int nitem, int stride) int info = pvm_upkdouble( double *dp, int nitem, int stride) int info = pvm_upkfloat( float *fp, int nitem, int stride) int info = pvm_upkint( int *ip, int nitem, int stride) int info = pvm_upkuint( unsigned int *ip, int nitem, int stride ) int info = pvm_upkushort( unsigned short *ip, int nitem, int stride ) int info = pvm_upkulong( unsigned long *ip, int nitem, int stride ) int info = pvm_upklong( long *ip, int nitem, int stride) int info = pvm_upkshort( short *jp, int nitem, int stride) int info = pvm_upkstr( char *sp ) Fortran call pvmfunpack( what, xp, nitem, stride, info ) PARAMETERS
fmt Printf-like format expression specifying what to pack. (See discussion) nitem The total number of items to be unpacked (not the number of bytes). stride The stride to be used when packing the items. For example, if stride = 2 in pvm_upkcplx, then every other complex number will be unpacked. xp Pointer to the beginning of a block of bytes. Can be any data type, but must match the corresponding pack data type. cp Complex array at least nitem*stride items long. zp Double precision complex array at least nitem*stride items long. dp Double precision real array at least nitem*stride items long. fp Real array at least nitem*stride items long. ip Integer array at least nitem*stride items long. jp Integer*2 array at least nitem*stride items long. sp Pointer to a null terminated character string. what Integer specifying the type of data being unpacked. what options STRING 0 REAL4 4 BYTE1 1 COMPLEX8 5 INTEGER2 2 REAL8 6 INTEGER4 3 COMPLEX16 7 info Integer status code returned by the routine. Values less than zero indicate an error. DESCRIPTION
Each of the pvm_upk* routines unpacks an array of the given data type from the active receive buffer. The arguments for each of the rou- tines are a pointer to the array to be unpacked into, nitem which is the total number of items to unpack, and stride which is the stride to use when unpacking. An exception is pvm_upkstr() which by definition unpacks a NULL terminated character string and thus does not need nitem or stride argu- ments. The Fortran routine pvmfunpack( STRING, ... ) expects nitem to be the number of characters in the string and stride to be 1. If the unpacking is successful, info will be 0. If some error occurs then info will be < 0. A single variable (not an array) can be unpacked by setting nitem = 1 and stride = 1. The routine pvm_unpackf() uses a printf-like format expression to specify what and how to unpack data from the receive buffer. All vari- ables are passed as addresses. A BNF-like description of the format syntax is: format : null | init | format fmt init : null | '%' '+' fmt : '%' count stride modifiers fchar fchar : 'c' | 'd' | 'f' | 'x' | 's' count : null | [0-9]+ | '*' stride : null | '.' ( [0-9]+ | '*' ) modifiers : null | modifiers mchar mchar : 'h' | 'l' | 'u' Formats: + means initsend - must match an int (how) in the param list. c pack/unpack bytes d integer f float x complex float s string Modifiers: h short (int) l long (int, float, complex float) u unsigned (int) Future extensions to the what argument in pvmfunpack will include 64 bit types when XDR encoding of these types is available. Meanwhile users should be aware that precision can be lost when passing data from a 64 bit machine like a Cray to a 32 bit machine like a SPARCsta- tion. As a mnemonic the what argument name includes the number of bytes of precision to expect. By setting encoding to PVMRAW (see pvmfinitsend) data can be transferred between two 64 bit machines with full precision even if the PVM configuration is heterogeneous. Messages should be unpacked exactly like they were packed to insure data integrity. Packing integers and unpacking them as floats will often fail because a type encoding will have occurred transferring the data between heterogeneous hosts. Packing 10 integers and 100 floats then trying to unpack only 3 integers and the 100 floats will also fail. EXAMPLES
C: info = pvm_recv( tid, msgtag ); info = pvm_upkstr( string ); info = pvm_upkint( &size, 1, 1 ); info = pvm_upkint( array, size, 1 ); info = pvm_upkdouble( matrix, size*size, 1 ); int count, *iarry; double darry[4]; pvm_unpackf("%d", &count); pvm_unpackf("%*d %4lf", count, iarry, darry); Fortran: CALL PVMFRECV( TID, MSGTAG, INFO ); CALL PVMFUNPACK( INTEGER4, NSIZE, 1, 1, INFO ) CALL PVMFUNPACK( STRING, STEPNAME, 8, 1, INFO ) CALL PVMFUNPACK( REAL4, A(5,1), NSIZE, NSIZE , INFO ) ERRORS
PvmNoData Reading beyond the end of the receive buffer. Most likely cause is trying to unpack more items than were originally packed into the buffer. PvmBadMsg The received message can not be decoded. Most likely because the hosts are heterogeneous and the user specified an incompatible encoding. Try setting the encoding to PvmDataDefault (see pvm_mkbuf). PvmNoBuf There is no active receive buffer to unpack. SEE ALSO
pvm_pack(3PVM) pvm_send(3PVM), pvm_recv(3PVM), pvm_pkmesg(3PVM) 30 August, 1993 UNPACK(3PVM)
All times are GMT -4. The time now is 10:13 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy