Sponsored Content
Top Forums Programming Suitable data structure large number of heterogeneous records Post 302506468 by shoaibjameel123 on Monday 21st of March 2011 01:07:53 AM
Old 03-21-2011
First of all my sincere apologies. I should have put in the comments there. Secondly, your code does seem to do the trick. It sorts in less than a second on my 1.3 million data. This means there's something wrong with my code. I was quite confident about my code as it gave perfect results on a small dataset but don't kow why it failed on such a huge data. I guess the sorting problem is solved now. Thanks again!
 

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract data from large file 80+ million records

Hello, I have got one file with more than 120+ million records(35 GB in size). I have to extract some relevant data from file based on some parameter and generate other output file. What will be the besat and fastest way to extract the ne file. sample file format :--... (2 Replies)
Discussion started by: learner16s
2 Replies

2. Shell Programming and Scripting

How to Pick Random records from a large file

Hi, I have a huge file say with 2000000 records. The file has 42 fields. I would like to pick randomly 1000 records from this huge file. Can anyone help me how to do this? (1 Reply)
Discussion started by: ajithshankar@ho
1 Replies

3. Shell Programming and Scripting

Find line number of bad data in large file

Hi Forum. I was trying to search the following scenario on the forum but was not able to. Let's say that I have a very large file that has some bad data in it (for ex: 0.0015 in the 12th column) and I would like to find the line number and remove that particular line. What's the easiest... (3 Replies)
Discussion started by: pchang
3 Replies

4. Shell Programming and Scripting

awk - splitting 1 large file into multiple based on same key records

Hello gurus, I am new to "awk" and trying to break a large file having 4 million records into several output files each having half million but at the same time I want to keep the similar key records in the same output file, not to exist accross the files. e.g. my data is like: Row_Num,... (6 Replies)
Discussion started by: kam66
6 Replies

5. Shell Programming and Scripting

AWK print number of records, divide this number

I would like to print the number of records of 2 files, and divide the two numbers awk '{print NR}' file1 > output1 awk '{print NR}' file2 > output2 paste output1 output2 > output awl '{print $1/$2}' output > output_2 is there a faster way? (8 Replies)
Discussion started by: programmerc
8 Replies

6. Shell Programming and Scripting

Compare two files with different number of records and output only the Extra records from file1

Hi Freinds , I have 2 files . File 1 |nag|HYd|1|Che |esw|Gun|2|hyd |pra|bhe|3|hyd |omu|hei|4|bnsj |uer|oeri|5|uery File 2 |nag|HYd|1|Che |esw|Gun|2|hyd |uer|oi|3|uery output : (9 Replies)
Discussion started by: i150371485
9 Replies

7. Shell Programming and Scripting

Split a large file in n records and skip a particular record

Hello All, I have a large file, more than 50,000 lines, and I want to split it in even 5000 records. Which I can do using sed '1d;$d;' <filename> | awk 'NR%5000==1{x="F"++i;}{print > x}'Now I need to add one more condition that is not to break the file at 5000th record if the 5000th record... (20 Replies)
Discussion started by: ibmtech
20 Replies

8. Shell Programming and Scripting

Quick way to select many records from a large file

I have a file, named records.txt, containing large number of records, around 0.5 million records in format below: 28433005 1 1 3 2 2 2 2 2 2 2 2 2 2 2 28433004 0 2 3 2 2 2 2 2 2 1 2 2 2 2 ... Another file is a key file, named key.txt, which is the list of some numbers in the first column of... (5 Replies)
Discussion started by: zenongz
5 Replies
H5FROMH4(1)							      h5utils							       H5FROMH4(1)

NAME
h5fromh4 - convert HDF4 scientific datasets to an HDF5 file SYNOPSIS
h5fromh4 [OPTION]... [HDF4FILE]... DESCRIPTION
h5fromh4 takes one or more files in HDF4 format and outputs files in HDF5 format containing the datasets from the HDF4 files. (Currently, only a single dataset per HDF4 file is converted.) HDF4 and HDF5 are free, portable binary formats and supporting libraries developed by the National Center for Supercomputing Applications at the University of Illinois in Urbana-Champaign. A single h5 file can contain multiple data sets; by default, h5fromh4 creates a dataset called "data", but this can be changed via the -d option, or by using the syntax HDF5FILE:DATASET with the -o option. The -a option can be used to append new datasets to an existing HDF5 file. If the -o option is used and multiple HDF4 files are specified, all the HDF4 datasets are output into that HDF5 file with the input filenames (minus the ".hdf" suffix) used as the dataset names. The most basic usage is something like 'h5fromh4 foo.hdf', which will output a file foo.h5 containing the scientific dataset from foo.hdf. OPTIONS
-h Display help on the command-line options and usage. -V Print the version number and copyright info for h5fromh4. -v Verbose output. -a If the HDF5 output file already exists, append the data as a new dataset rather than overwriting the file (the default behavior). An existing dataset of the same name within the file is overwritten, however. -o file Send HDF5 output to file rather than to the input filename with .hdf replaced with .h5 (the default). If multiple input files were specified, this causes all input datasets to be stored in file (rather than in separate files), with the input filenames (minus the .hdf suffix) as the dataset names. -d name Write to dataset name in the output; otherwise, the output dataset is called "data" by default. Alternatively, use the syntax HDF5FILE:DATASET with the -o option. BUGS
Send bug reports to S. G. Johnson, stevenj@alum.mit.edu. AUTHORS
Written by Steven G. Johnson. Copyright (c) 2005 by the Massachusetts Institute of Technology. h5utils March 9, 2002 H5FROMH4(1)
All times are GMT -4. The time now is 05:03 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy