Sponsored Content
Top Forums Shell Programming and Scripting How to extract a subset from a huge dataset Post 302403613 by cliffyiu on Saturday 13th of March 2010 11:06:43 AM
Old 03-13-2010
How to extract a subset from a huge dataset

Hi, All

I have a huge file which has 450G. Its tab-delimited format is as below

Code:
x1 A 50020 1
x1 B 50021 8
x1 C 50022 9
x1 A 50023 10
x2 D 50024 5
x2 C 50025 7
x2 F 50026 8
x2 N 50027 1
:
:

Now, I want to extract a subset from this file. In this subset, column 1 is x10, column 2 is from 600000 to 30000000. I wrote the following perl script but it doesn't work:

Code:
#!/usr/bin/perl

$file1 = $ARGV[0]; # Input file
$file2 = $ARGV[1]; # Output file

open (IN, $file1);
while ($line = <IN>)
{
  chomp($line);
  @array = split(/\t/,$line);

  if ($array[0] eq 'x10')
  {
    if (($array[2] >= 600000) && ($array[2] <= 26279795))
    {
      open (OUT, ">>$file2");
      print OUT "$line\n";
      close OUT;
    }
  }
}
close IN;
exit;

I guess the input file and output file are both too big that my script can't handle it.

Anyone knows if there is any good way to do it? Perl or Shell scripts are preferred..

All your help will be appreciated!

Last edited by Franklin52; 03-13-2010 at 01:47 PM.. Reason: Please indent your code and use code tags!!
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Accessing Mainframe Dataset

Hi May I know is there a way to read/copy a mainframe (IBM OS/390) dataset (sequential file) into a UNIX directory? Thank you for your time. IcyGuava (4 Replies)
Discussion started by: IcyGuava
4 Replies

2. Shell Programming and Scripting

How to extract data from a huge file?

Hi, I have a huge file of bibliographic records in some standard format.I need a script to do some repeatable task as follows: 1. Needs to create folders as the strings starts with "item_*" from the input file 2. Create a file "contents" in each folders having "license.txt(tab... (5 Replies)
Discussion started by: srsahu75
5 Replies

3. Shell Programming and Scripting

How to extract a piece of information from a huge file

Hello All, I need some assistance to extract a piece of information from a huge file. The file is like this one : database information ccccccccccccccccc ccccccccccccccccc ccccccccccccccccc ccccccccccccccccc os information cccccccccccccccccc cccccccccccccccccc... (2 Replies)
Discussion started by: Marcor
2 Replies

4. Shell Programming and Scripting

Normalize a dataset with AWK

Hello everyone, i have to normalize this dataset (with 20.000 rows): 2,4,4,3,2,7,8,2,9,11,7,7,1,8,5,6 4,7,5,5,5,5,9,6,4,8,7,9,2,9,7,10 7,10,8,7,4,8,8,5,10,11,2,8,2,5,5,10 4,9,5,7,4,7,7,13,1,7,6,8,3,8,0,8,8 6,7,8,5,4,7,6,3,7,10,7,9,3,8,3,7,8 in this form:... (1 Reply)
Discussion started by: [raven]
1 Replies

5. Programming

Dataset Library for C?

I am looking for an opensource dataset library for C. Something equivalent to ADO.Net. Specifically, I am looking for the following features: 1. Create a Dataset from a file (XML or CSV). 2. Create a Dataset from a select query using an ODBC connection. 3. Load a created Dataset into a... (1 Reply)
Discussion started by: a_programmer
1 Replies

6. Solaris

flarecreate for zfs root dataset and ignore multiple dataset

Hi All, I want to write a script to create flar images on multiple servers. In non zfs filesystem I am using -X option to refer a file to exclude mounts on different servers. but on ZFS -X option is not working. I want multiple mounts to be ignore on ZFS base system during flarecreate. I... (0 Replies)
Discussion started by: uxravi
0 Replies

7. Shell Programming and Scripting

How to remove a subset of data from a large dataset based on values on one line

Hello. I was wondering if anyone could help. I have a file containing a large table in the format: marker1 marker2 marker3 marker4 position1 position2 position3 position4 genotype1 genotype2 genotype3 genotype4 with marker being a name, position a numeric... (2 Replies)
Discussion started by: davegen
2 Replies

8. UNIX for Advanced & Expert Users

How to extract subset file from dataset?

Hello I have a data set which looks like this : progeny sire dam gender 12 1 3 M 13 2 4 F 14 2 5 F 15 6 5 ... (13 Replies)
Discussion started by: sajmar
13 Replies

9. Shell Programming and Scripting

Extract few content from a huge list of files

I have a huge list of files (about 300,000) which have a pattern like this. .I 1 .U 87049087 .S Am J Emerg .M Allied Health Personnel/*; Electric Countershock/*; .T Refibrillation managed by EMT-Ds: .P ARTICLE. .W Some patients converted from ventricular fibrillation to organized... (1 Reply)
Discussion started by: shoaibjameel123
1 Replies

10. UNIX for Advanced & Expert Users

SAS dataset to CSV

Hi Guys, Is there a way to export a sas file i.e .sas7bdat file to .csv file with header and data using unix. I dont want to use SAS program instead using unix tool or unix scripting is it possible ? (25 Replies)
Discussion started by: Master_Mind
25 Replies
tabs(1) 						      General Commands Manual							   tabs(1)

NAME
tabs - set tabs on a terminal SYNOPSIS
[tabspec] n] type] DESCRIPTION
sets the tab stops on the user's terminal according to the tab specification tabspec, after clearing any previous settings. The user's terminal must have remotely-settable hardware tabs. If you are using a non-HP terminal, you should keep in mind that behavior will vary for some tab settings. Four types of tab specification are accepted for tabspec: ``canned'', repetitive, arbitrary, and file. If no is given, the default value is i.e., UNIX ``standard'' tabs. The lowest column number is 1. Note that for tabs, column 1 always refers to the left-most column on a terminal, even one whose column markers begin at 0. Gives the name of one of a set of ``canned'' tabs. Recognized codes and their meanings are as follows: 1,10,16,36,72 Assembler, IBM S/370, first format 1,10,16,40,72 Assembler, IBM S/370, second format 1,8,12,16,20,55 COBOL, normal format 1,6,10,14,49 COBOL compact format (columns 1-6 omitted). Using this code, the first typed character corresponds to card column 7, one space gets you to column 8, and a tab reaches column 12. Files using this tab setup should have specify a format specification file as defined by below. The file should have the following format specification: 1,6,10,14,18,22,26,30,34,38,42,46,50,54,58,62,67 COBOL compact format (columns 1-6 omitted), with more tabs than This is the recommended format for COBOL. The appro- priate format specification is: 1,7,11,15,19,23 FORTRAN 1,5,9,13,17,21,25,29,33,37,41,45,49,53,57,61 PL/I 1,10,55 SNOBOL 1,12,20,44 UNIVAC 1100 Assembler In addition to these ``canned'' formats, three other types exist: A repetitive specification requests tabs at columns 1+n, 1+2xn, etc. Of particular importance is the value this represents the UNIX ``standard'' tab setting, and is the most likely tab setting to be found at a terminal. Another special case is the value implying no tabs at all. The arbitrary format permits the user to type any chosen set of numbers, separated by commas, in ascending order. Up to 40 numbers are allowed. If any number (except the first one) is preceded by a plus sign, it is taken as an increment to be added to the previous value. Thus, the tab lists 1,10,20,30 and 1,10,+10,+10 are considered identical. If the name of a file is given, reads the first line of the file, searching for a format specification. If it finds one there, it sets the tab stops according to it, otherwise it sets them as This type of specification can be used to ensure that a tabbed file is printed with correct tab settings, and is suitable for use with the command (see pr(1)): Any of the following can be used also; if a given option occurs more than once, the last value given takes effect: usually needs to know the type of terminal in order to set tabs and always needs to know the type to set margins. type is a name listed in term(5). If no option is supplied, searches for the value in the environment (see environ(5)). If is not defined in the environment, tries a sequence that will work for many terminals. The margin argument can be used for some terminals. It causes all tabs to be moved over n columns by making column n+1 the left margin. If is given without a value of n, the value assumed is 10. The normal (left-most) margin on most terminals is obtained by The margin for most terminals is reset only when the option is given explicitly. Tab and margin setting is performed via the standard output. EXTERNAL INFLUENCES
Environment Variables determines the interpretation of text within file as single- and/or multi-byte characters. determines the language in which messages are displayed. If or is not specified in the environment or is set to the empty string, the value of is used as a default for each unspecified or empty variable. If is not specified or is set to the empty string, a default of "C" (see lang(5)) is used instead of If any internationalization variable contains an invalid setting, behaves as if all internationalization variables are set to "C". See environ(5). International Code Set Support Single- and multi-byte character code sets are supported. DIAGNOSTICS
Arbitrary tabs are ordered incorrectly. A zero or missing increment found in an arbitrary specification. A ``canned'' code cannot be found. option was used and file cannot be opened. option was used and the specification in that file points to yet another file. Indirection of this form is not permitted. WARNINGS
There is no consistency among different terminals regarding ways of clearing tabs and setting the left margin. It is generally impossible to usefully change the left margin without also setting tabs. clears only 20 tabs (on terminals requiring a long sequence), but is willing to set 64. SEE ALSO
nroff(1), pr(1), tset(1), environ(5), term(5). STANDARDS CONFORMANCE
tabs(1)
All times are GMT -4. The time now is 08:55 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy