How to remove a subset of data from a large dataset based on values on one line Post: 302576002

10 More Discussions You Might Find Interesting

1. Programming

I have C++ exe file( no source code) and need to run many large dataset under unix, b

I have C++ exe file( no source code) and need to run many large dataset under unix, but how to know the memeroy usage for one dataset?http://www.codeproject.com/script/Forums/Images/New.gif I think "top" is not good and if using the profiler, it seems no free download, any ideas?

2. Shell Programming and Scripting

remove a specific line in a LARGE file

Hi guys, i have a really big file, and i want to remove a specific line. sed -i '5d' fileThis doesn't really work, it takes a lot of time... The whole script is supposed to remove every word containing less than 5 characters and currently looks like this: #!/bin/bash line="1"...

3. Shell Programming and Scripting

Remove duplicate line detail based on column one data

My input file: AVI.out <detail>named as the RRM .</detail> AVI.out <detail>Contains 1 RRM .</detail> AR0.out <detail>named as the tellurite-resistance.</detail> AWG.out <detail>Contains 2 HTH .</detail> ADV.out <detail>named as the DENR family.</detail> ADV.out ...

4. Shell Programming and Scripting

How to extract a subset from a huge dataset

Hi, All I have a huge file which has 450G. Its tab-delimited format is as below x1 A 50020 1 x1 B 50021 8 x1 C 50022 9 x1 A 50023 10 x2 D 50024 5 x2 C 50025 7 x2 F 50026 8 x2 N 50027 1 : : Now, I want to extract a subset from this file. In this subset, column 1 is x10, column 2 is...

5. Shell Programming and Scripting

Find line number of bad data in large file

Hi Forum. I was trying to search the following scenario on the forum but was not able to. Let's say that I have a very large file that has some bad data in it (for ex: 0.0015 in the 12th column) and I would like to find the line number and remove that particular line. What's the easiest...

6. UNIX for Advanced & Expert Users

How to extract subset file from dataset?

Hello I have a data set which looks like this : progeny sire dam gender 12 1 3 M 13 2 4 F 14 2 5 F 15 6 5 ...

7. Shell Programming and Scripting

How to read file line by line and compare subset of 1st line with 2nd?

8. Shell Programming and Scripting

Selecting random columns from large dataset in UNIX

Dear folks I have a large data set which contains 400K columns. I decide to select 50K determined columns from the whole 400K columns. Is there any command in unix which could do this process for me? I need to also mention that I store all of the columns id in one file which may help to select...

9. Shell Programming and Scripting

Reoccuring peak values in large data file and print the line..

Hi i have some large data files that contain several fields and rows the data in a field have a numeric value that is in a sine wave pattern what i would like todo is locate each peak and pick the highest value and print that complete line. the data looks something like this it is field nr4 which...

10. Shell Programming and Scripting

Parsing a subset of data from a large matrix

I do have a large matrix of the following format and it is tab delimited ch-ab1-20 ch-bb2-23 ch-ab1-34 ch-ab1-24 er-cc1-45 bv-cc1-78 ch-ab1-20 0 2 3 4 5 6 ch-bb2-23 3 0 5 ...

LEARN ABOUT DEBIAN

svm-subset

svm-subset(1)							   User Manuals 						     svm-subset(1)

NAME

       svm-subset - a subset selection tool for LIBSVM

SYNOPSIS

       svm-subset [ -s method ] dataset number [ output1 ] [ output2 ]

DESCRIPTION

       Training  large data is time consuming. Sometimes one should work on a smaller subset first. The python script subset.py randomly selects a
       specified number of samples. For classification data, we provide a stratified selection to ensure the same class distribution in  the  sub-
       set.

OPTIONS

       -s method

       0      -- stratified selection (classification only) (default)

       1      -- random selection

       output1
	      The subset. If output1 is omitted, the subset will be printed on the screen.

       output2
	      The rest of data.

FILES

       See svm-train(1) for the format of dataset

EXAMPLES

	      svm-subset heart_scale 100 file1 file2

       From heart_scale 100 samples are randomly selected and stored in file1. All remaining instances are stored in file2.

BUGS

       Please report bugs to the Debian BTS.

AUTHOR

       Chih-Chung Chang, Chih-Jen Lin <cjlin@csie.ntu.edu.tw>, Chen-Tse Tsai <ctse.tsai@gmail.com> (packaging)

SEE ALSO

       svm-train(1), svm-predict(1)

Linux								     DEC 2009							     svm-subset(1)