Sponsored Content
Top Forums Shell Programming and Scripting Unique values from a Terabyte File Post 302246874 by Legend986 on Tuesday 14th of October 2008 03:23:50 PM
Old 10-14-2008
Oh.. this is a text file too with a bunch of numbers from a network simulation experiment... I was thinking of actually splitting the file and getting the job done, but was just curious if there are better ways of doing things like matrixmadhan expressed....

Last edited by Legend986; 10-14-2008 at 09:47 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Getting Unique values in a file

Hi, I have a file like this: Some_String_Here 123 123 123 321 321 321 3432 3221 557 886 321 321 I would like to find only the unique values in the files and get the following output: Some_String_Here 123 321 3432 3221 557 886 I am trying to get this done using awk. Can someone please... (5 Replies)
Discussion started by: Legend986
5 Replies

2. UNIX Desktop Questions & Answers

Fetching unique values from file

After giving grep -A4 "feature 1," <file name> I have extracted the following text feature 1, subfeat 2, type 1, subtype 5, dump '30352f30312f323030392031313a33303a3337'H -- "05/01/2009 11:30:37" -- -- ... (1 Reply)
Discussion started by: shivi707
1 Replies

3. UNIX for Dummies Questions & Answers

Extract Unique Values from file

Hello all, I have a file with following sample data 2009-08-26 05:32:01.65 spid5 Process ID 86:214 owns resources that are blocking processes on Scheduler 0. 2009-08-26 05:32:01.65 spid5 Process ID 86:214 owns resources that are blocking processes on Scheduler 0. 2009-08-26... (5 Replies)
Discussion started by: simonsimon
5 Replies

4. Shell Programming and Scripting

How to count Unique Values from a file.

Hi I have the following info in a file - <Cell id="25D"/> <Cell id="26A"/> <Cell id="26B"/> <Cell id="26C"/> <Cell id="27A"/> <Cell id="27B"/> <Cell id="27C"/> <Cell id="28A"/> I would like to know how would you go about counting all... (4 Replies)
Discussion started by: Prega
4 Replies

5. Shell Programming and Scripting

List unique values and count instances in .csv file

I need to take the second column of a .csv file and count the number of instances of each unique value in that same second column. I'd like the output to be value,count sorted by most instances. Thanks for any guidance! Data example: 317476,317756,0 816063,318861,0 313123,319091,0... (4 Replies)
Discussion started by: batcho
4 Replies

6. Shell Programming and Scripting

Find and count unique date values in a file based on position

Hello, I need some sort of way to extract every date contained in a file, and count how many of those dates there are. Here are the specifics: The date format I'm looking for is mm/dd/yyyy I only need to look after line 45 in the file (that's where the data begins) The columns of... (2 Replies)
Discussion started by: ronan1219
2 Replies

7. Linux

To get all the columns in a CSV file based on unique values of particular column

cat sample.csv ID,Name,no 1,AAA,1 2,BBB,1 3,AAA,1 4,BBB,1 cut -d',' -f2 sample.csv | sort | uniq this gives only the 2nd column values Name AAA BBB How to I get all the columns of CSV along with this? (1 Reply)
Discussion started by: sanvel
1 Replies

8. Shell Programming and Scripting

Extracting unique values of a column from a feed file

Hi Folks, I have the below feed file named abc1.txt in which you can see there is a title and below is the respective values in the rows and it is completely pipe delimited file ,. ... (4 Replies)
Discussion started by: punpun66
4 Replies

9. Shell Programming and Scripting

Using grep and a parameter file to return unique values

Hello Everyone! I have updated the first post so that my intentions are easier to understand, and also attached sample files (post #18). I have over 500 text files in a directory. Over 1 GB of data. The data in those files is organised in lines: My intention is to return one line per... (23 Replies)
Discussion started by: clippertm
23 Replies

10. Shell Programming and Scripting

How to identify varying unique fields values from a text file in UNIX?

Hi, I have a huge unsorted text file. We wanted to identify the unique field values in a line and consider those fields as a primary key for a table in upstream system. Basically, the process or script should fetch the values from each line that are unique compared to the rest of the lines in... (13 Replies)
Discussion started by: manikandan23
13 Replies
simulation::montecarlo(n)				       Tcl Simulation Tools					 simulation::montecarlo(n)

__________________________________________________________________________________________________________________________________________________

NAME
simulation::montecarlo - Monte Carlo simulations SYNOPSIS
package require Tcl ?8.4? package require simulation::montecarlo 0.1 package require simulation::random package require math::statistics ::simulation::montecarlo::getOption keyword ::simulation::montecarlo::hasOption keyword ::simulation::montecarlo::setOption keyword value ::simulation::montecarlo::setTrialResult values ::simulation::montecarlo::setExpResult values ::simulation::montecarlo::getTrialResults ::simulation::montecarlo::getExpResult ::simulation::montecarlo::transposeData values ::simulation::montecarlo::integral2D ... ::simulation::montecarlo::singleExperiment args _________________________________________________________________ DESCRIPTION
The technique of Monte Carlo simulations is basically simple: o generate random values for one or more parameters. o evaluate the model of some system you are interested in and record the interesting results for each realisation of these parameters. o after a suitable number of such trials, deduce an overall characteristic of the model. You can think of a model of a network of computers, an ecosystem of some kind or in fact anything that can be quantitatively described and has some stochastic element in it. The package simulation::montecarlo offers a basic framework for such a modelling technique: # # MC experiments: # Determine the mean and median of a set of points and compare them # ::simulation::montecarlo::singleExperiment -init { package require math::statistics set prng [::simulation::random::prng_Normal 0.0 1.0] } -loop { set numbers {} for { set i 0 } { $i < [getOption samples] } { incr i } { lappend numbers [$prng] } set mean [::math::statistics::mean $numbers] set median [::math::statistics::median $numbers] ;# ? Exists? setTrialResult [list $mean $median] } -final { set result [getTrialResults] set means {} set medians {} foreach r $result { foreach {m M} $r break lappend means $m lappend medians $M } puts [getOption reportfile] "Correlation: [::math::statistics::corr $means $medians]" } -trials 100 -samples 10 -verbose 1 -columns {Mean Median} This example attemps to find out how well the median value and the mean value of a random set of numbers correlate. Sometimes a median value is a more robust characteristic than a mean value - especially if you have a statistical distribution with "fat" tails. PROCEDURES
The package defines the following auxiliary procedures: ::simulation::montecarlo::getOption keyword Get the value of an option given as part of the singeExperiment command. string keyword Given keyword (without leading minus) ::simulation::montecarlo::hasOption keyword Returns 1 if the option is available, 0 if not. string keyword Given keyword (without leading minus) ::simulation::montecarlo::setOption keyword value Set the value of the given option. string keyword Given keyword (without leading minus) string value (New) value for the option ::simulation::montecarlo::setTrialResult values Store the results of the trial for later analysis list values List of values to be stored ::simulation::montecarlo::setExpResult values Set the results of the entire experiment (typically used in the final phase). list values List of values to be stored ::simulation::montecarlo::getTrialResults Get the results of all individual trials for analysis (typically used in the final phase or after completion of the command). ::simulation::montecarlo::getExpResult Get the results of the entire experiment (typically used in the final phase or even after completion of the singleExperiment com- mand). ::simulation::montecarlo::transposeData values Interchange columns and rows of a list of lists and return the result. list values List of lists of values There are two main procedures: integral2D and singleExperiment. ::simulation::montecarlo::integral2D ... Integrate a function over a two-dimensional region using a Monte Carlo approach. Arguments PM ::simulation::montecarlo::singleExperiment args Iterate code over a number of trials and store the results. The iteration is gouverned by parameters given via a list of keyword- value pairs. int n List of keyword-value pairs, all of which are available during the execution via the getOption command. The singleExperiment command predefines the following options: o -init code: code to be run at start up o -loop body: body of code that defines the computation to be run time and again. The code should use setTrialResult to store the results of each trial (typically a list of numbers, but the interpretation is up to the implementation). Note: Required keyword. o -final code: code to be run at the end o -trials n: number of trials in the experiment (required) o -reportfile file: opened file to send the output to (default: stdout) o -verbose: write the intermediate results (1) or not (0) (default: 0) o -analysis proc: either "none" (no automatic analysis), standard (basic statistics of the trial results and a correlation matrix) or the name of a procedure that will take care of the analysis. o -columns list: list of column names, useful for verbose output and the analysis Any other options can be used via the getOption procedure in the body. TIPS
The procedure singleExperiment works by constructing a temporary procedure that does the actual work. It loops for the given number of tri- als. As it constructs a temporary procedure, local variables defined at the start continue to exist in the loop. KEYWORDS
math, montecarlo simulation, stochastic modelling CATEGORY
Mathematics COPYRIGHT
Copyright (c) 2008 Arjen Markus <arjenmarkus@users.sourceforge.net> simulation 0.1 simulation::montecarlo(n)
All times are GMT -4. The time now is 01:01 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy