Sponsored Content
Top Forums UNIX for Advanced & Expert Users Conversion from ASCII to binary for physical simulation code in C/C++ Post 302507396 by Cybertib on Wednesday 23rd of March 2011 05:59:37 PM
Old 03-23-2011
Conversion from ASCII to binary for physical simulation code in C/C++

Good evening, everybody

A good math friend told me that it would be possible to shrink the size of the numerical datas I produce with a physical simulation code I programmed for my PhD.

It usually writes at least 100 GB to complete the simulation, and it seems that it is too high. There are some quotas to respect, and I have been told that it would possible to use "binary" datas instead of ASCII datas.

Here are the inputs of the problem.
- I calculate and produce the datas using a C/C++ simulation code.
- I make post treatments using bash and awk.
- I make plots using GNUplot 4.4 [splot works great with pm3d now Smilie Have a look to "Not so frequently asked question" website to be aware of GNUplot possibilities]

I found how to use binary mode in Gawk and GNUplot, but the main point is missing: which conversion should I make to decrease the volume of the datas in a loseless way ? They are all numerical datas, then I would convert double precision number coded in ASCII (visible with "more" command) into "binary" (which I assume to be a language abuse, because converting datafile into binary using "od" command just multiply the size of the initial file... :-( ).

Example. Each line of my generated main data file is:
3000 -3.9e-13 -4.24661e-05 0 299.964 300 1.50018e+16 1.50005e+16 0 00 1 0 0

What conversion do you recommand to optimize space needed ?

By writing this post, I feel that I should convert double precision numbers coded with characters in a ascii file into double precision datas coded with numbers only. How can I do that? "od" command is sufficient ?

Glad of any help,

Cheers from France,
Thibault

Last edited by Cybertib; 03-23-2011 at 07:18 PM.. Reason: added example
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Convert ASCII to BINARY

Here is what I did . . . . I FTP'd several *.pdf files from a web site to a UNIX server, and did not set the transfer mode to BIN, now Adobe thinks that the documents are corrupted. Is there a way to convert the *.pdf files to Binary so that Adobe can open them again. I would just re-download... (2 Replies)
Discussion started by: pc9456
2 Replies

2. Shell Programming and Scripting

ascii conversion

after converting my ebcidic file to ascii i get the following output 2097152+0 records in 1797345+1 records out Why is there a difference in number of records. Is the converson chopping off any records. All i am doing is just a conversion using the following script dd if=xaa cbs=152 ... (0 Replies)
Discussion started by: rintingtong
0 Replies

3. SCO

ascii to binary conversion in sco 5.0.5

Here is what I did . . . . I FTP'd several *.dbf zipped files from a SCO 5.0.5 server to winXP machine, and did not set the transfer mode to BIN, now when i was uncompressing these files in SCO 5.0.5 , it was giving "Bad Decode Table error. Is there a way to convert the *.dbf.Z files to Binary so... (1 Reply)
Discussion started by: sameek1211
1 Replies

4. Shell Programming and Scripting

binary to ascii

Hi, Is there a way to convert the binary file to ascii . the binary file is pipe delimited. from source the file(pipe delimited) is ftped to mainframe and from mainframe it is ftped to the unix box using binary format. Is there a way to change it back to ascii and view it? Thanks! (3 Replies)
Discussion started by: dnat
3 Replies

5. UNIX for Dummies Questions & Answers

Ascii or Binary?

Hello all, I am working with ftp servers in unix, and always I have to get and put files but I don't know exactly if I have to get or put them as an ascii or binary. Some files that I use are: .txt, .sav, .fmb, .pct, .sh, .ksh, .dat, .log. Somebody can tell me what is the difference between... (2 Replies)
Discussion started by: Geller
2 Replies

6. Shell Programming and Scripting

binary to ascii conversion

Hi, I have got a library file, created by compiling C code. The file information with "file" command, gives it a "application/x-archive" type file. I want to extract the release string of my software from this file, so that i can know which version of C files were used to create the lib. Can... (3 Replies)
Discussion started by: atulmt
3 Replies

7. Solaris

EBCDIC to ASCII Binary conversion issue on Solaris i-series Unix

Hi All, I am facing EBCDIC to ASCII Binary conversion on Solaris i-series Unix system. However this is working fine on Solaris Sparc Unix system. Input file having EBCDIC format does not work on Solaris i-series Unix system. Could you please tell me, what will be the root cause for same? (14 Replies)
Discussion started by: amodkavi
14 Replies

8. Shell Programming and Scripting

Bash - binary data to ascii code

Hello, With bash-script (ubunto server) I'm trying to read a binary file and, for each character, give back its ascii code (including extended ascii). For example: HEX => ASCII => PRINT f5 => 245 => õ 50 => 80 => P To load the binary file into a variable I tried in this way: ... (2 Replies)
Discussion started by: math4
2 Replies

9. Solaris

ASN Binary to ASCII

Dears, I need help to convert the binary file into ASCII format. Actually we have CDRs which is generated by telecom switch at this is in ASN1 format or binary format, I need to convert those binary formatted file into ASCII format using Perl, or shell scripting. Is there any way to solve... (3 Replies)
Discussion started by: PRINCESS_RORO
3 Replies

10. Shell Programming and Scripting

File conversion from Binary to ASCII though UNIX command

Hi All , I have a mainframe file which contains the data in EBCDIC format.I have downloaded this file from mainframe to windows in binary format(unreadable raw data).Now I want convert this file to ASCII format(readable format data) through Unix command.I have tried iconv but that is not working... (2 Replies)
Discussion started by: STCET22
2 Replies
simulation::montecarlo(3tcl)				       Tcl Simulation Tools				      simulation::montecarlo(3tcl)

__________________________________________________________________________________________________________________________________________________

NAME
simulation::montecarlo - Monte Carlo simulations SYNOPSIS
package require Tcl ?8.4? package require simulation::montecarlo 0.1 package require simulation::random package require math::statistics ::simulation::montecarlo::getOption keyword ::simulation::montecarlo::hasOption keyword ::simulation::montecarlo::setOption keyword value ::simulation::montecarlo::setTrialResult values ::simulation::montecarlo::setExpResult values ::simulation::montecarlo::getTrialResults ::simulation::montecarlo::getExpResult ::simulation::montecarlo::transposeData values ::simulation::montecarlo::integral2D ... ::simulation::montecarlo::singleExperiment args _________________________________________________________________ DESCRIPTION
The technique of Monte Carlo simulations is basically simple: o generate random values for one or more parameters. o evaluate the model of some system you are interested in and record the interesting results for each realisation of these parameters. o after a suitable number of such trials, deduce an overall characteristic of the model. You can think of a model of a network of computers, an ecosystem of some kind or in fact anything that can be quantitatively described and has some stochastic element in it. The package simulation::montecarlo offers a basic framework for such a modelling technique: # # MC experiments: # Determine the mean and median of a set of points and compare them # ::simulation::montecarlo::singleExperiment -init { package require math::statistics set prng [::simulation::random::prng_Normal 0.0 1.0] } -loop { set numbers {} for { set i 0 } { $i < [getOption samples] } { incr i } { lappend numbers [$prng] } set mean [::math::statistics::mean $numbers] set median [::math::statistics::median $numbers] ;# ? Exists? setTrialResult [list $mean $median] } -final { set result [getTrialResults] set means {} set medians {} foreach r $result { foreach {m M} $r break lappend means $m lappend medians $M } puts [getOption reportfile] "Correlation: [::math::statistics::corr $means $medians]" } -trials 100 -samples 10 -verbose 1 -columns {Mean Median} This example attemps to find out how well the median value and the mean value of a random set of numbers correlate. Sometimes a median value is a more robust characteristic than a mean value - especially if you have a statistical distribution with "fat" tails. PROCEDURES
The package defines the following auxiliary procedures: ::simulation::montecarlo::getOption keyword Get the value of an option given as part of the singeExperiment command. string keyword Given keyword (without leading minus) ::simulation::montecarlo::hasOption keyword Returns 1 if the option is available, 0 if not. string keyword Given keyword (without leading minus) ::simulation::montecarlo::setOption keyword value Set the value of the given option. string keyword Given keyword (without leading minus) string value (New) value for the option ::simulation::montecarlo::setTrialResult values Store the results of the trial for later analysis list values List of values to be stored ::simulation::montecarlo::setExpResult values Set the results of the entire experiment (typically used in the final phase). list values List of values to be stored ::simulation::montecarlo::getTrialResults Get the results of all individual trials for analysis (typically used in the final phase or after completion of the command). ::simulation::montecarlo::getExpResult Get the results of the entire experiment (typically used in the final phase or even after completion of the singleExperiment com- mand). ::simulation::montecarlo::transposeData values Interchange columns and rows of a list of lists and return the result. list values List of lists of values There are two main procedures: integral2D and singleExperiment. ::simulation::montecarlo::integral2D ... Integrate a function over a two-dimensional region using a Monte Carlo approach. Arguments PM ::simulation::montecarlo::singleExperiment args Iterate code over a number of trials and store the results. The iteration is gouverned by parameters given via a list of keyword- value pairs. int n List of keyword-value pairs, all of which are available during the execution via the getOption command. The singleExperiment command predefines the following options: o -init code: code to be run at start up o -loop body: body of code that defines the computation to be run time and again. The code should use setTrialResult to store the results of each trial (typically a list of numbers, but the interpretation is up to the implementation). Note: Required keyword. o -final code: code to be run at the end o -trials n: number of trials in the experiment (required) o -reportfile file: opened file to send the output to (default: stdout) o -verbose: write the intermediate results(1) or not(0) (default: 0) o -analysis proc: either "none" (no automatic analysis), standard (basic statistics of the trial results and a correlation matrix) or the name of a procedure that will take care of the analysis. o -columns list: list of column names, useful for verbose output and the analysis Any other options can be used via the getOption procedure in the body. TIPS
The procedure singleExperiment works by constructing a temporary procedure that does the actual work. It loops for the given number of tri- als. As it constructs a temporary procedure, local variables defined at the start continue to exist in the loop. KEYWORDS
math, montecarlo simulation, stochastic modelling CATEGORY
Mathematics COPYRIGHT
Copyright (c) 2008 Arjen Markus <arjenmarkus@users.sourceforge.net> simulation 0.1 simulation::montecarlo(3tcl)
All times are GMT -4. The time now is 08:30 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy