Good evening, everybody
A good math friend told me that it would be possible to shrink the size of the numerical datas I produce with a physical simulation code I programmed for my PhD.
It usually writes at least 100 GB to complete the simulation, and it seems that it is too high. There are some quotas to respect, and I have been told that it would possible to use "binary" datas instead of ASCII datas.
Here are the inputs of the problem.
- I calculate and produce the datas using a C/C++ simulation code.
- I make post treatments using bash and awk.
- I make plots using GNUplot 4.4 [splot works great with pm3d now
Have a look to "Not so frequently asked question" website to be aware of GNUplot possibilities]
I found how to use binary mode in Gawk and GNUplot, but the main point is missing: which conversion should I make to decrease the volume of the datas in a loseless way ? They are all numerical datas, then I would convert double precision number coded in ASCII (visible with "more" command) into "binary" (which I assume to be a language abuse, because converting datafile into binary using "od" command just multiply the size of the initial file... :-( ).
Example. Each line of my generated main data file is:
3000 -3.9e-13 -4.24661e-05 0 299.964 300 1.50018e+16 1.50005e+16 0 00 1 0 0
What conversion do you recommand to optimize space needed ?
By writing this post, I feel that I should convert double precision numbers coded with characters in a ascii file into double precision datas coded with numbers only. How can I do that? "od" command is sufficient ?
Glad of any help,
Cheers from France,
Thibault