Making things run faster Post: 302248404

Sponsored Content

Top Forums UNIX for Advanced & Expert Users Making things run faster Post 302248404 by Legend986 on Friday 17th of October 2008 02:05:51 PM

10-17-2008

Registered User

Hmm... that's an interesting idea! Smilie

I'd love to try that out... Actually by the time I woke up, it was able to process some 1 TB (so that makes it 7 hours). I don't know how to formally write down the whole thing but I'll try:

Total Data Size: 2.2 TB (currently handling around 1TB though)
Special Info: The formats of the data were slightly different. There were a total of four data sets (lets call them DS):
DS1 & DS2: Format 1

Size of DS1: 556G RAW
Size of DS2: 105G Gzip Compressed

DS3 & DS4: Format 2

Size of DS3: 157G RAW
Size of DS4: 109G RAW

Further, there were two other tasks (extracting a 36G archive and copying some 10G worth data) running handled by a different processor (another computer in fact) on the disk of this main computer.

Tasks running Simultaneously:

Parsing DS1, DS2, DS3, DS4 and writing the result onto disk again
Extracting an archive on the same disk using a different computer on which the disk is mounted as a remote drive
Copying the gzipped files back into the main computer(if anyone has seen my other threads, yes, in fact these were the huge archives I was talking about converting into individual smaller archives )

As of now, I have finished parsing DS1, DS2, DS3 and DS4 but I am left with extracting the huge archives and then parsing the last DS5 which will be around 1.7TB uncompressed. I will perhaps run the optimization then.

Thanks for the advice and looking forward for a post from you.

Legend986

View Public Profile for Legend986

Find all posts by Legend986

8 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

making ssh run without password

Hello Everybody, Could anyone please tell me how to get ssh to work without asking for passwords? (i want to do a ssh <hostname> without getting a request for a password but getting connected straight away) I have attempted the following but to no avail :( ... I tried to generate a SSH...

2. Programming

Complicating things?

So basically what im trying to do is ... Open file, read that file, than try to find .. We or we and replace them with I, but not replace the cases where words contain We or we, such as Went, went, etc a and replace them with the, but not replace the cases where words contain a, such as...

3. Shell Programming and Scripting

When things doesn't run into crontab???

Could someone explain my problem? I've the following script... #! /bin/ksh ... vmquery -m $MediaID | awk ' BEGIN {FS=": " getline expdate <"ExpDate.txt" } $1 ~ /media ID/ {MediaNumber = $NF} ... $1 ~ /number of mounts/ { "date +%Y"|getline YearToday Year4 = YearToday - 4 if...

4. Shell Programming and Scripting

Can anyone make this script run faster?

One of our servers runs Solaris 8 and does not have "ls -lh" as a valid command. I wrote the following script to make the ls output easier to read and emulate "ls -lh" functionality. The script works, but it is slow when executed on a directory that contains a large number of files. Can anyone make...

5. UNIX for Dummies Questions & Answers

things root can't do

Hey all my co-workers and I are trying to put together a list of things root "Can't" do on any *NIX OS, so I wanted to come here and see what all we could come up with. Here are two to start this off: write to a read only mount FS kill a tape rewind Please add what you know. Thanks,...

6. Shell Programming and Scripting

Making script run faster

Can someone help me edit the below script to make it run faster? Shell: bash OS: Linux Red Hat The point of the script is to grab entire chunks of information that concerns the service "MEMORY_CHECK". For each chunk, the beginning starts with "service {", and ends with "}". I should...

7. Shell Programming and Scripting

Making a faster alternative to a slow awk command

Hi, I have a large number of input files with two columns of numbers. For example: 83 1453 99 3255 99 8482 99 7372 83 175 I only wish to retain lines where the numbers fullfil two requirements. E.g: =83 1000<=<=2000 To do this I use the following...

8. Shell Programming and Scripting

Optimize shell script to run faster

data.file: contact { contact_name=royce-rolls modified_attributes=0 modified_host_attributes=0 modified_service_attributes=0 host_notification_period=24x7 service_notification_period=24x7 last_host_notification=0 last_service_notification=0 host_notifications_enabled=1...

LEARN ABOUT OSX

play_sample

play_sample(3alleg4)						  Allegro manual					      play_sample(3alleg4)

NAME

       play_sample - Plays a sample. Allegro game programming library.

SYNOPSIS

       #include <allegro.h>

       int play_sample(const SAMPLE *spl, int vol, int pan, int freq, int loop);

DESCRIPTION

       Triggers  a  sample  at	the  specified volume, pan position, and frequency.  The parameters `vol' and `pan' range from 0 (min/left) to 255
       (max/right).  Frequency is relative rather than absolute: 1000 represents the frequency that the sample was  recorded  at,  2000  is  twice
       this,  etc. If `loop' is not zero, the sample will repeat until you call stop_sample(), and can be manipulated while it is playing by call-
       ing adjust_sample(). Example:

	  /* Scream from the left speaker, twice the freq. */
	  int sound = play_sample(scream, 255, 0, 2000, 0);

RETURN VALUE

       Returns the voice number that was allocated for the sample or negative if no voices were available.

SEE ALSO

       install_sound(3alleg4), load_sample(3alleg4), adjust_sample(3alleg4), stop_sample(3alleg4), exsample(3alleg4), exsprite(3alleg4)

Allegro 							   version 4.4.2					      play_sample(3alleg4)