|
|
Sponsored Content
Top Forums
UNIX for Advanced & Expert Users
Making things run faster
Post 302248404 by Legend986 on Friday 17th of October 2008 02:05:51 PM
10-17-2008
Hmm... that's an interesting idea! 
I'd love to try that out... Actually by the time I woke up, it was able to process some 1 TB (so that makes it 7 hours). I don't know how to formally write down the whole thing but I'll try:
Total Data Size: 2.2 TB (currently handling around 1TB though)
Special Info: The formats of the data were slightly different. There were a total of four data sets (lets call them DS):
DS1 & DS2: Format 1
DS3 & DS4: Format 2
Further, there were two other tasks (extracting a 36G archive and copying some 10G worth data) running handled by a different processor (another computer in fact) on the disk of this main computer.
Tasks running Simultaneously:
As of now, I have finished parsing DS1, DS2, DS3 and DS4 but I am left with extracting the huge archives and then parsing the last DS5 which will be around 1.7TB uncompressed. I will perhaps run the optimization then.
Thanks for the advice and looking forward for a post from you.
I'd love to try that out... Actually by the time I woke up, it was able to process some 1 TB (so that makes it 7 hours). I don't know how to formally write down the whole thing but I'll try:Total Data Size: 2.2 TB (currently handling around 1TB though)
Special Info: The formats of the data were slightly different. There were a total of four data sets (lets call them DS):
DS1 & DS2: Format 1
- Size of DS1: 556G RAW
- Size of DS2: 105G Gzip Compressed
DS3 & DS4: Format 2
- Size of DS3: 157G RAW
- Size of DS4: 109G RAW
Further, there were two other tasks (extracting a 36G archive and copying some 10G worth data) running handled by a different processor (another computer in fact) on the disk of this main computer.
Tasks running Simultaneously:
- Parsing DS1, DS2, DS3, DS4 and writing the result onto disk again
- Extracting an archive on the same disk using a different computer on which the disk is mounted as a remote drive
- Copying the gzipped files back into the main computer(if anyone has seen my other threads, yes, in fact these were the huge archives I was talking about converting into individual smaller archives )
As of now, I have finished parsing DS1, DS2, DS3 and DS4 but I am left with extracting the huge archives and then parsing the last DS5 which will be around 1.7TB uncompressed. I will perhaps run the optimization then.
Thanks for the advice and looking forward for a post from you.
|
|
8 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hello Everybody,
Could anyone please tell me how to get ssh to work without asking for passwords? (i want to do a ssh <hostname> without getting a request for a password but getting connected straight away)
I have attempted the following but to no avail :( ...
I tried to generate a SSH... (5 Replies)
Discussion started by: rkap
5 Replies
2. Programming
So basically what im trying to do is ...
Open file, read that file, than try to find ..
We or we and replace them with I, but not replace the cases where words contain We or we, such as Went, went, etc
a and replace them with the, but not replace the cases where words contain a, such as... (1 Reply)
Discussion started by: bconnor
1 Replies
3. Shell Programming and Scripting
Could someone explain my problem?
I've the following script...
#! /bin/ksh
...
vmquery -m $MediaID | awk '
BEGIN {FS=": "
getline expdate <"ExpDate.txt"
}
$1 ~ /media ID/ {MediaNumber = $NF}
...
$1 ~ /number of mounts/ {
"date +%Y"|getline YearToday
Year4 = YearToday - 4
if... (4 Replies)
Discussion started by: nymus7
4 Replies
4. Shell Programming and Scripting
One of our servers runs Solaris 8 and does not have "ls -lh" as a valid command. I wrote the following script to make the ls output easier to read and emulate "ls -lh" functionality. The script works, but it is slow when executed on a directory that contains a large number of files. Can anyone make... (10 Replies)
Discussion started by: shew01
10 Replies
5. UNIX for Dummies Questions & Answers
Hey all my co-workers and I are trying to put together a list of things root "Can't" do on any *NIX OS, so I wanted to come here and see what all we could come up with.
Here are two to start this off:
write to a read only mount FS
kill a tape rewind
Please add what you know.
Thanks,... (5 Replies)
Discussion started by: sunadmn
5 Replies
6. Shell Programming and Scripting
Can someone help me edit the below script to make it run faster?
Shell: bash
OS: Linux Red Hat
The point of the script is to grab entire chunks of information that concerns the service "MEMORY_CHECK".
For each chunk, the beginning starts with "service {", and ends with "}".
I should... (15 Replies)
Discussion started by: SkySmart
15 Replies
7. Shell Programming and Scripting
Hi,
I have a large number of input files with two columns of numbers.
For example:
83 1453
99 3255
99 8482
99 7372
83 175
I only wish to retain lines where the numbers fullfil two requirements. E.g:
=83
1000<=<=2000
To do this I use the following... (10 Replies)
Discussion started by: s052866
10 Replies
8. Shell Programming and Scripting
data.file:
contact {
contact_name=royce-rolls
modified_attributes=0
modified_host_attributes=0
modified_service_attributes=0
host_notification_period=24x7
service_notification_period=24x7
last_host_notification=0
last_service_notification=0
host_notifications_enabled=1... (8 Replies)
Discussion started by: SkySmart
8 Replies
LEARN ABOUT BSD
disktab
DISKTAB(5) File Formats Manual DISKTAB(5) NAME
disktab - disk description file SYNOPSIS
#include <disktab.h> DESCRIPTION
Disktab is a simple date base which describes disk geometries and disk partition characteristics. The format is patterned after the term- cap(5) terminal data base. Entries in disktab consist of a number of `:' separated fields. The first entry for each disk gives the names which are known for the disk, separated by `|' characters. The last name given should be a long name fully identifying the disk. The following list indicates the normal values stored for each disk entry. Name Type Description ns num Number of sectors per track nt num Number of tracks per cylinder nc num Total number of cylinders on the disk b0 str Filename of block zero primary bootstrap for device ba num Block size for partition `a' (bytes) bd num Block size for partition `d' (bytes) be num Block size for partition `e' (bytes) bf num Block size for partition `f' (bytes) bg num Block size for partition `g' (bytes) bh num Block size for partition `h' (bytes) fa num Fragment size for partition `a' (bytes) fd num Fragment size for partition `d' (bytes) fe num Fragment size for partition `e' (bytes) ff num Fragment size for partition `f' (bytes) fg num Fragment size for partition `g' (bytes) fh num Fragment size for partition `h' (bytes) pa num Size of partition `a' in sectors pb num Size of partition `b' in sectors pc num Size of partition `c' in sectors pd num Size of partition `d' in sectors pe num Size of partition `e' in sectors pf num Size of partition `f' in sectors pg num Size of partition `g' in sectors ph num Size of partition `h' in sectors se num Sector size in bytes (default 512) sf bool supports bad144-style bad sector forwarding so bool partition offsets in sectors ty str Type of disk (e.g. removable, winchester) Disktab entries may be automatically generated with the diskpart program. FILES
/etc/disktab SEE ALSO
newfs(8), diskpart(8), getdiskbyname(3) BUGS
This file shouldn't exist, the information should be stored on each disk pack. 4.2 Berkeley Distribution May 17, 1986 DISKTAB(5)