Unix/Linux Go Back    


Hardware Device drivers, hardware compatibility issues, motherboards, disk drives, graphics cards and other hardware related topics.

SAS or SSD for Ubuntu 14.04 and data analysis

Hardware


Tags
solved

Closed    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 02-11-2016
cmccabe cmccabe is offline
Registered User
 
Join Date: Nov 2013
Last Activity: 21 July 2017, 10:53 AM EDT
Location: Chicago
Posts: 1,156
Thanks: 694
Thanked 15 Times in 14 Posts
SAS or SSD for Ubuntu 14.04 and data analysis

I am in the process of building a workstation and have a question related to performance. I am a scientist who deals with big data (average file size 30-50gb). My OS is ubuntu 14.04 and so far I have a 128gb dual xeon E5-2630 with 6 cores each. I/O buffering is an issue so I am adding a 256/512? PCIe card and either 2 SSD or SAS drives for the OS and software. Since the PCIe will be separate its main purpose will be for file transfer, so would a SAS or SSD be a better fit for the OS? I am leaning towards SAS for the buffering issue, but wanted to ask more knowledgeable users. I forgot to mention that there will be a separate 1 or 2TB drive. Any recommendations for the size of the SAS or SSD? Thanks Linux.
Sponsored Links
    #2  
Old Unix and Linux 02-12-2016
jgt's Unix or Linux Image
jgt jgt is offline Forum Advisor  
Registered User
 
Join Date: Apr 2007
Last Activity: 22 July 2017, 3:32 PM EDT
Location: 44.21.48N 80.50.15W
Posts: 1,950
Thanks: 2
Thanked 255 Times in 241 Posts
do you access your data files randomly or sequentially?
Sponsored Links
    #3  
Old Unix and Linux 02-12-2016
cmccabe cmccabe is offline
Registered User
 
Join Date: Nov 2013
Last Activity: 21 July 2017, 10:53 AM EDT
Location: Chicago
Posts: 1,156
Thanks: 694
Thanked 15 Times in 14 Posts
I access files sequentially. Thank you Linux.
    #4  
Old Unix and Linux 02-12-2016
jim mcnamara jim mcnamara is offline Forum Staff  
...@...
 
Join Date: Feb 2004
Last Activity: 23 July 2017, 7:50 PM EDT
Location: NM
Posts: 11,115
Thanks: 549
Thanked 1,075 Times in 997 Posts
SSD is more than an order of magnitude (or much) faster than SAS high-rpm disks.
SSD is limited - usually to 1-2 TB of storage. With 128GB of memory, you could easily use SSD disks to load whatever file you want into memory - e.g., usual term is a RAMDISK. Ubuntu supports this. It also caches files very effectively without much human intervention other than configuration.

Learn about pdflush: The Linux Page Cache and pdflush

There is also vmtouch. You can force any file to be read entirely into memory. Which would definitely favor SSD.

https://hoytech.com/vmtouch/ Also note some other tools on that site.

So, I would suggest: SSD's and vmtouch (or an analagous tool.)
The Following User Says Thank You to jim mcnamara For This Useful Post:
cmccabe (02-13-2016)
Sponsored Links
    #5  
Old Unix and Linux 02-12-2016
jgt's Unix or Linux Image
jgt jgt is offline Forum Advisor  
Registered User
 
Join Date: Apr 2007
Last Activity: 22 July 2017, 3:32 PM EDT
Location: 44.21.48N 80.50.15W
Posts: 1,950
Thanks: 2
Thanked 255 Times in 241 Posts
You might also consider m sata ssd
Samsung SSD 840 EVO mSATA | Samsung SSD
The Following User Says Thank You to jgt For This Useful Post:
cmccabe (02-13-2016)
Sponsored Links
    #6  
Old Unix and Linux 02-13-2016
bakunin bakunin is offline Forum Staff  
Bughunter Extraordinaire
 
Join Date: May 2005
Last Activity: 22 July 2017, 5:51 PM EDT
Location: In the leftmost byte of /dev/kmem
Posts: 5,481
Thanks: 104
Thanked 1,527 Times in 1,132 Posts
Not directly related but i had a longer workshop yesterday about our new storage system (EMC VMax 200k). EMC claims that they had intended the 300GB 15k-SAS drives for high-performance, but phase them out now because (quoting from memory) with the development of Flash-SSDs its just not worth it any more. They also claim that, because they use SLC-based hardware, they have even lower rates of disk-replacement, even in heavy-duty transactional storage systems, than with rotational disks, to which a much lower energy consumption of the SSDs compared to the 15k-SAS disks contributes. There is simply less heat involved and that shows when you pack some ~2500 disks into a rack.

You haven't said where you are going to place the workstation, but in case it is going to be somewhere near your desk: 15k-disks are awefully LOUD in addition to be premier heating devices while SSDs are completely silent.

I hope this helps.

bakunin
The Following 2 Users Say Thank You to bakunin For This Useful Post:
cmccabe (02-13-2016), jim mcnamara (02-13-2016)
Sponsored Links
    #7  
Old Unix and Linux 02-13-2016
cmccabe cmccabe is offline
Registered User
 
Join Date: Nov 2013
Last Activity: 21 July 2017, 10:53 AM EDT
Location: Chicago
Posts: 1,156
Thanks: 694
Thanked 15 Times in 14 Posts
Thank you all Linux.
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Data analysis, Regular Expression - Unix @man UNIX for Dummies Questions & Answers 2 07-10-2012 10:18 AM
Help with analysis data based on particular column content perl_beginner Shell Programming and Scripting 2 03-22-2012 08:37 AM
What is the best tools for performance data gathering and analysis? devyfong Red Hat 6 12-21-2011 10:08 AM



All times are GMT -4. The time now is 02:46 AM.