Single table or split the table


 
Thread Tools Search this Thread
Special Forums UNIX and Linux Applications Single table or split the table
# 1  
Old 02-12-2013
Single table or split the table

I have an application that collect data from 10 server every minutes and stored to mysql db,. there are about 10K record from each server on every minutes. that data will need to stored in 1 month.

Which ones is better, create single table for all servers and put field to identified data source or create multiple tables for each server.

Thanks
# 2  
Old 02-12-2013
Having used both strategies, I'm not sure mysql is really suited for frequent data logging at all. You end up with huge, weighty indexes that take minutes to load and hog all your RAM for data that's already sorted by time anyway.

I ended up just storing data in raw text flatfiles. No index, ergo no index issues. A few years of various kinds of sensor data, logged every 5 minutes, is still just 72 megs.
# 3  
Old 02-12-2013
@before4 you didn't mentioned how you'll use the logs.

In my opinion storing in mysql is better than flat files. querying become really easy.
Make sure to partition your table like on date & server_name eg: partition for every day and create local indexes based on kind of usage you want.

Last edited by 47shailesh; 02-12-2013 at 06:30 PM.. Reason: location=local typo correction
# 4  
Old 02-12-2013
What is really slick is a RDBMS with partitioned tables, so removing and adding are fast and simple. You can name and recycle the same partitions for the 31 days. If the primary key is time or insert order, you can mix servers in the same partition, but RDBMS guys are hyper this way, so they usually use separate partitions and you get strict time order in each for free. Yes, indexes to suport slow queries are good, if they do not slow insert too much. Definitely one table.

Ever consider near real time flow using remote clients? It often makes the data more useful, and there are not such large processing surges. Batch went out with punched cards.
This User Gave Thanks to DGPickett For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Read Table in file and split

HELLO I need your help please , i need to read a file that contain a table like : Name | Status ------------------ DB 1 | UP DB 2 | UP DB 3 | DOWN DB 4 | UP DB 5 | UP the objective to read each line and check if DB is UP or Down and give me the name of Down database.... (10 Replies)
Discussion started by: Abdellah
10 Replies

2. Web Development

Getting Rid of Annoying Bootstrap Table Borders and Wayward Table Lines

Bootstrap is great; but we have had some issues with Bootstrapped <tables> (and legacy <fieldset> elements) showing annoying, wayward lines. I solved that problem today with this simple jQuery in the footer: <script> $(function(){ $('tr, td, fieldset,... (0 Replies)
Discussion started by: Neo
0 Replies

3. Shell Programming and Scripting

awk to convert table-by-row to matrix table

Hello, I need some help to reformat this table-by-row to matrix? infile: site1 A:o,p,q,r,s,t site1 C:y,u site1 T:v,w site1 -:x,z site2 A:p,r,t,v,w,z site2 C:u,y site2 G:q,s site2 -:o,x site3 A:o,q,s,t,u,z site3 C:y site3 T:v,w,x site3 -:p,routfile: SITE o p q r s t v u w x y... (7 Replies)
Discussion started by: yifangt
7 Replies

4. UNIX and Linux Applications

Help in copying table structure to another table with constraints in Oracle

hi, i need to copy one table with data into another table, right now am using create table table1 as select * from table2 i want the constraints of table1 to be copied to table2 also , can anyone give me some solution to copy the constraints also, now am using oracle 10.2.0.3.0... (1 Reply)
Discussion started by: senkerth
1 Replies

5. Shell Programming and Scripting

Build a table from a list by comparing existing table entries

I am new to this shell scripting.... I have a file which contains list of users. This files get updated when new user comes into the system. I want to create script which will give a table containing unique list of users. When I say unique, it means script should match table while parsing... (3 Replies)
Discussion started by: dchavan1901
3 Replies

6. UNIX for Dummies Questions & Answers

Creating a condensed table from a pre-existing table in putty

Hello, I'm working with putty on Windows 7 professional and I'd like to know if there's a way to gather specific lines from a pre-existing table and make a new table with that information. More specifically, I'd like the program to look at a specific column, say column N, and see if any of the... (5 Replies)
Discussion started by: Deedee393
5 Replies

7. UNIX and Linux Applications

create table via stored procedure (passing the table name to it)

hi there, I am trying to create a stored procedure that i can pass the table name to and it will create a table with that name. but for some reason it creates with what i have defined as the variable name . In the case of the example below it creates a table called 'tname' for example ... (6 Replies)
Discussion started by: rethink
6 Replies

8. Shell Programming and Scripting

select values from db1 table and insert into table of DB2

Hi I am having three oracle databases running in three different machine. their ip address is different. from one of the DB am able to access both the databases.(means am able to select values and insert values in to tables individually.) I need to fetch some data from DB1 table(say DB1 ip is... (2 Replies)
Discussion started by: aemunathan
2 Replies

9. Shell Programming and Scripting

Check the record count in table (table in oracle)

I have requirement: 1) Check the record count in table (table in oracle) 2) If records exists generate the file for existing records and wait for some time then Go to sleep mode and Again check the record count after 10 min.......... (Loop this process if record count >0). 3) Generate touch... (1 Reply)
Discussion started by: kamineni
1 Replies

10. Shell Programming and Scripting

Check the record count in table (table in oracle)

I have requirement: 1) Check the record count in table (table in oracle) 2) If records exists generate the file for existing records and wait for some time (Go to sleep mode) and Again check the record count after 10 min.......... (Loop this process if record count >0). 3) Generate touch... (1 Reply)
Discussion started by: kamineni
1 Replies
Login or Register to Ask a Question