How to incrementally backup data & use most recent data


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to incrementally backup data & use most recent data
# 1  
Old 06-28-2008
How to incrementally backup data & use most recent data

Hey, dunno if the title really explains this well; basically my problem is (and this is on a router which is why the flash issue)...

I have some data I want to store (it's really irelevant, i'm just counting the lines per file, basically a simple counter).
I could store this data in flash but I'm wary of writing to it more than I have to given it's wear limit.
Therefore I want to store this data in memory and back it up to flash every few days (ok, I could do better but it's a simple problem so I use a simple solution), and assume reboots may occur causing the tmp data to be irelevant.

Thing is, I also need to reference the latest version of the data, which I know at times will be that in flash, but before I back it up it'll be that in memory, and I'm not sure what the best way to go about checking is.

Example (and note only 2 events, causing 1 \n 2, ever occur, and i need to know this always):

A) Tmp file contents:
Perm backup file contents:

I run a backup script every couple days (if a reboot happens between then I'm not so bothered), so far just "cat tmp.file > perm.file.

2 events cause data to fill tmp.

B) Tmp file contents: 1 \n 2
Perm backup file contents:


Now if I reference these, I can just use whatever has the higher line count.

Now I backup. cat tmp.file > perm.file

C) Tmp file contents: 1 \n 2
Perm backup file contents: 1 \n 2

Time goes by. another backup. cat tmp.file > perm.file

D) Tmp file contents: 1 \n 2
Perm backup file contents: 1 \n 2

reboot happens.

E) Tmp file contents:
Perm backup file contents: 1 \n 2

at this point use of whatever has the higher line count is still fine. but on a backup using cat tmp.file > perm.file:

F) Tmp file contents:
Perm backup file contents:



Not good. So I try cat tmp.file >> perm.file

A) Tmp file contents:
Perm backup file contents:

B) Tmp file contents: 1 \n 2
Perm backup file contents:

C) Tmp file contents: 1 \n 2
Perm backup file contents: 1 \n 2

D) Tmp file contents: 1 \n 2
Perm backup file contents: 1 \n 2 \n 1 \n 2

E) Tmp file contents:
Perm backup file contents: 1 \n 2 \n 1 \n 2

F) Tmp file contents:
Perm backup file contents: 1 \n 2 \n 1 \n 2

So now even though 2 events only ever happened, perm has 4 recorded, and just checking for the file with most data to get correct stats from ain't gonna help.



So.... My solution so far is: cat tmp.file >> perm.file && > tmp.file (wipe tmp.file of any data).

A) Tmp file contents:
Perm backup file contents:

B) Tmp file contents: 1 \n 2
Perm backup file contents:

C) Tmp file contents:
Perm backup file contents: 1 \n 2

D) Tmp file contents:
Perm backup file contents: 1 \n 2

E) Tmp file contents:
Perm backup file contents: 1 \n 2

F) Tmp file contents:
Perm backup file contents: 1 \n 2

So far so good. But if another call happens at some point...

A) Tmp file contents:
Perm backup file contents:

2 events cause data to fill tmp.
B) Tmp file contents: 1 \n 2
Perm backup file contents:

backup. cat tmp.file >> perm.file && wipe tmp
C) Tmp file contents:
Perm backup file contents: 1 \n 2

another event. now there's been 3.
D) Tmp file contents: 3
Perm backup file contents: 1 \n 2

backup. cat tmp.file >> perm.file && wipe tmp
C) Tmp file contents:
Perm backup file contents: 1 \n 2 \n 3



So my best idea so far is cat x >> y && wipe x, and when checking for data just add the (line count of the) two files together as the contents of both files should always equal the total number of events.

I'd just like to know if I'm going about that completely the wrong way? Is there a better / more efficient way of doing the above, and anyone see any issues in doing it the latter way? Thanks.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

How to get cpanel backup data in rescue mode?

How to get cpanel backup data in rescue mode? Server OS 6.3 minimal with cPanel /dev/sdb1 is main partition root@rescue ~ # fdisk -l Anyone can help Thank you (0 Replies)
Discussion started by: jaydul
0 Replies

2. Shell Programming and Scripting

Oracle Data BackUp with Help Of Perl

Hi All I am able to connect my oracle database with Perl and able to execute the queries. There is one schema names master19 which contains 50 table. I want to take backup of each table in and store at a given directory location in Perl For example suppose if a table student is present in... (3 Replies)
Discussion started by: parthmittal2007
3 Replies

3. Shell Programming and Scripting

Oracle Data BackUp with Help Of Perl

Hi All I am able to connect my oracle database with Perl and able to execute the queries. There is one schema names master19 which contains 50 table. I want to take backup of each table in and store at a given directory location in Perl For example suppose if a table student is present in... (1 Reply)
Discussion started by: parthmittal2007
1 Replies

4. Shell Programming and Scripting

Sort a the file & refine data column & row format

cat file1.txt field1 "user1": field2:"data-cde" field3:"data-pqr" field4:"data-mno" field1 "user1": field2:"data-dcb" field3:"data-mxz" field4:"data-zul" field1 "user2": field2:"data-cqz" field3:"data-xoq" field4:"data-pos" Now i need to have the date like below. i have just... (7 Replies)
Discussion started by: ckaramsetty
7 Replies

5. Shell Programming and Scripting

Extract data based on match against one column data from a long list data

My input file: data_5 Ali 422 2.00E-45 102/253 140/253 24 data_3 Abu 202 60.00E-45 12/23 140/23 28 data_1 Ahmad 256 7.00E-45 120/235 140/235 22 data_4 Aman 365 8.00E-45 15/65 140/65 20 data_10 Jones 869 9.00E-45 65/253 140/253 18... (12 Replies)
Discussion started by: patrick87
12 Replies

6. UNIX for Dummies Questions & Answers

Search for & edit rows & columns in data file and pipe

Dear unix gurus, I have a data file with header information about a subject and also 3 columns of n rows of data on various items he owns. The data file looks something like this: adam peter blah blah blah blah blah blah car 01 30 200 02 31 400 03 57 121 .. .. .. .. .. .. n y... (8 Replies)
Discussion started by: tintin72
8 Replies

7. Shell Programming and Scripting

data backup script

someone please guide me for following task 1. how can i make script that copy data from specified folder to destination folder 2. make tar for the current system date 3. establish ftp connection to backup server and place that tar file there. Regards, Mazhar (2 Replies)
Discussion started by: mazhar99
2 Replies

8. Shell Programming and Scripting

Get data from 3 differrnt oracle DB & then compare data

Hi All, I have a requirement to write a shell script for the following... we have 3 different database.... lets say A, B, C From these 3 DBs, i need to get data.. all have 3 different table...a,b, c A.a => Emp_code, count(*) B.b => emp_code, count(*) C.c => emp_code, copunt(*) Once get this... (4 Replies)
Discussion started by: Amit.Sagpariya
4 Replies

9. UNIX for Dummies Questions & Answers

Backup Data to tape

Hi everyone!!! I have a question on incremental backup`s and full backup`s? We currently use the Round-Robin schema to do full backup`s on a daily basis but want to change that as our data is increasing. I would like to imply the Grandfather-father-Son policy taking a full backup on the 1st of... (1 Reply)
Discussion started by: donovan
1 Replies

10. UNIX for Advanced & Expert Users

backup data with tar and show them

Hey everyone, i just want to backup data with tar. I know it works with: tar cvf. To show which data i have saved i can use tar tvf. But how can i do it simoultanous, that unix not first has to save my data to disk and after that read the data again to show me the list of my saved data? Please... (2 Replies)
Discussion started by: grashuepfer
2 Replies
Login or Register to Ask a Question