Sponsored Content
Top Forums Shell Programming and Scripting Using sqlite query to build script... Post 302947617 by NyxPDX on Friday 19th of June 2015 02:31:46 PM
Old 06-19-2015
Using sqlite query to build script...

Okay, so this one is a bit above my knowledge level so I'm hoping for some pointers.

Here's the scenario:

I have a backup system on my network that makes single file images of the machines it's backing up and uses an sqlite database to keep track of everything. As is pretty typical with backup systems, it simply creates new backups and deletes the old. It also names these files with the date and time it ran as part of the file name. There is no way to alter this behavior.

I also have a cloud backup service that uses delta-blocking to only upload changed file blocks.

Now, because the local backup system is creating new files, if I simply point the cloud backups to those folders it's going to try and upload the whole dang thing every time since it sees it as a completely new file (which it technically is). However, the new file is the same format and everything as the old one, and after doing some test, if I can maintain a hardlink of a single name to the most current backup image, the cloud backup service sees it as the same file and uses the delta-blocking to only update data that is different in the new image from the previous.

I am decent with SQL queries so I've got as far as using the data in the DB and some fancy string manipulation to create the list of commands that would do this for me... i.e. query output is several lines similar to:
Code:
ln <path to file1> <hardlink1>
ln <path to file2> <hardlink2>
ln <path to file3> <hardlink3>

I'm stuck at creating a bash script or something that I can schedule to run that will rm the old links, run the query, and use the results to create the new links... unless you guys think there's a better or easier way. I'm just going this direction because SQL is what I know. I was actually going to try to create a bash script that basically creates another bash script by echoing a few lines of commands, appending the query output somehow, and then executes what it just created... that's when I decided to post here. Smilie

Thanks everyone!

Last edited by NyxPDX; 06-19-2015 at 03:52 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

sqlite issue?

i'm on freebsd 5.2.1, and from a fresh installation i've used pkg_add for the latest ported version of apache, as well as installing php 5. supposedly php5 comes with native support for sqlite (in the binary package), and this is what i added. i am trying to install a site engine (the 'gyrator'... (0 Replies)
Discussion started by: brandan
0 Replies

2. Shell Programming and Scripting

Perl - SQLite question

Hello All, I am trying to write a Perl script that is using 'SQLite' as the application needs a very light weight Database. I wanted to know how to catch exceptions when I run queries in SQLite. Without this the Perl script comes to a halt everytime an exception occurs. Please help. Regards,... (4 Replies)
Discussion started by: garric
4 Replies

3. Shell Programming and Scripting

hell and sqlite

Hi everyone, I have a requirement that requires me to fill an sqlite database with 100,000 entries (no duplicates). I will start out by giving the command that will insert the values necessary to populate the database: # sqlite /var/local/database/dblist "insert into list... (2 Replies)
Discussion started by: ogoy
2 Replies

4. Shell Programming and Scripting

Query Oracle tables and return values to shell script that calls the query

Hi, I have a requirement as below which needs to be done viz UNIX shell script (1) I have to connect to an Oracle database (2) Exexute "SELECT field_status from table 1" query on one of the tables. (3) Based on the result that I get from point (2), I have to update another table in the... (6 Replies)
Discussion started by: balaeswari
6 Replies

5. Shell Programming and Scripting

[ask]sqlite with shell script

hlow all i have file like this : BDG0100.2011091620162100CF5341.DAT BDG0100.2011091720175500CF5342.DAT BDG0100.2011091820192900CF5343.DAT BDG0100.2011091920210600CF5344.DAT but now i want make file like this 20110916.DAT 20110919.DAT 20110918.DAT 20110919.DAT so what i can do that... (3 Replies)
Discussion started by: zvtral
3 Replies

6. Shell Programming and Scripting

Shell Script to execute Oracle query taking input from a file to form query

Hi, I need to query Oracle database for 100 users. I have these 100 users in a file. I need a shell script which would read this User file (one user at a time) & query database. For instance: USER CITY --------- ---------- A CITY_A B CITY_B C ... (2 Replies)
Discussion started by: DevendraG
2 Replies

7. UNIX for Dummies Questions & Answers

Adding SDK Build on Kernel Source Build

Hi, So I downloaded this kernel source and was able to build it successfully. But I want to add this SDK source code inside, can anyone help me how to do this? Note that the SDK source can be built by itself. I added the SDK in the main Makefile: init-y := init/ #added SDK... (0 Replies)
Discussion started by: h0ujun
0 Replies

8. Shell Programming and Scripting

Build.xml invocation by Build Script

Hi I have a build.xml file and I can run it on Windows via cmd. Now I want to write a script to invoke the same. Is there a way to do this? (1 Reply)
Discussion started by: ankur328
1 Replies

9. Web Development

MySQL Query to Build Mockup Vue.js UserCP Timeline Page

Here is the query (and some sample results) I plan to use to build a new timeline page in the mockup vue.js usercp I am working on. When the postid is the same as lastpostid, this means the timeline entry will be - "{{Member}} Started Discussion {{Thread Title}} at {{date and time}}" and when... (4 Replies)
Discussion started by: Neo
4 Replies

10. UNIX for Beginners Questions & Answers

Manipulating UNIX and sqlite db

Hi there, I have multiple rows of data. For example: S/N | Name| Age I would like to store them into sqlite database after doing some grepping in CSV and output them into console/html format. Will it be possible? (1 Reply)
Discussion started by: alvinoo
1 Replies
EXPIRE_BACKUPS(1)						       S3QL							 EXPIRE_BACKUPS(1)

NAME
expire_backups - Intelligently expire old backups SYNOPSIS
expire_backups [options] <age> [<age> ...] DESCRIPTION
The expire_backups command intelligently remove old backups that are no longer needed. To define what backups you want to keep for how long, you define a number of age ranges. expire_backups ensures that you will have at least one backup in each age range at all times. It will keep exactly as many backups as are required for that and delete any backups that become redundant. Age ranges are specified by giving a list of range boundaries in terms of backup cycles. Every time you create a new backup, the existing backups age by one cycle. Example: when expire_backups is called with the age range definition 1 3 7 14 31, it will guarantee that you always have the following backups available: 1. A backup that is 0 to 1 cycles old (i.e, the most recent backup) 2. A backup that is 1 to 3 cycles old 3. A backup that is 3 to 7 cycles old 4. A backup that is 7 to 14 cycles old 5. A backup that is 14 to 31 cycles old Note If you do backups in fixed intervals, then one cycle will be equivalent to the backup interval. The advantage of specifying the age ranges in terms of backup cycles rather than days or weeks is that it allows you to gracefully handle irregular backup intervals. Imagine that for some reason you do not turn on your computer for one month. Now all your backups are at least a month old, and if you had specified the above backup strategy in terms of absolute ages, they would all be deleted! Specifying age ranges in terms of backup cycles avoids these sort of problems. expire_backups usage is simple. It requires backups to have names of the forms year-month-day_hour:minute:seconds (YYYY-MM-DD_HH:mm:ss) and works on all backups in the current directory. So for the above backup strategy, the correct invocation would be: expire_backups.py 1 3 7 14 31 When storing your backups on an S3QL file system, you probably want to specify the --use-s3qlrm option as well. This tells expire_backups to use the s3qlrm command to delete directories. expire_backups uses a "state file" to keep track which backups are how many cycles old (since this cannot be inferred from the dates con- tained in the directory names). The standard name for this state file is .expire_backups.dat. If this file gets damaged or deleted, expire_backups no longer knows the ages of the backups and refuses to work. In this case you can use the --reconstruct-state option to try to reconstruct the state from the backup dates. However, the accuracy of this reconstruction depends strongly on how rigorous you have been with making backups (it is only completely correct if the time between subsequent backups has always been exactly the same), so it's gener- ally a good idea not to tamper with the state file. OPTIONS
The expire_backups command accepts the following options: --quiet be really quiet --debug activate debugging output --version just print program version and exit --state <file> File to save state information in (default: ".expire_backups.dat") -n Dry run. Just show which backups would be deleted. --reconstruct-state Try to reconstruct a missing state file from backup dates. --use-s3qlrm Use s3qlrm command to delete backups. EXIT STATUS
expire_backups returns exit code 0 if the operation succeeded and 1 if some error occured. SEE ALSO
expire_backups is shipped as part of S3QL, http://code.google.com/p/s3ql/. COPYRIGHT
2008-2011, Nikolaus Rath 1.11.1 August 27, 2014 EXPIRE_BACKUPS(1)
All times are GMT -4. The time now is 09:09 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy