Sponsored Content
Top Forums Shell Programming and Scripting Using sqlite query to build script... Post 302947617 by NyxPDX on Friday 19th of June 2015 02:31:46 PM
Old 06-19-2015
Using sqlite query to build script...

Okay, so this one is a bit above my knowledge level so I'm hoping for some pointers.

Here's the scenario:

I have a backup system on my network that makes single file images of the machines it's backing up and uses an sqlite database to keep track of everything. As is pretty typical with backup systems, it simply creates new backups and deletes the old. It also names these files with the date and time it ran as part of the file name. There is no way to alter this behavior.

I also have a cloud backup service that uses delta-blocking to only upload changed file blocks.

Now, because the local backup system is creating new files, if I simply point the cloud backups to those folders it's going to try and upload the whole dang thing every time since it sees it as a completely new file (which it technically is). However, the new file is the same format and everything as the old one, and after doing some test, if I can maintain a hardlink of a single name to the most current backup image, the cloud backup service sees it as the same file and uses the delta-blocking to only update data that is different in the new image from the previous.

I am decent with SQL queries so I've got as far as using the data in the DB and some fancy string manipulation to create the list of commands that would do this for me... i.e. query output is several lines similar to:
Code:
ln <path to file1> <hardlink1>
ln <path to file2> <hardlink2>
ln <path to file3> <hardlink3>

I'm stuck at creating a bash script or something that I can schedule to run that will rm the old links, run the query, and use the results to create the new links... unless you guys think there's a better or easier way. I'm just going this direction because SQL is what I know. I was actually going to try to create a bash script that basically creates another bash script by echoing a few lines of commands, appending the query output somehow, and then executes what it just created... that's when I decided to post here. Smilie

Thanks everyone!

Last edited by NyxPDX; 06-19-2015 at 03:52 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

sqlite issue?

i'm on freebsd 5.2.1, and from a fresh installation i've used pkg_add for the latest ported version of apache, as well as installing php 5. supposedly php5 comes with native support for sqlite (in the binary package), and this is what i added. i am trying to install a site engine (the 'gyrator'... (0 Replies)
Discussion started by: brandan
0 Replies

2. Shell Programming and Scripting

Perl - SQLite question

Hello All, I am trying to write a Perl script that is using 'SQLite' as the application needs a very light weight Database. I wanted to know how to catch exceptions when I run queries in SQLite. Without this the Perl script comes to a halt everytime an exception occurs. Please help. Regards,... (4 Replies)
Discussion started by: garric
4 Replies

3. Shell Programming and Scripting

hell and sqlite

Hi everyone, I have a requirement that requires me to fill an sqlite database with 100,000 entries (no duplicates). I will start out by giving the command that will insert the values necessary to populate the database: # sqlite /var/local/database/dblist "insert into list... (2 Replies)
Discussion started by: ogoy
2 Replies

4. Shell Programming and Scripting

Query Oracle tables and return values to shell script that calls the query

Hi, I have a requirement as below which needs to be done viz UNIX shell script (1) I have to connect to an Oracle database (2) Exexute "SELECT field_status from table 1" query on one of the tables. (3) Based on the result that I get from point (2), I have to update another table in the... (6 Replies)
Discussion started by: balaeswari
6 Replies

5. Shell Programming and Scripting

[ask]sqlite with shell script

hlow all i have file like this : BDG0100.2011091620162100CF5341.DAT BDG0100.2011091720175500CF5342.DAT BDG0100.2011091820192900CF5343.DAT BDG0100.2011091920210600CF5344.DAT but now i want make file like this 20110916.DAT 20110919.DAT 20110918.DAT 20110919.DAT so what i can do that... (3 Replies)
Discussion started by: zvtral
3 Replies

6. Shell Programming and Scripting

Shell Script to execute Oracle query taking input from a file to form query

Hi, I need to query Oracle database for 100 users. I have these 100 users in a file. I need a shell script which would read this User file (one user at a time) & query database. For instance: USER CITY --------- ---------- A CITY_A B CITY_B C ... (2 Replies)
Discussion started by: DevendraG
2 Replies

7. UNIX for Dummies Questions & Answers

Adding SDK Build on Kernel Source Build

Hi, So I downloaded this kernel source and was able to build it successfully. But I want to add this SDK source code inside, can anyone help me how to do this? Note that the SDK source can be built by itself. I added the SDK in the main Makefile: init-y := init/ #added SDK... (0 Replies)
Discussion started by: h0ujun
0 Replies

8. Shell Programming and Scripting

Build.xml invocation by Build Script

Hi I have a build.xml file and I can run it on Windows via cmd. Now I want to write a script to invoke the same. Is there a way to do this? (1 Reply)
Discussion started by: ankur328
1 Replies

9. Web Development

MySQL Query to Build Mockup Vue.js UserCP Timeline Page

Here is the query (and some sample results) I plan to use to build a new timeline page in the mockup vue.js usercp I am working on. When the postid is the same as lastpostid, this means the timeline entry will be - "{{Member}} Started Discussion {{Thread Title}} at {{date and time}}" and when... (4 Replies)
Discussion started by: Neo
4 Replies

10. UNIX for Beginners Questions & Answers

Manipulating UNIX and sqlite db

Hi there, I have multiple rows of data. For example: S/N | Name| Age I would like to store them into sqlite database after doing some grepping in CSV and output them into console/html format. Will it be possible? (1 Reply)
Discussion started by: alvinoo
1 Replies
digitaglinktree(1)						Linux User's Manual						digitaglinktree(1)

NAME
digitaglinktree - Export tag structure of photos in digikam to the filesystem. SYNOPSIS
digitaglinktree -l taglinkdir | -A archivedir -d database [-r rootdir] [-H|-f|-a|-v|-C] DESCRIPTION
digitaglinktree will create a linktree for all photos in a digikam database that have tags set on them. Tags (like eg. "family", "events", ...) are used in digikam to create virtual folders containing images that all have one or more tags assigned. Please note: Photos that have no tags at all assigned are silently ignored by this script. The program will not modify or even touch your original photos managed by digikam. The script can be used in two ways: If you call it using Option -l taglinkdir the script will create the user specified directory taglinkdir and inside this directory it will create sub directories for digikam tags set on the photos. Inside these subdirectories it will finally place symbolic or hard links (see -H) to photos having the tags in question. As a result you will see the tags of your pho- tos as folders and in these folders you will find links to your original photos. In this way you can access the collection of all images that share a certain tag by changing directory to the folder with the tags name created by this script. This allows you e.g. to run JAlbum a photo album software that needs to find the pictures to be put into a web album in the filesystem because JAlbum cannot access digikams virtual folders directly. The second way of calling this script is the so called archive-mode by setting option -A archiveDir. Archive mode is thought for people who want to archive tagged photos independently of digikams root directories and the photos therein. This way you can put your photos and their tag structure in eg. a tar archive and send it to a friend, who can look at the photos via their tag structure. In this mode the script creates the directory given as parameter to -A and in this directory two more subdirectories. One named Photos and a second named Tags. The Photos directory contains hard links to your original photos, and the Tags directory con- tains a subdirectory for each Tag used by any of your photos. Inside this subdirectory there are links (either symbolic or hard links) to the files in the Photos directory. This way the archive directory needs nearly no additional space on your harddisk and you have an ar- chive that allows you or a friend to easily look at the photos tag structure. Another benefit from using this script is that you have kind of a backup of your tag settings for all of your photos. The backup is simply the directory structure containing links to the original images that wear the tags. This could become important if for whatever reason the digikam.db file gets corrupted or even lost. COMMAND-LINE OPTIONS -l taglinkdir Parameter taglinkdir denotes a directory in which the tag structure of all your photos stored in rootdir will be exported to by creating subdirectories for each tag and placing symbolic links in these subdirectories that point to the original photo wearing the tags. If calling the script with option -l taglinkDir you also have to specify options -r rootdir as well as -d database. -A archivedirectory archivedirectory denotes a directory into which the script will export the photos and their tag structure. -A has to be used together with option -r rootdir if using digikam version < 0.10, as well as -d database else the script will terminate. Inside the archive directory the script will create a Photos and a Tags directory. It will put hard links in the Photos directory that point to your original photos. By using hard links you are independent of changes in your digikam root directory but on the other hand you are limited to one filesystem. So the directory given by -r rootdir and the directory specified for -A archivedir have to be one the same filesystem. If using digikam in version >= 0.10 you cannot specify -r rootdir . Instead the root paths of all albums are taken from digikams database directly. However still the requirement holds, that archivedir has to be on the same filesystem like all the root directories containing photos you defined in digikam. If one of digikams root directories is on another filesystem this one will not be processed, since hardlinking the photos inside the Photos directory would not work in this case! The Tags subdirectory will contain links to the files in the Photos directory. This way you have one archive directory that is completely self contained. You can tar it, send it to a friend or just put it somewhere for archivel or backup purposes. Usually only those photos will be archived that have a digikam tag set on them. By using option -C however you can perform a complete ar- chive. See -C for more infos. -d database database is the complete path including the filename to digikams photo database which usually can be found in digikams root direc- tory. The files name is usually digikam.db . -r rootdir rootdir denotes the digikam base directory containing all your photos if you are using digikam in a version before 0.10. If you are using digikam 0.10 or newer you must not use this option. If you do it anyway you will see a hint that the option given will be ignored because digikams version 0.10 database contains the root directories of all albums defined in digikam. -C When the script is called with option -A archivedir only those photos will be archived (by placing links) in the Photos subdirectory of archivedir that have at least one digikam tag set. By setting option -C all photos will be archived to archivedir no matter if they have a tag set or not. Note: This only changes the contents of the Photos subdirectory not of the Tags subdirectory in the archivedir directory. -a By default the script will try to create relative symbolic links from the directory taglinkdir set by option -l to the photo files under rootdir given by option -r. Using this option will result in absolute symbolic links beeing created instead of relative ones. -H By default the script will create soft (symbolic) links from the Tag-Tree to the photos. By setting option -H the script will use hard links instead. Please note that hard links can only be created inside one filesystem. So your photos and the Tag tree have to be one the same filesystem. If not you will see a warning about this problem and the script will not run. -f In digikam photos can have hierachical tags (tags that have subtags). In this case digitaglinktree would by default add a directory for the tag and a subdirectory for each of the subtags of this tag. By setting -f a subtag is treated like a regular tag just as its parent tag so digitaglinktree will create all subdirectories for tags and subtags at the same level independent of the tag - subtag hierarchy. -v Prints the scripts version number and exits. CONFIGURATION
By default this script will run with all photo databases created by digikam version 0.10 as well as older version like 0.9 and 0.8. If you still have digikam version 0.7 then you have to reconfigure the script itself. You have to reconfigure the script by setting the path to the sqlite binary that is used by the script to query the digikam database digikam.db. Since very old digikam version use sqlite in version 2, but later digikam versions need sqlite version 3 you have to take care to install the correct version of sqlite for the installed digikam version and to set the path to the correct sqlite executable in the scripts head: Choose $SQLITE="/usr/bin/sqlite3"; for digikam version 0.8x and 0.9x and 0.10x $SQLITE="/usr/bin/sqlite"; for digikam version 0.7x. EXAMPLE
A call to digitaglinktree is shown below: digiTagLinktree -l /home/user/tags -d /home/user/photos/digikam.db In this example we assume that you are running digikam version 0.10 or higher so that no option -r was used to specify the photo root dir. Instead this information will automatically be fetched from digikams database. In case you want to run the script on a digikam database that was created by digikam version 0.9 or earlier you have to use -r to specify the root directory where you keep all your photos that are managed by digikam: digiTagLinktree -r /home/user/photos -l /home/user/tags -d /home/user/photos/digikam.db In this example digikams photo root denoted by -r is /home/user/photos. Option -l /home/user/tags tells the script that all the subdirectories and symbolic links will be placed in the directory /home/user/tags. The folder was chosen so that the tags-directory is not under digikams photo root. You may put the tags folder inside digikams photoroot but this is not the preferred method. Because the link directory contains only links this tag structure does hardly need any additional space on your harddisk. AUTHORS
digitaglinktree was written by Rainer Krienke <krienke at uni-koblenz.de> 16 Aug 2006 digitaglinktree(1)
All times are GMT -4. The time now is 08:39 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy