Sponsored Content
Top Forums UNIX for Dummies Questions & Answers I need a database and a plan of attack! Post 302600940 by ws6transam on Wednesday 22nd of February 2012 01:01:14 PM
Old 02-22-2012
I need a database and a plan of attack!

Hi everyone,

I've got an extensive collection of seismic files that I am trying to turn into workable subsurface data collection. It's all real-time history and it is being loaded onto the main linux computer from a collection of about 1000 CDs. There are about 4000 seismic files on each CD, and these CDs were generated by approximately 20 seismic stations over the course of five to eight years.

There are gaps in the record as some of the stations went offline and online throughout the eight year history.

One of the tools that I need to develop is a workable database that can inventory the various seismic files and record within the database the times of when data is on-hand and the times when there are gaps in the seismic records.

Each of the files contained within the database has the file creation date embedded in the file name. As an example:

Quote:
-rwxrwxr-x 1 msugws users 174592 Nov 1 14:52 20011226223330.out
-rwxrwxr-x 1 msugws users 174592 Nov 1 14:52 20011226223932.out
-rwxrwxr-x 1 msugws users 174592 Nov 1 14:52 20011226224537.out
-rwxrwxr-x 1 msugws users 174592 Nov 1 14:52 20011226225141.out
-rwxrwxr-x 1 msugws users 174592 Nov 1 14:52 20011226225743.out
-rwxrwxr-x 1 msugws users 174592 Nov 1 14:52 20011226230348.out
-rwxrwxr-x 1 msugws users 174592 Nov 1 14:52 20011226230950.out
-rwxrwxr-x 1 msugws users 174592 Nov 1 14:52 20011226231554.out
-rwxrwxr-x 1 msugws users 174592 Nov 1 14:52 20011226232157.out
-rwxrwxr-x 1 msugws users 174592 Nov 1 14:52 20011226232803.out
Note that the first file name (20011226223330.out) breaks the time down (in GMT) as: 2001, 12,26,22,33,30 which translates to 2001-Dec-26 22:33:30 GMT as the initial time within the file.

I would like to first of all,

1) Pick an appropriate database and RDMS which is workable via sql and automated scripting

2) Develop a script that can:
a. recursively move through all of the files within the directory structure,
b. seek the file name of each .OUT file as well as the station designator as found in the folder name
c. Create an entry within the database consisting of
ca. The decoded date & time that is sourced from the file name
cb. The station name designator
cc. The actual file name including full path
d. Place the entry into the database to catalog the file.

This would help me identify where it is that I have data and where I have gaps.

However, I am really new to scripting, and my database skills are like, 22 years out of date.

If you have the time and insight, could you maybe give me a hint on where to start ? It's been a big project thus far and is getting bigger. Putting a database together that inventories the data segments would be greatly beneficial, especially when I move onto the next phase, which is time validation of the existing data, and the conversion of that data into a researchable format.

---------- Post updated at 01:01 PM ---------- Previous update was at 12:40 PM ----------

One thing that comes to mind is to come up with a recursive ls command that outputs to an ascii file the full path name for each .OUT file within the entire directory structure. This would contain all of the raw information which could then be parsed, maybe with a script, into a .CSV format text file? :shrug:
 

6 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

plan

i want to make a program run when a person types 'finger USERNAME', username being my username. how can i do this? (2 Replies)
Discussion started by: cypher
2 Replies

2. What is on Your Mind?

401 Keg Plan

If you had purchased $1,000 of AIG stock one year ago, you would have about $40.00 left. With Lehman, you have about $6.00 left. With Fannie or Freddie, you would have less than $5.00 left. But, if you had purchased $1,000 worth of beer one year ago, and had drunk all of the beer, then turned... (4 Replies)
Discussion started by: Neo
4 Replies

3. Filesystems, Disks and Memory

an advice regarding backup plan

Hi all i'm looking for good advice regarding backup plan becuase its first time to me handle large scale database expected to grow up 10000 - 20000 record per year with daily operations on it I'm working as sysAdmin in educational organization ( junior level ) with mixed OSes environment... (3 Replies)
Discussion started by: h@foorsa.biz
3 Replies

4. AIX

Plan to shutdown servers

Hello everyone I need to shutdown all my servers and my storage. I would like to hear your opinions about this. This is my little plan about all this. 1.-Stop the applications 2.-Stop the webservers 3.-Stop the ihs 4.-Stop the databases 5.-Verify no process are running 6.-Close the... (1 Reply)
Discussion started by: lo-lp-kl
1 Replies

5. What is on Your Mind?

Plan to Buy a New Laptop

It is time for me to buy a new laptop computer and i was searching the net for the last few days looking for one. The results haven't really been satisfying and i thought you might have information i lack - if so, i'd be grateful for sharing it with me and the probably interested audience. What... (22 Replies)
Discussion started by: bakunin
22 Replies

6. UNIX for Beginners Questions & Answers

Cpio Restore didnt go to plan

Hello folks, one of the RAID drives in our SCO system crashed recently and being hot swap it was replaced. Problem was that on boot it stops at: Checking protected password and protected subsystem databases.... First I did #authck -a and checked /etc/auth/system/ttys as per instructions in a... (7 Replies)
Discussion started by: Redstar
7 Replies
curl_getdate(3) 						  libcurl Manual						   curl_getdate(3)

NAME
curl_getdate - Convert a date string to number of seconds since January 1, 1970 SYNOPSIS
#include <curl/curl.h> time_t curl_getdate(char *datestring, time_t *now ); DESCRIPTION
This function returns the number of seconds since January 1st 1970 in the UTC time zone, for the date and time that the datestring parame- ter specifies. The now parameter is not used, pass a NULL there. NOTE: This function was rewritten for the 7.12.2 release and this documentation covers the functionality of the new one. The new one is not feature-complete with the old one, but most of the formats supported by the new one was supported by the old too. PARSING DATES AND TIMES
A "date" is a string containing several items separated by whitespace. The order of the items is immaterial. A date string may contain many flavors of items: calendar date items Can be specified several ways. Month names can only be three-letter english abbreviations, numbers can be zero-prefixed and the year may use 2 or 4 digits. Examples: 06 Nov 1994, 06-Nov-94 and Nov-94 6. time of the day items This string specifies the time on a given day. You must specify it with 6 digits with two colons: HH:MM:SS. To not include the time in a date string, will make the function assume 00:00:00. Example: 18:19:21. time zone items Specifies international time zone. There are a few acronyms supported, but in general you should instead use the specific relative time compared to UTC. Supported formats include: -1200, MST, +0100. day of the week items Specifies a day of the week. Days of the week may be spelled out in full (using english): `Sunday', `Monday', etc or they may be abbreviated to their first three letters. This is usually not info that adds anything. pure numbers If a decimal number of the form YYYYMMDD appears, then YYYY is read as the year, MM as the month number and DD as the day of the month, for the specified calendar date. EXAMPLES
Sun, 06 Nov 1994 08:49:37 GMT Sunday, 06-Nov-94 08:49:37 GMT Sun Nov 6 08:49:37 1994 06 Nov 1994 08:49:37 GMT 06-Nov-94 08:49:37 GMT Nov 6 08:49:37 1994 06 Nov 1994 08:49:37 06-Nov-94 08:49:37 1994 Nov 6 08:49:37 GMT 08:49:37 06-Nov-94 Sunday 94 6 Nov 08:49:37 1994 Nov 6 06-Nov-94 Sun Nov 6 94 1994.Nov.6 Sun/Nov/6/94/GMT Sun, 06 Nov 1994 08:49:37 CET 06 Nov 1994 08:49:37 EST Sun, 12 Sep 2004 15:05:58 -0700 Sat, 11 Sep 2004 21:32:11 +0200 20040912 15:05:58 -0700 20040911 +0200 STANDARDS
This parser was written to handle date formats specified in RFC 822 (including the update in RFC 1123) using time zone name or time zone delta and RFC 850 (obsoleted by RFC 1036) and ANSI C's asctime() format. These formats are the only ones RFC2616 says HTTP applications may use. RETURN VALUE
This function returns -1 when it fails to parse the date string. Otherwise it returns the number of seconds as described. If the year is larger than 2037 on systems with 32 bit time_t, this function will return 0x7fffffff (since that is the largest possible signed 32 bit number). Having a 64 bit time_t is not a guarantee that dates beyond 03:14:07 UTC, January 19, 2038 will work fine. On systems with a 64 bit time_t but with a crippled mktime(), curl_getdate will return -1 in this case. REWRITE
The former version of this function was built with yacc and was not only very large, it was also never quite understood and it wasn't pos- sible to build with non-GNU tools since only GNU Bison could make it thread-safe! The rewrite was done for 7.12.2. The new one is much smaller and uses simpler code. libcurl 7.0 12 Aug 2005 curl_getdate(3)
All times are GMT -4. The time now is 01:09 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy