Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Importing a unix file dump into a PC capable database Post 24978 by DaltonF on Tuesday 23rd of July 2002 04:01:23 AM
Old 07-23-2002
Importing a unix file dump into a PC capable database

My development team has been trying to figure out how to import a unix data dump into SQL Server or convert it into an intermediate file format for several days.

The data dump in question looks like this:
$RecordID: 1<eof>
$Version: 1<eof>
Category: 1<eof>
Poster: John Doe<eof>
ProductName: Test Product<eof>
SKU: 10045689<eof>
Line1: Test Product Line 1 Description<eof>
Line2: Test Product Line 2 Description<eof>
Comments: Test Product Comments<eof>
<eor>

There are nearly 100,000 of records that have nearly 4,000 fields that vary based on the product's category. The field order varies per record within each category. When data does not exist for a given field, the field/value pair is simply excluded for the dump.

We were going to write a parsing application that converted this dump to XML, read it into a dataset, and then uploaded the dataset. About half way through that development, however, I realized that the parsing program would require a minimum of eight gigabytes to run. That obviously won't work.
(100,000 records X 4,000 fields = 400,000,000 fields) X 20 bytes per field name = 8,000,000,000 or 8 billion bytes

Do you know anyone that could tell us an easier way to import this unix dump into SQL? I'm sure there is a standard way of dealing with these dumps, but no one on our team has any experience with unix.

Any help or referral would be GREATLY APPRECIATED. This problem is holding up our entire development process.

Sincerely,

Dalton D. Franklin, MCP
Chief Executive Officer
Simplicity Technology
http://www.simplicitycorp.net
daltonf@simplicitycorp.net
615-327-9797 Telephone
615-985-0060 Fax
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

importing database from unix to winnt

i am a unix-super-beginner (swaddled and weaned on windows) and am trying to import a database from a unix directory into winnt. can someone help me or am i a hopeless case? (4 Replies)
Discussion started by: sadiecutie
4 Replies

2. UNIX for Advanced & Expert Users

building flat files in unix and importing them from windows

what is a flat file in unix? i have to import a unix flat files from windows based programme. my question is not to export from unix but only to import from windows only. how to build that flat files? how to create export to windows how to import from windows (3 Replies)
Discussion started by: tunirayavarapu
3 Replies

3. UNIX for Advanced & Expert Users

writing database tables into a file in unix

I want to collect the database tables(Ex: Emp,Sal,Bonus...etc) in a file & give this file at the command prompt, when prompted for the tables list at command prompt. How can i do that ? Ex:- Import jason/jason1 tables=emp,sal,bonus log=j1.log i want to change this into Ex:- Import... (3 Replies)
Discussion started by: dreams5617
3 Replies

4. Shell Programming and Scripting

Importing dump file

Hi, I am trying to import 22 .dmp files but facing the problem with the last table file it never ends the import command, only the table is created but the rows of the table don't get imported. This is the problem with only ine table rest 21 tables are being imported properly. Thanks in... (2 Replies)
Discussion started by: anushilrai
2 Replies

5. Shell Programming and Scripting

getting dump from database by UNIX script

can i make a dump to happen from my script in a databse.and download that dump. I want to schedule a script i cron so that i will make a dump and download that inot my server. (1 Reply)
Discussion started by: arunkumar_mca
1 Replies

6. UNIX for Dummies Questions & Answers

Import dump to database

Hi... I have dump in unix machine...How can I this import dump to Oracle database? Many thanks in advance. (2 Replies)
Discussion started by: agarwal
2 Replies

7. UNIX for Dummies Questions & Answers

Importing R cosine similarity to UNIX?

I really need help in this :( I have a file and would like to calculate the cosine similarity of the values in it... For now I do use R which has an easy function for doing so test <- as.matrix(read.csv(file="file.csv", sep=",", header=FALSE)) result<- cosine(t(test)) I am using unix of... (3 Replies)
Discussion started by: A-V
3 Replies

8. Boot Loaders

Is Unetbootin capable of making any iso file bootable

1)It is a great experience to achieve what I got today. I had downloaded GParted iso file (135 MB size). Now to make a bootable usb of it, I used unetbootin and to surprise I succeded in entering Gparted application and modified my partitions. Earlier I was in a notion that it is used only for... (0 Replies)
Discussion started by: ravisingh
0 Replies

9. AIX

Issue while importing Oracle Dump File on AIX 5.3

Hi All, I am facing one problem related to importing Oracle Dump file. We have two different version of AIX boxes with oracle (version 10.2.0.4.0) installed. On one AIX box (version 6.1) we are creating oracle dump file (*.dmp) using oracle exp utility and importing it onto another AIX box... (1 Reply)
Discussion started by: gunjan_thakur
1 Replies

10. UNIX for Advanced & Expert Users

Load a file from UNIX into database

Hi, I want your help to see if there is a way wherein we can load a set of files which are lying in unix server into my database using sql loader. Also the person who will be running sql loader does not have access to unix system. So is there a way i can provide any sql loader script wherein... (1 Reply)
Discussion started by: Vivekit82
1 Replies
savecore(1M)                                              System Administration Commands                                              savecore(1M)

NAME
savecore - save a crash dump of the operating system SYNOPSIS
/usr/bin/savecore [-Lvd] [-f dumpfile] [directory] DESCRIPTION
The savecore utility saves a crash dump of the kernel (assuming that one was made) and writes a reboot message in the shutdown log. It is invoked by the dumpadm service each time the system boots. savecore saves the crash dump data in the file directory/vmcore.n and the kernel's namelist in directory/unix.n. The trailing .n in the pathnames is replaced by a number which grows every time savecore is run in that directory. Before writing out a crash dump, savecore reads a number from the file directory/minfree. This is the minimum number of kilobytes that must remain free on the file system containing directory. If after saving the crash dump the file system containing directory would have less free space the number of kilobytes specified in minfree, the crash dump is not saved. if the minfree file does not exist, savecore assumes a minfree value of 1 megabyte. The savecore utility also logs a reboot message using facility LOG_AUTH (see syslog(3C)). If the system crashed as a result of a panic, savecore logs the panic string too. OPTIONS
The following options are supported: -d Disregard dump header valid flag. Force savecore to attempt to save a crash dump even if the header information stored on the dump device indicates the dump has already been saved. -f dumpfile Attempt to save a crash dump from the specified file instead of from the system's current dump device. This option may be useful if the information stored on the dump device has been copied to an on-disk file by means of the dd(1M) command. -L Save a crash dump of the live running Solaris system, without actually rebooting or altering the system in any way. This option forces savecore to save a live snapshot of the system to the dump device, and then immediately to retrieve the data and to write it out to a new set of crash dump files in the specified directory. Live system crash dumps can only be per- formed if you have configured your system to have a dedicated dump device using dumpadm(1M). savecore -L does not suspend the system, so the contents of memory continue to change while the dump is saved. This means that live crash dumps are not fully self-consistent. -v Verbose. Enables verbose error messages from savecore. OPERANDS
The following operands are supported: directory Save the crash dump files to the specified directory. If directory is not specified, savecore saves the crash dump files to the default savecore directory, configured by dumpadm(1M). FILES
directory/vmcore.n directory/unix.n directory/bounds directory/minfree /var/crash/'uname -n' default crash dump directory ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ SEE ALSO
adb(1), mdb(1), svcs(1), dd(1M), dumpadm(1M), svcadm(1M), syslog(3C), attributes(5), smf(5) NOTES
The system crash dump service is managed by the service management facility, smf(5), under the service identifier: svc:/system/dumpadm:default Administrative actions on this service, such as enabling, disabling, or requesting restart, can be performed using svcadm(1M). The ser- vice's status can be queried using the svcs(1) command. If the dump device is also being used as a swap device, you must run savecore very soon after booting, before the swap space containing the crash dump is overwritten by programs currently running. SunOS 5.10 25 Sep 2004 savecore(1M)
All times are GMT -4. The time now is 01:54 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy