Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Importing a unix file dump into a PC capable database Post 24978 by DaltonF on Tuesday 23rd of July 2002 04:01:23 AM
Old 07-23-2002
Importing a unix file dump into a PC capable database

My development team has been trying to figure out how to import a unix data dump into SQL Server or convert it into an intermediate file format for several days.

The data dump in question looks like this:
$RecordID: 1<eof>
$Version: 1<eof>
Category: 1<eof>
Poster: John Doe<eof>
ProductName: Test Product<eof>
SKU: 10045689<eof>
Line1: Test Product Line 1 Description<eof>
Line2: Test Product Line 2 Description<eof>
Comments: Test Product Comments<eof>
<eor>

There are nearly 100,000 of records that have nearly 4,000 fields that vary based on the product's category. The field order varies per record within each category. When data does not exist for a given field, the field/value pair is simply excluded for the dump.

We were going to write a parsing application that converted this dump to XML, read it into a dataset, and then uploaded the dataset. About half way through that development, however, I realized that the parsing program would require a minimum of eight gigabytes to run. That obviously won't work.
(100,000 records X 4,000 fields = 400,000,000 fields) X 20 bytes per field name = 8,000,000,000 or 8 billion bytes

Do you know anyone that could tell us an easier way to import this unix dump into SQL? I'm sure there is a standard way of dealing with these dumps, but no one on our team has any experience with unix.

Any help or referral would be GREATLY APPRECIATED. This problem is holding up our entire development process.

Sincerely,

Dalton D. Franklin, MCP
Chief Executive Officer
Simplicity Technology
http://www.simplicitycorp.net
daltonf@simplicitycorp.net
615-327-9797 Telephone
615-985-0060 Fax
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

importing database from unix to winnt

i am a unix-super-beginner (swaddled and weaned on windows) and am trying to import a database from a unix directory into winnt. can someone help me or am i a hopeless case? (4 Replies)
Discussion started by: sadiecutie
4 Replies

2. UNIX for Advanced & Expert Users

building flat files in unix and importing them from windows

what is a flat file in unix? i have to import a unix flat files from windows based programme. my question is not to export from unix but only to import from windows only. how to build that flat files? how to create export to windows how to import from windows (3 Replies)
Discussion started by: tunirayavarapu
3 Replies

3. UNIX for Advanced & Expert Users

writing database tables into a file in unix

I want to collect the database tables(Ex: Emp,Sal,Bonus...etc) in a file & give this file at the command prompt, when prompted for the tables list at command prompt. How can i do that ? Ex:- Import jason/jason1 tables=emp,sal,bonus log=j1.log i want to change this into Ex:- Import... (3 Replies)
Discussion started by: dreams5617
3 Replies

4. Shell Programming and Scripting

Importing dump file

Hi, I am trying to import 22 .dmp files but facing the problem with the last table file it never ends the import command, only the table is created but the rows of the table don't get imported. This is the problem with only ine table rest 21 tables are being imported properly. Thanks in... (2 Replies)
Discussion started by: anushilrai
2 Replies

5. Shell Programming and Scripting

getting dump from database by UNIX script

can i make a dump to happen from my script in a databse.and download that dump. I want to schedule a script i cron so that i will make a dump and download that inot my server. (1 Reply)
Discussion started by: arunkumar_mca
1 Replies

6. UNIX for Dummies Questions & Answers

Import dump to database

Hi... I have dump in unix machine...How can I this import dump to Oracle database? Many thanks in advance. (2 Replies)
Discussion started by: agarwal
2 Replies

7. UNIX for Dummies Questions & Answers

Importing R cosine similarity to UNIX?

I really need help in this :( I have a file and would like to calculate the cosine similarity of the values in it... For now I do use R which has an easy function for doing so test <- as.matrix(read.csv(file="file.csv", sep=",", header=FALSE)) result<- cosine(t(test)) I am using unix of... (3 Replies)
Discussion started by: A-V
3 Replies

8. Boot Loaders

Is Unetbootin capable of making any iso file bootable

1)It is a great experience to achieve what I got today. I had downloaded GParted iso file (135 MB size). Now to make a bootable usb of it, I used unetbootin and to surprise I succeded in entering Gparted application and modified my partitions. Earlier I was in a notion that it is used only for... (0 Replies)
Discussion started by: ravisingh
0 Replies

9. AIX

Issue while importing Oracle Dump File on AIX 5.3

Hi All, I am facing one problem related to importing Oracle Dump file. We have two different version of AIX boxes with oracle (version 10.2.0.4.0) installed. On one AIX box (version 6.1) we are creating oracle dump file (*.dmp) using oracle exp utility and importing it onto another AIX box... (1 Reply)
Discussion started by: gunjan_thakur
1 Replies

10. UNIX for Advanced & Expert Users

Load a file from UNIX into database

Hi, I want your help to see if there is a way wherein we can load a set of files which are lying in unix server into my database using sql loader. Also the person who will be running sql loader does not have access to unix system. So is there a way i can provide any sql loader script wherein... (1 Reply)
Discussion started by: Vivekit82
1 Replies
CMP(1)								   User Commands							    CMP(1)

NAME
cmp - compare two files byte by byte SYNOPSIS
cmp [OPTION]... FILE1 [FILE2 [SKIP1 [SKIP2]]] DESCRIPTION
Compare two files byte by byte. -b --print-bytes Print differing bytes. -i SKIP --ignore-initial=SKIP Skip the first SKIP bytes of input. -i SKIP1:SKIP2 --ignore-initial=SKIP1:SKIP2 Skip the first SKIP1 bytes of FILE1 and the first SKIP2 bytes of FILE2. -l --verbose Output byte numbers and values of all differing bytes. -n LIMIT --bytes=LIMIT Compare at most LIMIT bytes. -s --quiet --silent Output nothing; yield exit status only. -v --version Output version info. --help Output this help. SKIP1 and SKIP2 are the number of bytes to skip in each file. SKIP values may be followed by the following multiplicative suffixes: kB 1000, K 1024, MB 1,000,000, M 1,048,576, GB 1,000,000,000, G 1,073,741,824, and so on for T, P, E, Z, Y. If a FILE is `-' or missing, read standard input. AUTHOR
Written by Torbjorn Granlund and David MacKenzie. REPORTING BUGS
Report bugs to <bug-gnu-utils@gnu.org>. COPYRIGHT
Copyright (C) 2002 Free Software Foundation, Inc. This program comes with NO WARRANTY, to the extent permitted by law. You may redistribute copies of this program under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING. SEE ALSO
The full documentation for cmp is maintained as a Texinfo manual. If the info and cmp programs are properly installed at your site, the command info diff should give you access to the complete manual. diffutils 2.8.1 April 2002 CMP(1)
All times are GMT -4. The time now is 12:52 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy