Data to import the database as snippets Post: 302983471

Sponsored Content

Top Forums Shell Programming and Scripting Data to import the database as snippets Post 302983471 by blastit.fr on Tuesday 11th of October 2016 05:40:30 PM

10-11-2016

Registered User

Hi,

Below is a update of the script data2imp.sh with many comments for help.

Regarding any enhancement on the algorithm , I cannot guess what is to be done without seeing the files.

Could you for instance provide the header of the files, once formatted by this script ( the .lst files ) .

The command will create an output of the top 10 records of each file.

Code:

#head -10 *.lst > headers.txt 
# zip headers.zip headers.txt

Then attach the zipped file to your reply, if this content is not confidential.
( this zipped file size shouldn't exceed 100 kbytes for 1000 .lst files .

Code:

#data2imp.sh
#this script will process all files as *.txt in the current directory.
#The result file contains one line for each file, with the first field as the original file name.
#
#we need first to format the input file
function format_item {

awk '
# Filter non blank lines. NF = Number of Fields ( NF = 0 for empty line)
#  replace all consecutive blanks with 1 single blank space
NF {gsub(/  */," ")
# print line without Line Feed
 printf "%s" ,$0 
 istext++
 next
}
# Print a Line Feed when text is followed an empty line (NF==0)
istext {print "";istext=0}' $1
}
# Creation of a sample csv record 
# Take care there is no semicolumn inside the original text. In such case the field separator might be changed
function insert_item {
awk -v OFS=";" -v ITEM=$itemname '
# Record 1 is the item date
NR==1 {DATE=$1 " " $2}
# Record 2 is item title
# if length <= 100 : TITLE is the full record
# else seek for a dot in position between 51 and 100 of the record and cut record to this position
#    if no dot  in position between 51 and 100 : cut to the 100 first characters.           
NR==2 { if (length($0) <= 100) TITLE = $0
        else {
           dotposition=index(substr($0,51),".")
           if (dotposition == 0) {
             TITLE = substr($0,1,100) "..."}
           else {
             TITLE = substr($0,1,50 + dotposition)
           }
        }
       }
# same method as for  TITLE
NR==4{ if (length($0) <= 500) SNIPPET = $0
        else {
           dotposition=index(substr($0,401),".")
           if (dotposition == 0) {
             SNIPPET = substr($0,1,500) "..."}
           else {
             SNIPPET = substr($0,1,400 + dotposition)
           }
        }
print ITEM,DATE,TITLE,SNIPPET
}' $1
}
#--- main -----------------------------------
# Output initialisation
#
>items.csv
#
for i in *.txt
do
itemname=$(basename $i .txt)
echo Processing item $itemname
format_item $i > $itemname.lst
insert_item $itemname.lst >> items.csv
done

Last edited by blastit.fr; 10-11-2016 at 06:48 PM.. Reason: typo

blastit.fr

View Public Profile for blastit.fr

Find all posts by blastit.fr

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Import data from compressed file

HI I need to import data from a file which is in comressed format but system doesn't have enough space to uncompress file Is there any way so that i can do import from compressed file.

2. Programming

Code Snippets

Can Anyone give me an implimentation of virtual memory (simulation using paging only) .it should have the following algos for page replacement 1. LRU 2.FIFO 3.Clock references to web sites would be gr8 too it should have the code/algo no executables(in C only)

3. Windows & DOS: Issues & Discussions

import data files from Unix to Windows?

Hi, Is there any way to import data files from Unix system to Windows system? I have many data files on Unix machine generated every night. I need to pick certain data from each file and plug them into this windows file on the network share drive. Anyone has any idea? Thanks in advance!

4. UNIX for Dummies Questions & Answers

How can import data files to XL sheet.

Hi, I have the file(F1.XL) in Unix Box. it's updating every 1hr. I would like to import f1.xl to Windows excel sheet, when i need see the reports. can any one clarify, is there any VB script for importing data from UNIX, like sql connection.... thanks

5. Shell Programming and Scripting

Data Import perl script

Hi, I have a requirement for creating a Perl Script which will perform Data Import process in an automated way and I am elaborating herewith : Section 1 ) - use the following command line format : "./import.pl -h hostname -p port -f datafile.txt" Section 2) datafile.txt will...

6. Shell Programming and Scripting

Shell snip to import CSV data into BASH array

I have been trying to write a simple snip of bash shell code to import from 1 to 100 records into a BASH array. I have a CSV file that is structured like: record1,item1,item2,item3,item4,etc.,etc. .... (<= 100 items) record2,item1,item2,item3,item4,etc.,etc. .... (<= 100 items)...

7. UNIX for Dummies Questions & Answers

Import dump to database

Hi... I have dump in unix machine...How can I this import dump to Oracle database? Many thanks in advance.

8. Solaris

import lun data to mount point - Solaris 10

Hi Guys, I have EMC Storage and from this storage I have maped lun5 to Sun Solaris server and I have created on this lun mount point with name /application I have anothere Sun Solaris server and I'll colne lun5 to lun10 from storage level so the data of lun5 will be in lun10 how to...

9. Shell Programming and Scripting

shellscript to read data from txt file and import to oracle db

Hi all, Help needed urgently. I am currently writing a shellscript to read data/record from a flat file (.txt) file, and import/upload the data to oracle database. The script is working fine, but it takes too long time (for 18000 records, it takes around 90 mins). I guess it takes so long...

10. UNIX for Dummies Questions & Answers

SSH import mysql database

Hi all, I am trying to import a database in putty with the syntax: mysql –u database_username –p database_name < filename.mysql As you can see in the screenshot it asks me for the database password - which suggests that the syntax is correct - but then after I enter the password it gives...

LEARN ABOUT DEBIAN

svn::dump::record

SVN::Dump::Record(3pm)					User Contributed Perl Documentation				    SVN::Dump::Record(3pm)

NAME

       SVN::Dump::Record - A SVN dump record

SYNOPSIS

	   # SVN::Dump::Record objects are returned by the next_record()
	   # method of SVN::Dump

DESCRIPTION

       An "SVN::Dump::Record" object represents a Subversion dump record.

METHODS

       "SVN::Dump" provides the following gourps of methods:

   Record methods
       new()
	   Create a new empty "SVN::Dump::Record" object.

       type()
	   Return the record type, as guessed from its headers.

	   The method dies if the record type cannot be determined.

       set_header( $h, $v )
	   Set the header $h to the value $v.

       get_header( $h )
	   Get the value of header $h.

       set_property( $p, $v )
	   Set the property $p to the value $v.

       get_property( $p )
	   Get the value of property $p.

       delete_property( @k )
	   Delete the

       set_text( $t )
	   Set the value of the text block.

       get_text()
	   Get the value of the text block.

   Inner blocks manipulation
       A "SVN::Dump::Record" is composed of several inner blocks of various kinds: "SVN::Dump::Headers", "SVN::Dump::Property" and
       "SVN::Dump::Text".

       The following methods provide access to these blocks:

       set_headers_block( $headers )
       get_headers_block()
	   Get or set the "SVN::Dump::Headers" object that represents the record headers.

       set_property_block( $property )
       get_property_block()
	   Get or set the "SVN::Dump::Property" object that represents the record property block.

       delete_property( @keys )
	   Delete the given properties. Behave like the builtin "delete()".

       set_text_block( $text )
       get_text_block()
	   Get or set the "SVN::Dump::Text" object that represents the record text block.

       set_included_record( $record )
       get_included_record()
	   Some special record are actually output recursiveley by svnadmin dump.  The "record in the record" is stored within the parent record,
	   so they are parsed as a single record with an included record.

	   "get_record()" / "set_record()" give access to the included record.

	   According to the Subversion sources (subversion/libsvn_repos/dump.c), this is a "delete original, then add-with-history" node. The dump
	   looks like this:

	       Node-path: tags/mytag/myfile
	       Node-kind: file
	       Node-action: delete

	       Node-path: tags/mytag/myfile
	       Node-kind: file
	       Node-action: add
	       Node-copyfrom-rev: 23
	       Node-copyfrom-path: trunk/myfile

	   Note that there is a single blank line after the first header block, and four after the included one.

       update_headers()
	   Update the various "...-length" headers. Used internally.

	   You must call this method if you update the inner property or text blocks directly, or the results of "as_string()" will be
	   inconsistent.

   Information methods
       has_prop()
	   Return a boolean value indicating if the record has a property block.

       has_text()
	   Return a boolean value indicating if the record has a text block.

       has_prop_only()
	   Return a boolean value indicating if the record has only a property block (and no text block).

       has_prop_or_text()
	   Return a boolean value indicating if the record has a property block or a text block.

       property_length()
	   Return the length of the property block.

       text_length()
	   Return the length of the text block.

   Output method
       as_string()
	   Return a string representation of the record.

	   Warning: dumping a record currenly gives back the information that was read from the original dump. Which means that if you modified
	   the property or text block of a record, the headers will be inconstent.

ENCAPSULATION

       When using "SVN::Dump" to manipulate a SVN dump, one should not directly access the "SVN::Dump::Headers", "SVN::Dump::Property" and
       "SVN::Dump::Text" components of a "SVN::Dump::Record" object, but use the appropriate "set_...()" and "get_...()" methods of the record
       object.

       These methods compute the appropriate modifications of the header values, so that the "as_string()" method outputs the correct information
       after any modification of the record.

SEE ALSO

       "SVN::Dump::Headers", "SVN::Dump::Property", "SVN::Dump::Text".

COPYRIGHT

       Copyright 2006-2011 Philippe 'BooK' Bruhat, All Rights Reserved.

LICENSE

       This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

perl v5.10.1							    2011-03-22						    SVN::Dump::Record(3pm)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Import data from compressed file

Discussion started by: ap_gore79

2. Programming

Code Snippets

Discussion started by: wojtyla

3. Windows & DOS: Issues & Discussions

import data files from Unix to Windows?

Discussion started by: whatisthis

4. UNIX for Dummies Questions & Answers

How can import data files to XL sheet.

Discussion started by: koti_rama