Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Perl or Awk script to copy a part of text file.
Top Forums Shell Programming and Scripting Perl or Awk script to copy a part of text file. Post 302368832 by asandy1234 on Thursday 5th of November 2009 06:27:49 PM
Old 11-05-2009
Quote:
Originally Posted by frans
Doesn't the ETL module of SAS DI provide any ability to pre-process xml data ?
Don't you have any tool (SQL...) to extract data from the DB2 ? It could then be processed and stored in an understandable format for the ETL.
It would be a pity that the ETL could'nt do that by entering a couple of lines of code to configure it.
Bash, Awk and Sed can do a lot, Perl could be more appropriate, hey can all proccess any kind of CSV by coding scripts.
Hi Frans,
Ok let me explain the complexity of the process.
First of all SAS DI processes xml well, the issue here is all the xml doesn't have one XSD. There are more than 100 XSD's to process these xml's and now we have to group xml's according to their XSD's to be able for any xml parser to read it(which by itself is a big process and SAS doesn't recognize XSD if it is more than three levels so we need to create an equivalent .map file) secondly reading all xml thru SAS DI will translate into 100's of tables.
We can reduce the number tables by selecting only the nodes we need, but again there is complexity in associating the records from xml tables to their original records from db2 table as there is no link inside the xml.
Next is there is not much time line for this project(1 month) and finally the memory and the load window constrains, so taking all this into consideration we thought it will be better if we could process the xml as string.

Thanks,
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Shell script to search for text in a file and copy file

Compete noob question.... I need a script to search through a directory and find files containing text string abcde1234 for example and then copy that file with that text string to another directory help please :eek: (9 Replies)
Discussion started by: imeadows
9 Replies

2. Shell Programming and Scripting

awk, perl Script for processing a single line text file

I need a script to process a huge single line text file: The sample of the text is: "forward_inline_item": "Inline", "options_region_Australia": "Australia", "server_event_err_msg": "There was an error attempting to save", "Token": "Yes", "family": "Family","pwd_login_tab": "Enter Your... (1 Reply)
Discussion started by: hmsadiq
1 Replies

3. Shell Programming and Scripting

shell script to take input from a text file and perform check on each servers and copy files

HI all, I want to script where all the server names will be in a text file like server1 server2 server3 . and the script should take servernames from a text file and perform copy of files if the files are not present on those servers.after which it should take next servername till the end of... (0 Replies)
Discussion started by: joseph.dmello
0 Replies

4. Shell Programming and Scripting

search needed part in text file (awk?)

Hello! I have text file: From aaa@bbb Fri Jun 1 10:04:29 2010 --____OSPHWOJQGRPHNTTXKYGR____ Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline My code '234565'. ... (2 Replies)
Discussion started by: candyme
2 Replies

5. UNIX for Dummies Questions & Answers

Copy a part of file

Hi, I want to copy text between expressions ">bcr1" and ">bcr2" to another file. Any simple solutions? Thanks (4 Replies)
Discussion started by: alpesh
4 Replies

6. UNIX for Dummies Questions & Answers

Copy the last part since the file has been updated

Input File1 constatntly running and growing in size. My Program Erorr ddmmyy hh:mm:ss My Program Error **Port 123 terminated **ID PIN 12345 Comamnd Successful Command Terminated Command Successful Command Terminated **My Program Erorr ddmmyy hh:mm:ss My Program Error **Port 345... (3 Replies)
Discussion started by: eurouno
3 Replies

7. Shell Programming and Scripting

Copy part of file between two strings to another

I am a newbie to shell scripting I have a large log file , i need to work on the part of the log file for a particular date. Is there a way to find the first occurance of the date string and last occurance of the next day date date string and move this section to a new file. to explain it... (3 Replies)
Discussion started by: swayam123
3 Replies

8. Shell Programming and Scripting

Copy a file from local host to a list of remote hosts --- perl script

Hi friends, i need to prepare a script ( in perl) i have a file called "demo.exe" in my local unix host. i have a list of remote hosts in a file "hosts.txt" now i need to push "demo.exe" file to all the hosts in "hosts.txt" file. for this i need to prepare a script(in perl, but shell... (5 Replies)
Discussion started by: siva kumar
5 Replies

9. Shell Programming and Scripting

Not able to copy the file in perl cgi script

Hello experts, I am facing an very typical problem and hope the issue can be solved. I have a page download.cgi in /cgi-bin folder. use CGI; use CGI::Carp qw ( fatalsToBrowser ); use File::Copy copy("C:\\Program Files\\Apache Software... (8 Replies)
Discussion started by: scriptscript
8 Replies

10. Shell Programming and Scripting

Help in UNIX shell to copy part of file name to new file name

Hi, I want to do the following in a Unix shell script and wonder if someone could assist me? I want to take files in a specific directory that start with the name pxpur012 and copy them to the same directory with the file name not containg pxpur012. For example, I have files like... (4 Replies)
Discussion started by: lnemitz
4 Replies

LEARN ABOUT DEBIAN

evolvotron_mutate

EVOLVOTRON_MUTATE(1)						    Evolvotron						      EVOLVOTRON_MUTATE(1)

NAME

       evolvotron_mutate - Render an evolvotron function tree to an image.

SYNOPSIS

       evolvotron_mutate < function_in.xml > function_out.xml

       evolvotron_mutate -g > function_out.xml

DESCRIPTION

       evolvotron_mutate either mutates an existing image function (saved to an XML file), or (with the -g option) creates a new image function.

       The mutation parameters and function weightings are the same as used by evolvotron in its default reset state.

       See the evolvotron user manual (accessible from the evolvotron application's Help menu) for more information on image functions.

COMMANDLINE OPTIONS

       -g   Specifies that no function should be read from standard input.  The output function is created at random.

EXAMPLES

       evolvotron_mutate -g | tee function0.xml | evolvotron_render function0.ppm

       evolvtron_mutate < function0.xml > function1.xml

AUTHOR

       evolvotron_mutate  was written by Tim Day (www.timday.com) and is released under the conditions of the GNU General Public License.  See the
       file LICENSE supplied with the source code for details.

SEE ALSO

       evolvotron(1), evolvotron_render(1)

www.timday.com							    25 Feb 2004 					      EVOLVOTRON_MUTATE(1)
All times are GMT -4. The time now is 07:24 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy