Help with Split shell script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with Split shell script
# 1  
Old 04-06-2009
Help with Split shell script

Hi all,

Following is the shell script which I am using for splitting the contents from a flat file to the multiple XMLs. The problem is that this script is working file when the input file is having 10000 lines. When the number of lines increases; the performance degrades drastically. Please help me ASAP for this:

Code:
 
#!/bin/bash
# This Shell Script reads the Input Master XML file line by line and splits into multiple XMLs.
        #Name of the Input Master XML file
        file=$1/MASTER_XML.out
        #Creating Directory in AFSOutput Folder
        outputdir=`date +%m%d%Y_%H%M`
        mkdir $1/$outputdir
        chmod 777 $1/$outputdir
        #Reading the filename column from the XML
        filename=($(grep -o "[0-9A-Za-z]\+\.[0-9A-Za-z]\+\.[tif]\{2,3\}" $1/MASTER_XML.out))
        x=0
        while [ $x -lt $(wc -l <$file) ]
                do
                        let x=x+1
 head -n $x $file | tail -n  1  >$1/$outputdir/${filename["$x-1"]%.*}.xml
 chmod 777 $1/$outputdir/${filename["$x-1"]%.*}.xml
 done
        exit 0

Please help

Thanks,
Amit

Last edited by jim mcnamara; 04-06-2009 at 04:25 PM..
# 2  
Old 04-06-2009
Quote:
Originally Posted by amtrip
Hi all,

Following is the shell script which I am using for splitting the contents from a flat file to the multiple XMLs. The problem is that this script is working file when the input file is having 10000 lines. When the number of lines increases; the performance degrades drastically.

That's what happens when you call multiple external commands for every line of the file.
Quote:


Code:
 
#!/bin/bash
# This Shell Script reads the Input Master XML file line by line and splits into multiple XMLs.
        #Name of the Input Master XML file
        file=$1/MASTER_XML.out
        #Creating Directory in AFSOutput Folder
        outputdir=`date +%m%d%Y_%H%M`
        mkdir $1/$outputdir
        chmod 777 $1/$outputdir
        #Reading the filename column from the XML
        filename=($(grep -o "[0-9A-Za-z]\+\.[0-9A-Za-z]\+\.[tif]\{2,3\}" $1/MASTER_XML.out))
        x=0
        while [ $x -lt $(wc -l <$file) ]
                do
                        let x=x+1
 head -n $x $file | tail -n  1  >$1/$outputdir/${filename["$x-1"]%.*}.xml
 chmod 777 $1/$outputdir/${filename["$x-1"]%.*}.xml
 done


To read a file line by line, use a while read loop:

Code:
n=0
while IFS= read -r line
do
  n=$(( $n + 1 ))
  : do whatever
done < FILENAME

However, with a large file, you would be better off using awk rather than a pure shell script.
Quote:
Code:
        exit 0

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Shell script to Split matrix file with delimiter into multiple files

I have a large semicolon delimited file with thousands of columns and many thousands of line. It looks like: ID1;ID2;ID3;ID4;A_1;B_1;C_1;A_2;B_2;C_2;A_3;B_3;C_3 AA;ax;ay;az;01;02;03;04;05;06;07;08;09 BB;bx;by;bz;03;05;33;44;15;26;27;08;09 I want to split this table in to multiple files: ... (1 Reply)
Discussion started by: trymega
1 Replies

2. UNIX for Beginners Questions & Answers

Shell script to split data with a delimiter having chars and special chars

Hi Team, I have a file a1.txt with data as follows. dfjakjf...asdfkasj</EnableQuotedIDs><SQL><SelectStatement modified='1' type='string'><! The delimiter string: <SelectStatement modified='1' type='string'><! dlm="<SelectStatement modified='1' type='string'><! The above command is... (7 Replies)
Discussion started by: kmanivan82
7 Replies

3. Shell Programming and Scripting

Split a file into two different file using shell script

could you please provide solution to split file and load into different files in shell script. cat test.txt "01"|""emp_name"":ram|""emp_sal"":600000|""emp_value"":""CREDITCARD""|""1410"":] "02"|""emp_name"":syam|""emp_sal"":500000|""emp_value"":""voucher""|""1610"":]|""1510"":]... (7 Replies)
Discussion started by: shabeena
7 Replies

4. Shell Programming and Scripting

How to split files using shell script?

solid top facet normal 0 1 0 outer loop vertex 0 1 0 vertex 1 1 1 vertex 1 1 0 endloop endfacet facet normal 0 1 0 outer loop vertex 0 1 0 vertex 0 1 1 vertex 1 1 1 endloop endfacet endsolid top solid bottom facet normal 0 -1 ... (3 Replies)
Discussion started by: linuxUser_
3 Replies

5. Shell Programming and Scripting

Shell Script:split the values

Hi, I have file like below as file.txt Linux sptpp1a 2.6.18 348.1.1.el5 xxxxx 05/02/2013 %usr %sys %wio %idle 14:18:30 0 1 0 99 14:18:40 1 1 0 98 14:18:50 1 1 0 98 14:19:00 ... (5 Replies)
Discussion started by: Mani_apr08
5 Replies

6. Shell Programming and Scripting

Split line to multiple files Awk/Sed/Shell Script help

Hi, I need help to split lines from a file into multiple files. my input look like this: 13 23 45 45 6 7 33 44 55 66 7 13 34 5 6 7 87 45 7 8 8 9 13 44 55 66 77 8 44 66 88 99 6 I want to split every 3 lines from this file to be written to individual files. (3 Replies)
Discussion started by: saint2006
3 Replies

7. Shell Programming and Scripting

SPLIT STRING in bash shell script

i need one help.... if i have a string like aaaaa,bbbbb,ccccc,aaaaa How to to split the string and check howmany times aaaaa will be in that string? Thanks (7 Replies)
Discussion started by: karthinvk
7 Replies

8. Shell Programming and Scripting

Shell script to parse/split input string and display the tokens

Hi, How do I parse/split lines (strings) read from a file and display the individual tokens in a shell script? Given that the length of individual lines is not constant and number of tokens in each line is also not constant. The input file could be as below: ... (3 Replies)
Discussion started by: yajaykumar
3 Replies

9. UNIX for Dummies Questions & Answers

awk Shell Script error : "Syntax Error : `Split' unexpected

hi there i write one awk script file in shell programing the code is related to dd/mm/yy to month, day year format but i get an error please can anybody help me out in this problem ?????? i give my code here including error awk ` # date-month -- convert mm/dd/yy to month day,... (2 Replies)
Discussion started by: Herry
2 Replies

10. Shell Programming and Scripting

Shell script to split XML file

Hi, I'm experiencing difficulty in loading an XML file to an Oracle destination table.I keep running into a memory problem due to the large size of the file. I want to split the XML file into several smaller files based on the keyword(s)/tags : '' and '' and would like to use a Unix shell... (2 Replies)
Discussion started by: bayflash27
2 Replies
Login or Register to Ask a Question