Sponsored Content
Full Discussion: Help with Split shell script
Top Forums Shell Programming and Scripting Help with Split shell script Post 302304605 by cfajohnson on Monday 6th of April 2009 10:47:34 PM
Old 04-06-2009
Quote:
Originally Posted by amtrip
Hi all,

Following is the shell script which I am using for splitting the contents from a flat file to the multiple XMLs. The problem is that this script is working file when the input file is having 10000 lines. When the number of lines increases; the performance degrades drastically.

That's what happens when you call multiple external commands for every line of the file.
Quote:


Code:
 
#!/bin/bash
# This Shell Script reads the Input Master XML file line by line and splits into multiple XMLs.
        #Name of the Input Master XML file
        file=$1/MASTER_XML.out
        #Creating Directory in AFSOutput Folder
        outputdir=`date +%m%d%Y_%H%M`
        mkdir $1/$outputdir
        chmod 777 $1/$outputdir
        #Reading the filename column from the XML
        filename=($(grep -o "[0-9A-Za-z]\+\.[0-9A-Za-z]\+\.[tif]\{2,3\}" $1/MASTER_XML.out))
        x=0
        while [ $x -lt $(wc -l <$file) ]
                do
                        let x=x+1
 head -n $x $file | tail -n  1  >$1/$outputdir/${filename["$x-1"]%.*}.xml
 chmod 777 $1/$outputdir/${filename["$x-1"]%.*}.xml
 done


To read a file line by line, use a while read loop:

Code:
n=0
while IFS= read -r line
do
  n=$(( $n + 1 ))
  : do whatever
done < FILENAME

However, with a large file, you would be better off using awk rather than a pure shell script.
Quote:
Code:
        exit 0

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script to split XML file

Hi, I'm experiencing difficulty in loading an XML file to an Oracle destination table.I keep running into a memory problem due to the large size of the file. I want to split the XML file into several smaller files based on the keyword(s)/tags : '' and '' and would like to use a Unix shell... (2 Replies)
Discussion started by: bayflash27
2 Replies

2. UNIX for Dummies Questions & Answers

awk Shell Script error : "Syntax Error : `Split' unexpected

hi there i write one awk script file in shell programing the code is related to dd/mm/yy to month, day year format but i get an error please can anybody help me out in this problem ?????? i give my code here including error awk ` # date-month -- convert mm/dd/yy to month day,... (2 Replies)
Discussion started by: Herry
2 Replies

3. Shell Programming and Scripting

Shell script to parse/split input string and display the tokens

Hi, How do I parse/split lines (strings) read from a file and display the individual tokens in a shell script? Given that the length of individual lines is not constant and number of tokens in each line is also not constant. The input file could be as below: ... (3 Replies)
Discussion started by: yajaykumar
3 Replies

4. Shell Programming and Scripting

SPLIT STRING in bash shell script

i need one help.... if i have a string like aaaaa,bbbbb,ccccc,aaaaa How to to split the string and check howmany times aaaaa will be in that string? Thanks (7 Replies)
Discussion started by: karthinvk
7 Replies

5. Shell Programming and Scripting

Split line to multiple files Awk/Sed/Shell Script help

Hi, I need help to split lines from a file into multiple files. my input look like this: 13 23 45 45 6 7 33 44 55 66 7 13 34 5 6 7 87 45 7 8 8 9 13 44 55 66 77 8 44 66 88 99 6 I want to split every 3 lines from this file to be written to individual files. (3 Replies)
Discussion started by: saint2006
3 Replies

6. Shell Programming and Scripting

Shell Script:split the values

Hi, I have file like below as file.txt Linux sptpp1a 2.6.18 348.1.1.el5 xxxxx 05/02/2013 %usr %sys %wio %idle 14:18:30 0 1 0 99 14:18:40 1 1 0 98 14:18:50 1 1 0 98 14:19:00 ... (5 Replies)
Discussion started by: Mani_apr08
5 Replies

7. Shell Programming and Scripting

How to split files using shell script?

solid top facet normal 0 1 0 outer loop vertex 0 1 0 vertex 1 1 1 vertex 1 1 0 endloop endfacet facet normal 0 1 0 outer loop vertex 0 1 0 vertex 0 1 1 vertex 1 1 1 endloop endfacet endsolid top solid bottom facet normal 0 -1 ... (3 Replies)
Discussion started by: linuxUser_
3 Replies

8. Shell Programming and Scripting

Split a file into two different file using shell script

could you please provide solution to split file and load into different files in shell script. cat test.txt "01"|""emp_name"":ram|""emp_sal"":600000|""emp_value"":""CREDITCARD""|""1410"":] "02"|""emp_name"":syam|""emp_sal"":500000|""emp_value"":""voucher""|""1610"":]|""1510"":]... (7 Replies)
Discussion started by: shabeena
7 Replies

9. UNIX for Beginners Questions & Answers

Shell script to split data with a delimiter having chars and special chars

Hi Team, I have a file a1.txt with data as follows. dfjakjf...asdfkasj</EnableQuotedIDs><SQL><SelectStatement modified='1' type='string'><! The delimiter string: <SelectStatement modified='1' type='string'><! dlm="<SelectStatement modified='1' type='string'><! The above command is... (7 Replies)
Discussion started by: kmanivan82
7 Replies

10. UNIX for Beginners Questions & Answers

Shell script to Split matrix file with delimiter into multiple files

I have a large semicolon delimited file with thousands of columns and many thousands of line. It looks like: ID1;ID2;ID3;ID4;A_1;B_1;C_1;A_2;B_2;C_2;A_3;B_3;C_3 AA;ax;ay;az;01;02;03;04;05;06;07;08;09 BB;bx;by;bz;03;05;33;44;15;26;27;08;09 I want to split this table in to multiple files: ... (1 Reply)
Discussion started by: trymega
1 Replies
exit(1)                                                            User Commands                                                           exit(1)

NAME
exit, return, goto - shell built-in functions to enable the execution of the shell to advance beyond its sequence of steps SYNOPSIS
sh exit [n] return [n] csh exit [ ( expr )] goto label ksh *exit [n] *return [n] DESCRIPTION
sh exit will cause the calling shell or shell script to exit with the exit status specified by n. If n is omitted the exit status is that of the last command executed (an EOF will also cause the shell to exit.) return causes a function to exit with the return value specified by n. If n is omitted, the return status is that of the last command exe- cuted. csh exit will cause the calling shell or shell script to exit, either with the value of the status variable or with the value specified by the expression expr. The goto built-in uses a specified label as a search string amongst commands. The shell rewinds its input as much as possible and searches for a line of the form label: possibly preceded by space or tab characters. Execution continues after the indicated line. It is an error to jump to a label that occurs between a while or for built-in command and its corresponding end. ksh exit will cause the calling shell or shell script to exit with the exit status specified by n. The value will be the least significant 8 bits of the specified status. If n is omitted then the exit status is that of the last command executed. When exit occurs when executing a trap, the last command refers to the command that executed before the trap was invoked. An end-of-file will also cause the shell to exit except for a shell which has the ignoreeof option (See set below) turned on. return causes a shell function or '.' script to return to the invoking script with the return status specified by n. The value will be the least significant 8 bits of the specified status. If n is omitted then the return status is that of the last command executed. If return is invoked while not in a function or a '.' script, then it is the same as an exit. On this man page, ksh(1) commands that are preceded by one or two * (asterisks) are treated specially in the following ways: 1. Variable assignment lists preceding the command remain in effect when the command completes. 2. I/O redirections are processed after variable assignments. 3. Errors cause a script that contains them to abort. 4. Words, following a command preceded by ** that are in the format of a variable assignment, are expanded with the same rules as a vari- able assignment. This means that tilde substitution is performed after the = sign and word splitting and file name generation are not performed. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | |Availability |SUNWcsu | +-----------------------------+-----------------------------+ SEE ALSO
break(1), csh(1), ksh(1), sh(1), attributes(5) SunOS 5.10 15 Apr 1994 exit(1)
All times are GMT -4. The time now is 03:45 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy