Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Help with Split shell script
Top Forums Shell Programming and Scripting Help with Split shell script Post 302304605 by cfajohnson on Monday 6th of April 2009 10:47:34 PM
Old 04-06-2009
Quote:
Originally Posted by amtrip
Hi all,

Following is the shell script which I am using for splitting the contents from a flat file to the multiple XMLs. The problem is that this script is working file when the input file is having 10000 lines. When the number of lines increases; the performance degrades drastically.

That's what happens when you call multiple external commands for every line of the file.
Quote:


Code:
 
#!/bin/bash
# This Shell Script reads the Input Master XML file line by line and splits into multiple XMLs.
        #Name of the Input Master XML file
        file=$1/MASTER_XML.out
        #Creating Directory in AFSOutput Folder
        outputdir=`date +%m%d%Y_%H%M`
        mkdir $1/$outputdir
        chmod 777 $1/$outputdir
        #Reading the filename column from the XML
        filename=($(grep -o "[0-9A-Za-z]\+\.[0-9A-Za-z]\+\.[tif]\{2,3\}" $1/MASTER_XML.out))
        x=0
        while [ $x -lt $(wc -l <$file) ]
                do
                        let x=x+1
 head -n $x $file | tail -n  1  >$1/$outputdir/${filename["$x-1"]%.*}.xml
 chmod 777 $1/$outputdir/${filename["$x-1"]%.*}.xml
 done


To read a file line by line, use a while read loop:

Code:
n=0
while IFS= read -r line
do
  n=$(( $n + 1 ))
  : do whatever
done < FILENAME

However, with a large file, you would be better off using awk rather than a pure shell script.
Quote:
Code:
        exit 0

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script to split XML file

Hi, I'm experiencing difficulty in loading an XML file to an Oracle destination table.I keep running into a memory problem due to the large size of the file. I want to split the XML file into several smaller files based on the keyword(s)/tags : '' and '' and would like to use a Unix shell... (2 Replies)
Discussion started by: bayflash27
2 Replies

2. UNIX for Dummies Questions & Answers

awk Shell Script error : "Syntax Error : `Split' unexpected

hi there i write one awk script file in shell programing the code is related to dd/mm/yy to month, day year format but i get an error please can anybody help me out in this problem ?????? i give my code here including error awk ` # date-month -- convert mm/dd/yy to month day,... (2 Replies)
Discussion started by: Herry
2 Replies

3. Shell Programming and Scripting

Shell script to parse/split input string and display the tokens

Hi, How do I parse/split lines (strings) read from a file and display the individual tokens in a shell script? Given that the length of individual lines is not constant and number of tokens in each line is also not constant. The input file could be as below: ... (3 Replies)
Discussion started by: yajaykumar
3 Replies

4. Shell Programming and Scripting

SPLIT STRING in bash shell script

i need one help.... if i have a string like aaaaa,bbbbb,ccccc,aaaaa How to to split the string and check howmany times aaaaa will be in that string? Thanks (7 Replies)
Discussion started by: karthinvk
7 Replies

5. Shell Programming and Scripting

Split line to multiple files Awk/Sed/Shell Script help

Hi, I need help to split lines from a file into multiple files. my input look like this: 13 23 45 45 6 7 33 44 55 66 7 13 34 5 6 7 87 45 7 8 8 9 13 44 55 66 77 8 44 66 88 99 6 I want to split every 3 lines from this file to be written to individual files. (3 Replies)
Discussion started by: saint2006
3 Replies

6. Shell Programming and Scripting

Shell Script:split the values

Hi, I have file like below as file.txt Linux sptpp1a 2.6.18 348.1.1.el5 xxxxx 05/02/2013 %usr %sys %wio %idle 14:18:30 0 1 0 99 14:18:40 1 1 0 98 14:18:50 1 1 0 98 14:19:00 ... (5 Replies)
Discussion started by: Mani_apr08
5 Replies

7. Shell Programming and Scripting

How to split files using shell script?

solid top facet normal 0 1 0 outer loop vertex 0 1 0 vertex 1 1 1 vertex 1 1 0 endloop endfacet facet normal 0 1 0 outer loop vertex 0 1 0 vertex 0 1 1 vertex 1 1 1 endloop endfacet endsolid top solid bottom facet normal 0 -1 ... (3 Replies)
Discussion started by: linuxUser_
3 Replies

8. Shell Programming and Scripting

Split a file into two different file using shell script

could you please provide solution to split file and load into different files in shell script. cat test.txt "01"|""emp_name"":ram|""emp_sal"":600000|""emp_value"":""CREDITCARD""|""1410"":] "02"|""emp_name"":syam|""emp_sal"":500000|""emp_value"":""voucher""|""1610"":]|""1510"":]... (7 Replies)
Discussion started by: shabeena
7 Replies

9. UNIX for Beginners Questions & Answers

Shell script to split data with a delimiter having chars and special chars

Hi Team, I have a file a1.txt with data as follows. dfjakjf...asdfkasj</EnableQuotedIDs><SQL><SelectStatement modified='1' type='string'><! The delimiter string: <SelectStatement modified='1' type='string'><! dlm="<SelectStatement modified='1' type='string'><! The above command is... (7 Replies)
Discussion started by: kmanivan82
7 Replies

10. UNIX for Beginners Questions & Answers

Shell script to Split matrix file with delimiter into multiple files

I have a large semicolon delimited file with thousands of columns and many thousands of line. It looks like: ID1;ID2;ID3;ID4;A_1;B_1;C_1;A_2;B_2;C_2;A_3;B_3;C_3 AA;ax;ay;az;01;02;03;04;05;06;07;08;09 BB;bx;by;bz;03;05;33;44;15;26;27;08;09 I want to split this table in to multiple files: ... (1 Reply)
Discussion started by: trymega
1 Replies

LEARN ABOUT DEBIAN

insdseqget

INSDSEQGET(1)						     NCBI Tools User's Manual						     INSDSEQGET(1)

NAME

       insdseqget - format sequences from GenBank as an XML INSDSet

SYNOPSIS

       insdseqget [-] [-d filename] [-i filename] [-m n/p/b] [-n] [-o filename] [-v]

DESCRIPTION

       insdseqget retrieves biological sequence data from GenBank (according to an input list of GI accession numbers) and prints it out as an XML
       INSDSet document.

OPTIONS

       A summary of options is included below.

       -      Print usage message

       -d filename
	      Input file name for date list (desired accessions, one per line, followed by a blank line and a list of allowed dates, also one  per
	      line)

       -i filename
	      Input file name for GI list (default = stdin)

       -m n/p/b
	      Molecule type:
	      n      Nucleotide (default)
	      p      Protein
	      b      Both

       -n     Return only new records (for which the given GI refers to an old version)

       -o filename
	      Output file name for the XML INSDSet (default = stdout)

       -v     Fetch SNP Variations

AUTHOR

       The National Center for Biotechnology Information.

SEE ALSO

       asn2gb(1), gbseqget(1).

NCBI
								    2006-04-14							     INSDSEQGET(1)
All times are GMT -4. The time now is 09:23 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy