Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Efficient shell script code
Top Forums Shell Programming and Scripting Efficient shell script code Post 302830553 by shoaibjameel123 on Tuesday 9th of July 2013 05:32:31 AM
Old 07-09-2013
Efficient shell script code
Hi all,
I am working on an extremely large collection of text data (about 2 million XML files) in a directory. I have changed the extension from .xml to .dat. Right now I am using this code to remove the XML tags, but the code is way too slow. It seems that it is taking fore-ever:
Code:
#ls -1 *.dat | while read page
find . -name "*.dat" -print | while read page
do
links -dump $page>$page.txt
done

Just to let the readers know that the commented line with ls does not even work as it gives Argument list too long message.

Then I modified the code, and came up with this:
Code:
#ls -1 *.dat | while read page
#find . -name "*.dat" -print | while read page
num=1
for page in *.dat;
do
links -dump $page>$page.txt
let num=num+1
done

Just wish to know will this speed up my task? What I want to do is that instead of doing ls or find, I should generate the filename using my code, and the program should then process that file which has been automatically generated. The trick that I have used is that I have re-named all the 2 million files with "contiguous" numbers 1.dat, 2.dat, 3.dat, 4.dat and so on without leaving any number in between and then using a counter, I generate those numbers and read those files.
Or, Is there any other better way to fasten up my task? I am using Linux with BASH.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

code formatter for shell script

hello, do anybody know a program to format a shell script code ? i tried "editrocket.com" but this product doesn't format a shell script code. i searched for programs but can't find a shell script code formatter. i have to change a shell script and the style of code is ..... regards (5 Replies)
Discussion started by: bora99
5 Replies

2. Programming

Making FORTRAN code more efficient

Hi, I have a very large, very old FORTRAN code that I work with. The code is quite messy and I was wondering if I can speed up execution time by finding subroutines that code execution spends the most time in. Is there any kind of software I can use to see where the code spends most of the... (1 Reply)
Discussion started by: rks171
1 Replies

3. Emergency UNIX and Linux Support

Help to make awk script more efficient for large files

Hello, Error awk: Internal software error in the tostring function on TS1101?05044400?.0085498227?0?.0011041461?.0034752266?.00397045?0?0?0?0?0?0?11/02/10?09/23/10???10?no??0??no?sct_det3_10_20110516_143936.txt What it is It is a unix shell script that contains an awk program as well as... (4 Replies)
Discussion started by: script_op2a
4 Replies

4. Programming

Help with make this Fortran code more efficient (in HPC manner)

Hi there, I had run into some fortran code to modify. Obviously, it was written without thinking of high performance computing and not parallelized... Now I would like to make the code "on track" and parallel. After a whole afternoon thinking, I still cannot find where to start. Can any one... (3 Replies)
Discussion started by: P_E_M_Lee
3 Replies

5. UNIX for Dummies Questions & Answers

Script Shell in java code

Hello, I try to run a script shell from a java program: but it runs only if i do :chmod 777 myShellScript in the terminal Please how can i insert chmod 777 in my java code without going through the terminal? Thank you (1 Reply)
Discussion started by: chercheur857
1 Replies

6. Shell Programming and Scripting

Efficient rewrite of code?

egrep -v "#" ${SERVERS} | while read shosts do grep -Pi "|" ${LOGFILE} | egrep "${snhosts}" | egrep "NOTIFICATION:" | awk -F";" '{print $3}' | sort -n | uniq | while read CEXIST do ... (6 Replies)
Discussion started by: SkySmart
6 Replies

7. Shell Programming and Scripting

Help with Shell script code

Hello all, I am in a middle of an assignment and i would appreciate any help. How can i write a bash shell script code that checks if all elements in an array are the same numbers. I mean -->array = ( 0,0,0,0,0 ) ( e.g., if then return "OK' fi ) Thank you in advance, (9 Replies)
Discussion started by: Geekie
9 Replies

8. Shell Programming and Scripting

How to capture the exit code of a shell script in a perl script.?

hi, i want to pop up an alert box using perl script. my requirement is. i am using a html page which calls a perl script. this perl script calls a shell script.. after the shell script ends its execution, i am using exit 0 to terminate the shell script successfully and exit 1 to terminate the... (3 Replies)
Discussion started by: Little
3 Replies

9. Shell Programming and Scripting

Colour code in shell script

Hello, I am trying to colour code a single word in whole line. Can you please help. I am able to colour code the whole line but not able to do only for single word Query) I want to echo below line and colur code red to word FAILED only. This server is FAILED in check. (2 Replies)
Discussion started by: saurabh84g
2 Replies

10. Shell Programming and Scripting

Search and replace multiple patterns in a particular column only - efficient script

Hi Bigshots, I have a pattern file with two columns. I have another data file. If column 1 in the pattern file appears as the 4th column in the data file, I need to replace it (4th column of data file) with column 2 of the pattern file. If the pattern is found in any other column, it should not... (6 Replies)
Discussion started by: ss112233
6 Replies

LEARN ABOUT DEBIAN

sdpa

SDPA(1) 						      General Commands Manual							   SDPA(1)

NAME

       SDPA - High-performance software package for SemiDefinite Programs

SYNOPSIS

       sdpa DataFile OutputFile [InitialPtFile] [-pt parameters] [-dimacs] [-numThreads  numThreads]
       sdpa [options] files...
       sdpa --version

DESCRIPTION

       sdpa  - "SDPA (SemiDefinite Programming Algorithm)" is one of the most efficient and stable software packages for solving SDPs based on the
       primal-dual interior-point method.

       SDP (SemiDefinite Program) is used for financial engineering, machine learning, control theory, sensor network problem, quantum	chemistry,
       quantum information, combinatorial optimizaiton, polynomial optimization, and so on.

       Futher information on SDP and SDPA can be found at
       http://sdpa.sourceforge.net/

OPTIONS

       ---- option type 1 ----
       sdpa DataFile OutputFile [InitialPtFile] [-pt parameters] [-dimacs] [-numThreads numThreads]
       parameters = 0 default, 1 fast (unstable), 2 slow (stable)
       example1-1: sdpa example1.dat example1.result
       example1-2: sdpa example1.dat-s example1.result
       example1-3: sdpa example1.dat example1.result example1.ini
       example1-4: sdpa example1.dat example1.result -pt 2
       example1-5: sdpa example1.dat example1.result -dimacs
       example1-6: sdpa example1.dat example1.result -numThreads 4
       ---- option type 2 ----
       sdpa [option filename]+
	 -dd : data dense :: -ds : data sparse
	 -id : init dense :: -is : init sparse
	 -o  : output	  :: -p  : parameter
	 -pt : parameters , 0 default, 1 fast (unstable)
			    2 slow (stable)
       example2-1: sdpa -o example1.result -dd example1.dat
       example2-2: sdpa -ds example1.dat-s -o example1.result -p param.sdpa
       example2-3: sdpa -ds example1.dat-s -o example1.result -pt 2
       example2-4: sdpa -ds example1.dat-s -o example1.result -dimacs
       example2-5: sdpa -ds example1.dat-s -o example1.result -numThreads 4
       ---- option type 3 ----
       sdpa --version
       to print out version and exit.

       PARAMETER_FILE
	      is decided by the following priority
		 1: The file assigned by '-p' option of 'option type 2'.
		     For 'option type1', this is skipped.
		 2: ./param.sdpa
		     For 'option type2', this is skipped.
		 3: /usr/share/sdpa/param.sdpa
		 4: Default parameter

       -dimacs
	      printout dimacs information incurring additional computation cost

       -numThreads
	      number of pthreads for internal computation

AUTHOR

       SDPA was written by SDPA Project <sdpa-developers@lists.sourceforge.net>.

       This manual page was written by Makoto Yamashita <Makoto.Yamashita@is.titech.ac.jp>.

								   July 28, 2011							   SDPA(1)
All times are GMT -4. The time now is 09:26 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy