Sponsored Content
Top Forums Shell Programming and Scripting Shell script reading file slow Post 302906120 by rbatte1 on Tuesday 17th of June 2014 09:30:38 AM
Old 06-17-2014
It might be worth putting the time stamp in your messages so you can be sure where the time is spent:-
Code:
echo "`date` starting unziping files in $Process_dir"
gunzip $Processing_dir/*.gz
echo "`date` done unziping files in $Process_dir"

For every record you read in, you have the following :-
Code:
cname=$(echo ${line} | awk '{split($11,c,"/"); print c[3]}')

This means that for every single record, you will start several processes (sub-shell, awk and possibly an echo) and this takes time. These processes are not re-used next time round your loop so it's very expensive. You then do pretty much the same with this for every record too:-
Code:
scode=$(echo ${line} | awk -F"[ ]" '{print $9}')

Assuming that what you really want from these two lines is a section of the record, can you share a sample of the data and I'm sure we can give you a much more efficient process. Highlight what you need to use from each record.

Can you explain your logic so we can be sure of what you are trying to achieve please.

I can give you ksh commands if that's what is more useful, but someone may be able to code a single awk command to remove your loop entirely which would be better still.



Robin
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Reading file names from a file and executing the relative file from shell script

Hi How can i dynamically read files names from a list file and execute them from a single shell script. Please help its urgent Thanks in Advance (4 Replies)
Discussion started by: anushilrai
4 Replies

2. Shell Programming and Scripting

Reading data from a file through shell script

There is one Text file data.txt. Data within this file looks like: a.sql b.sql c.sql d.sql ..... ..... want to write a shell script which will access these values within a loop, access one value at a time and store into a variable. can anyone plz help me. (2 Replies)
Discussion started by: Dip
2 Replies

3. Shell Programming and Scripting

file reading through shell script

For reading a file through shell script I am using yhe code : while read line do echo $line done<data.txt It reads all the line of that file data.txt. Content of data.txt looks like: code=y sql=y total no of sql files=4 a.sql b.sql c.sql d.sql cpp=n c=y total no of c files=1 (4 Replies)
Discussion started by: Dip
4 Replies

4. Shell Programming and Scripting

File reading problem via shell script

Hi, Data file named parameter contains : DB=y Alter_def.sql Create_abc.sql SQL=y database.sql my_data.sql To read this file I use var_sql=$(awk -F= '$1 == "SQL" { print $2 }' parameter.txt) if then sql_f_name=`grep "\.sql" parameter.txt` echo $sql_f_name fi (2 Replies)
Discussion started by: Dip
2 Replies

5. Shell Programming and Scripting

Reading the Properties File From Shell script

Hi, I am new to the shell script please I need help for following question. I have properties file name called "com.test.properties" I have No of key values in this properties. com.person.name = xyz com.person.age = 55 com.person.address = hello I want read this properties but i... (1 Reply)
Discussion started by: venukjs
1 Replies

6. Shell Programming and Scripting

Reading a property file through shell script???

Hi! i need a script that can read a property file. i.e., A script to read a "property" from property file. Read the property value and based on value of property, decide whether to start the some dataload activity or not. Its urngent. Can anyone help me out???:( (7 Replies)
Discussion started by: sukhdip
7 Replies

7. Shell Programming and Scripting

Error while reading from a file in shell script

Hi All, I'm writing a script to read a file line by line and then perform awk function on it. I am getting an error . My file has one name in it "James". I'm expecting my o/p to be youareJamesbond James ./users.sh: line 7: =: command not found #script to read file line by line #adding... (5 Replies)
Discussion started by: Irishboy24
5 Replies

8. Shell Programming and Scripting

Reading a csv file using shell script

Hello All, I have a csv file that looks like below ProdId_A,3.3.3,some text,some/text,sometext_1.2.3 ProdId_B,3.3.3,some text,some/text,sometext_1.2.3 ProdId_C,3.3.3,some text,some/text,sometext_1.2.3 ProdId_A,6.6.6,some text,some/text,sometext_9.9.9 I will get ProdId from... (5 Replies)
Discussion started by: anand.shah
5 Replies

9. UNIX for Dummies Questions & Answers

C-Shell script help reading from txt file

I need to write a C-Shell script with these properties: It should accept two arguments on the command line. The first argument is the name of a file which contains a list of names, and the second argument is the name of a directory. For each file in the directory, the script should print the... (1 Reply)
Discussion started by: cerce
1 Replies

10. Shell Programming and Scripting

Slow Running Script (Reading 8000 lines)

Slow runnin script. The problem seems to be the sed calls. In summary the script reads list of users in file1. For each username search two files (file 1 & file2) for the username and get the value in the next line after "=". Compare these values with each other. If the same then output... (9 Replies)
Discussion started by: u20sr
9 Replies
NPM-RUN-SCRIPT(1)                                                                                                                NPM-RUN-SCRIPT(1)

NAME
npm-run-script - Run arbitrary package scripts SYNOPSIS
npm run-script <command> [--silent] [-- <args>...] alias: npm run DESCRIPTION
This runs an arbitrary command from a package's "scripts" object. If no "command" is provided, it will list the available scripts. run[-script] is used by the test, start, restart, and stop commands, but can be called directly, as well. When the scripts in the package are printed out, they're separated into lifecycle (test, start, restart) and directly-run scripts. As of ` https://blog.npmjs.org/post/98131109725/npm-2-0-0, you can use custom arguments when executing scripts. The special option -- is used by getopt https://goo.gl/KxMmtG to delimit the end of the options. npm will pass all the arguments after the -- directly to your script: npm run test -- --grep="pattern" The arguments will only be passed to the script specified after npm run and not to any pre or post script. The env script is a special built-in command that can be used to list environment variables that will be available to the script at run- time. If an "env" command is defined in your package, it will take precedence over the built-in. In addition to the shell's pre-existing PATH, npm run adds node_modules/.bin to the PATH provided to scripts. Any binaries provided by locally-installed dependencies can be used without the node_modules/.bin prefix. For example, if there is a devDependency on tap in your package, you should write: "scripts": {"test": "tap test/*.js"} instead of "scripts": {"test": "node_modules/.bin/tap test/*.js"} to run your tests. The actual shell your script is run within is platform dependent. By default, on Unix-like systems it is the /bin/sh command, on Windows it is the cmd.exe. The actual shell referred to by /bin/sh also depends on the system. As of ` https://github.com/npm/npm/releases/tag/v5.1.0 you can customize the shell with the script-shell configuration. Scripts are run from the root of the module, regardless of what your current working directory is when you call npm run. If you want your script to use different behavior based on what subdirectory you're in, you can use the INIT_CWD environment variable, which holds the full path you were in when you ran npm run. npm run sets the NODE environment variable to the node executable with which npm is executed. Also, if the --scripts-prepend-node-path is passed, the directory within which node resides is added to the PATH. If --scripts-prepend-node-path=auto is passed (which has been the default in npm v3), this is only performed when that node executable is not found in the PATH. If you try to run a script without having a node_modules directory and it fails, you will be given a warning to run npm install, just in case you've forgotten. You can use the --silent flag to prevent showing npm ERR! output on error. You can use the --if-present flag to avoid exiting with a non-zero exit code when the script is undefined. This lets you run potentially undefined scripts without breaking the execution chain. SEE ALSO
o npm help 7 scripts o npm help test o npm help start o npm help restart o npm help stop o npm help 7 config January 2019 NPM-RUN-SCRIPT(1)
All times are GMT -4. The time now is 02:25 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy