Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Shell script reading file slow
Top Forums Shell Programming and Scripting Shell script reading file slow Post 302906120 by rbatte1 on Tuesday 17th of June 2014 09:30:38 AM
Old 06-17-2014
It might be worth putting the time stamp in your messages so you can be sure where the time is spent:-
Code:
echo "`date` starting unziping files in $Process_dir"
gunzip $Processing_dir/*.gz
echo "`date` done unziping files in $Process_dir"

For every record you read in, you have the following :-
Code:
cname=$(echo ${line} | awk '{split($11,c,"/"); print c[3]}')

This means that for every single record, you will start several processes (sub-shell, awk and possibly an echo) and this takes time. These processes are not re-used next time round your loop so it's very expensive. You then do pretty much the same with this for every record too:-
Code:
scode=$(echo ${line} | awk -F"[ ]" '{print $9}')

Assuming that what you really want from these two lines is a section of the record, can you share a sample of the data and I'm sure we can give you a much more efficient process. Highlight what you need to use from each record.

Can you explain your logic so we can be sure of what you are trying to achieve please.

I can give you ksh commands if that's what is more useful, but someone may be able to code a single awk command to remove your loop entirely which would be better still.



Robin
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Reading file names from a file and executing the relative file from shell script

Hi How can i dynamically read files names from a list file and execute them from a single shell script. Please help its urgent Thanks in Advance (4 Replies)
Discussion started by: anushilrai
4 Replies

2. Shell Programming and Scripting

Reading data from a file through shell script

There is one Text file data.txt. Data within this file looks like: a.sql b.sql c.sql d.sql ..... ..... want to write a shell script which will access these values within a loop, access one value at a time and store into a variable. can anyone plz help me. (2 Replies)
Discussion started by: Dip
2 Replies

3. Shell Programming and Scripting

file reading through shell script

For reading a file through shell script I am using yhe code : while read line do echo $line done<data.txt It reads all the line of that file data.txt. Content of data.txt looks like: code=y sql=y total no of sql files=4 a.sql b.sql c.sql d.sql cpp=n c=y total no of c files=1 (4 Replies)
Discussion started by: Dip
4 Replies

4. Shell Programming and Scripting

File reading problem via shell script

Hi, Data file named parameter contains : DB=y Alter_def.sql Create_abc.sql SQL=y database.sql my_data.sql To read this file I use var_sql=$(awk -F= '$1 == "SQL" { print $2 }' parameter.txt) if then sql_f_name=`grep "\.sql" parameter.txt` echo $sql_f_name fi (2 Replies)
Discussion started by: Dip
2 Replies

5. Shell Programming and Scripting

Reading the Properties File From Shell script

Hi, I am new to the shell script please I need help for following question. I have properties file name called "com.test.properties" I have No of key values in this properties. com.person.name = xyz com.person.age = 55 com.person.address = hello I want read this properties but i... (1 Reply)
Discussion started by: venukjs
1 Replies

6. Shell Programming and Scripting

Reading a property file through shell script???

Hi! i need a script that can read a property file. i.e., A script to read a "property" from property file. Read the property value and based on value of property, decide whether to start the some dataload activity or not. Its urngent. Can anyone help me out???:( (7 Replies)
Discussion started by: sukhdip
7 Replies

7. Shell Programming and Scripting

Error while reading from a file in shell script

Hi All, I'm writing a script to read a file line by line and then perform awk function on it. I am getting an error . My file has one name in it "James". I'm expecting my o/p to be youareJamesbond James ./users.sh: line 7: =: command not found #script to read file line by line #adding... (5 Replies)
Discussion started by: Irishboy24
5 Replies

8. Shell Programming and Scripting

Reading a csv file using shell script

Hello All, I have a csv file that looks like below ProdId_A,3.3.3,some text,some/text,sometext_1.2.3 ProdId_B,3.3.3,some text,some/text,sometext_1.2.3 ProdId_C,3.3.3,some text,some/text,sometext_1.2.3 ProdId_A,6.6.6,some text,some/text,sometext_9.9.9 I will get ProdId from... (5 Replies)
Discussion started by: anand.shah
5 Replies

9. UNIX for Dummies Questions & Answers

C-Shell script help reading from txt file

I need to write a C-Shell script with these properties: It should accept two arguments on the command line. The first argument is the name of a file which contains a list of names, and the second argument is the name of a directory. For each file in the directory, the script should print the... (1 Reply)
Discussion started by: cerce
1 Replies

10. Shell Programming and Scripting

Slow Running Script (Reading 8000 lines)

Slow runnin script. The problem seems to be the sed calls. In summary the script reads list of users in file1. For each username search two files (file 1 & file2) for the username and get the value in the next line after "=". Compare these values with each other. If the same then output... (9 Replies)
Discussion started by: u20sr
9 Replies

LEARN ABOUT CENTOS

grub-render-label

GRUB-RENDER-LABEL(1)                                               User Commands                                              GRUB-RENDER-LABEL(1)

NAME

       grub-render-label - generate a .disk_label for Apple Macs.

SYNOPSIS

       grub-render-label [OPTION...] [OPTIONS]

DESCRIPTION

       Render Apple .disk_label.

       -b, --bgcolor=COLOR
              use COLOR for background

       -c, --color=COLOR
              use COLOR for text

       -f, --font=FILE
              use FILE as font (PF2).

       -i, --input=FILE
              read text from FILE.

       -o, --output=FILE
              set output filename. Default is STDOUT

       -t, --text=STRING
              set the label to render

       -v, --verbose
              print verbose messages.

       -?, --help
              give this help list

       --usage
              give a short usage message

       -V, --version
              print program version

       Mandatory or optional arguments to long options are also mandatory or optional for any corresponding short options.

REPORTING BUGS

       Report bugs to <bug-grub@gnu.org>.

SEE ALSO

       The  full  documentation  for grub-render-label is maintained as a Texinfo manual.  If the info and grub-render-label programs are properly
       installed at your site, the command

              info grub-render-label

       should give you access to the complete manual.

grub-render-label (GRUB) 2.02-2ubuntu8.3                             July 2018                                                GRUB-RENDER-LABEL(1)
All times are GMT -4. The time now is 02:19 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy