Sponsored Content
Full Discussion: While loop read line Issue
Top Forums Shell Programming and Scripting While loop read line Issue Post 302415431 by saurabhkumar198 on Thursday 22nd of April 2010 02:59:16 PM
Old 04-22-2010
Error While loop read line Issue

Hi

I am using while loop, below, to read lines from a very large file, around 400,000 rows. The script works fine until around line 300k but then starts giving incorrect result.

I have tried running the script with a smaller data set and it works fine. I made sure to include the line where the problem starts in the data set to make sure there are no special characters that are causing the issue.

Please let me know if this is a script issue or some size limit issue.

Code:
while read line
do
count=0
echo $line | cut -c250-310  | cut -d"_" -f3 | cut -c1-9 | sed 's/ //g' | read tran_id
echo $line | awk '{ n += gsub( "<TCRMContractPartyRoleBObj>", "") } END { print n }' | sed 's/ //g' | read count
count=$n
echo $tran_id"|"$count >> $XML_OUTPUT
done < $XML_INPUT_FILE

Output - Problem starts at line 386501 below and continues till end of file
Code:
+386498 TK1251683|1
+386499 TK1251684|2
+386500 TK1251684|4
+386501 nNameBObj|1
+386502 hodBObj><|1
+386503 e><XSpoke|1
+386504 tractType|1
+386505 ourceIden|1
+386506 ountryTyp|1

Thanks
Saurabh

Last edited by Scott; 04-22-2010 at 04:10 PM.. Reason: Please use code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Nested while read line loop

Hi, Can anyone please help me: i'm trying to read a file with directory-names , then go to that directory and read another (output) file to perform some tasks per line (second read line in the part of script below). The problem is that after the nested while loop has finished, the first while... (7 Replies)
Discussion started by: Rakker
7 Replies

2. Shell Programming and Scripting

read line in a for loop

Hi All, How can we use read line using the index value of a FOR loop? eg: pt_mstr,pt_status,8 pt_mstr,pt_buyer,8 pt_mstr,pt_sfty_stk,8 pt_mstr,pt_ord_pol,3 pt_mstr,pt_capacity,8 pt_mstr,pt_plan_ord,3 pt_mstr,pt_ord_mult,8 From this file i want to read the line2, 3 and 4 only using a FOR... (3 Replies)
Discussion started by: balajim
3 Replies

3. Shell Programming and Scripting

While read line loop

Hi I'm writing a bash script which will read an input file and look for occurrences of the current user ($USER) executing the script. When i find the occurrence of the username I take that line and append it to a file with a line number and bracket display next to line. The input file has been... (12 Replies)
Discussion started by: BundBash
12 Replies

4. Shell Programming and Scripting

While loop read line not working

Hi, I am trying to read a file line by line inside of a while loop. This while loop is part of a here document. while read line do ssh -t $2@$remotehost <<REMOTE ls path/to/dir > $path_to_dir while read line1 do echo "LINE --- $line" done... (4 Replies)
Discussion started by: mnanavati
4 Replies

5. Shell Programming and Scripting

while read LINE issue

Hi, This is the script and the error I am receiving Can anyone please suggest ? For the exmaple below assume we are using vg01 #!/bin/ksh echo "##### Max Mount Count Fixer #####" echo "Please insert Volume Group name to check" read VG lvs |grep $VG | awk {'print $1'} > /tmp/audit.log ... (2 Replies)
Discussion started by: galuzan
2 Replies

6. Shell Programming and Scripting

IF awk in a while read line-loop

Hi As a newbe in scripting, i struggle hard with my first script. What i want to do is, bringing data of two files together. file1: .... 05/14/12-04:00:00 41253 4259 5135 5604 5812 5372 05/14/12-04:10:00 53408 5501 6592 7402 7354 6639 05/14/12-04:20:00 58748 6037 7292 8223... (13 Replies)
Discussion started by: IMPe
13 Replies

7. Shell Programming and Scripting

[BASH] read 'line' issue with leading tabs and virtual line breaks

Heyas I'm trying to read/display a file its content and put borders around it (tui-cat / tui-cat -t(ypwriter). The typewriter-part is a 'bonus' but still has its own flaws, but thats for later. So in some way, i'm trying to rewrite cat using bash and other commands. But sadly it fails on... (2 Replies)
Discussion started by: sea
2 Replies

8. Shell Programming and Scripting

While read line loop

Based on text file: PATH:/media/hdd/Media/Video/title1 FILE:/media/cache/281662-14.jpg PATH:/media/hdd/Media/Video/title2 FILE:/media/cache/281662-15.jpg PATH:/media/hdd/Media/Video/title3 FILE:/media/cache/281662-16.jpg PATH:/media/hdd/Media/Video/title4 FILE:/media/cache/281662-17.jpg... (12 Replies)
Discussion started by: TiedCone
12 Replies

9. UNIX for Beginners Questions & Answers

Performance issue to read line by line

Hi All- we have performance issue in unix to read line by line. I am looking at processing all the records. description: Our script will read data from a flat file, it will pickup first four character and based on the value it will set up variables accordingly and appended the final output to... (11 Replies)
Discussion started by: balu1729
11 Replies

10. Shell Programming and Scripting

Performance issue - to read line by line

All- We have a performance issue in reading a file line by line. Please find attached scripts for the same. Currently it is taking some 45 min to parse "512444" lines. Could you please have a look at it and provide any suggestions to improve the performance. Thanks, Balu ... (12 Replies)
Discussion started by: balu1729
12 Replies
PHP(1)								Scripting Language							    PHP(1)

NAME
php - PHP Command Line Interface 'CLI' SYNOPSIS
php [options] [ -f ] file [[--] args...] php [options] -r code [[--] args...] php [options] [-B code] -R code [-E code] [[--] args...] php [options] [-B code] -F file [-E code] [[--] args...] php [options] -- [ args...] php [options] -a DESCRIPTION
PHP is a widely-used general-purpose scripting language that is especially suited for Web development and can be embedded into HTML. This is the command line interface that enables you to do the following: You can parse and execute files by using parameter -f followed by the name of the file to be executed. Using parameter -r you can directly execute PHP code simply as you would do inside a .php file when using the eval() function. It is also possible to process the standard input line by line using either the parameter -R or -F. In this mode each separate input line causes the code specified by -R or the file specified by -F to be executed. You can access the input line by $argn. While processing the input lines $argi contains the number of the actual line being processed. Further more the parameters -B and -E can be used to execute code (see -r) before and after all input lines have been processed respectively. Notice that the input is read from STDIN and therefore reading from STDIN explicitly changes the next input line or skips input lines. If none of -r -f -B -R -F or -E is present but a single parameter is given then this parameter is taken as the filename to parse and exe- cute (same as with -f). If no parameter is present then the standard input is read and executed. OPTIONS
--interactive -a Run PHP interactively. This lets you enter snippets of PHP code that directly get executed. When readline support is enabled you can edit the lines and also have history support. --bindpath address:port|port -b address:port|port Bind Path for external FASTCGI Server mode (CGI only). --no-chdir -C Do not chdir to the script's directory (CGI only). --no-header -q Quiet-mode. Suppress HTTP header output (CGI only). --timing count -T count Measure execution time of script repeated count times (CGI only). --php-ini path|file -c path|file Look for php.ini file in the directory path or use the specified file --no-php-ini -n No php.ini file will be used --define foo[=bar] -d foo[=bar] Define INI entry foo with value bar -e Generate extended information for debugger/profiler --file file -f file Parse and execute file --global name -g name Make variable name global in script. --help -h This help --hide-args -H Hide script name (file) and parameters (args...) from external tools. For example you may want to use this when a php script is started as a daemon and the command line contains sensitive data such as passwords. --info -i PHP information and configuration --syntax-check -l Syntax check only (lint) --modules -m Show compiled in modules --run code -r code Run PHP code without using script tags '<?..?>' --process-begin code -B code Run PHP code before processing input lines --process-code code -R code Run PHP code for every input line --process-file file -F file Parse and execute file for every input line --process-end code -E code Run PHP code after processing all input lines --syntax-highlight -s Output HTML syntax highlighted source --version -v Version number --stripped -w Output source with stripped comments and whitespace --zend-extension file -z file Load Zend extension file args... Arguments passed to script. Use '--' args when first argument starts with '-' or script is read from stdin --rfunction name --rf name Shows information about function name --rclass name --rc name Shows information about class name --rextension name --re name Shows information about extension name --rzendextension name --rz name Shows information about Zend extension name --rextinfo name --ri name Shows configuration for extension name --ini Show configuration file names FILES
/etc/php5/cli/php.ini The configuration file for the CLI version of PHP. /etc/php5/cgi/php.ini The configuration file for the CGI version of PHP. /etc/php5/apache2/php.ini The configuration file for the version of PHP that apache2 uses. EXAMPLES
php -r 'echo "Hello World ";' This command simply writes the text "Hello World" to standard out. php -r 'print_r(gd_info());' This shows the configuration of your gd extension. You can use this to easily check which image formats you can use. If you have any dynamic modules you may want to use the same ini file that php uses when executed from your webserver. There are more extensions which have such a function. For dba use: php -r 'print_r(dba_handlers(1));' php -R 'echo strip_tags($argn)." ";' This PHP command strips off the HTML tags line by line and outputs the result. To see how it works you can first look at the following PHP command 'php -d html_errors=1 -i' which uses PHP to output HTML formatted configuration information. If you then combine those two 'php ...|php ...' you'll see what happens. php -E 'echo "Lines: $argi ";' Using this PHP command you can count the lines being input. php -R '@$l+=count(file($argn));' -E 'echo "Lines:$l ";' In this example PHP expects each input line being a file. It counts all lines of the files specified by each input line and shows the summarized result. You may combine this with tools like find and change the php scriptlet. php -R 'echo "$argn "; fgets(STDIN);' Since you have access to STDIN from within -B -R -F and -E you can skip certain input lines with your code. But note that in such cases $argi only counts the lines being processed by php itself. Having read this you will guess what the above program does: skipping every second input line. TIPS
You can use a shebang line to automatically invoke php from scripts. Only the CLI version of PHP will ignore such a first line as shown below: #!/bin/php <?php // your script ?> SEE ALSO
For a more or less complete description of PHP look here: http://www.php.net/manual/ BUGS
You can view the list of known bugs or report any new bug you found at: http://bugs.php.net AUTHORS
The PHP Group: Thies C. Arntzen, Stig Bakken, Andi Gutmans, Rasmus Lerdorf, Sam Ruby, Sascha Schumann, Zeev Suraski, Jim Winstead, Andrei Zmievski. Additional work for the CLI sapi was done by Edin Kadribasic, Marcus Boerger and Johannes Schlueter. A List of active developers can be found here: http://www.php.net/credits.php And last but not least PHP was developed with the help of a huge amount of contributors all around the world. VERSION INFORMATION
This manpage describes php, version 5.4.4-14+deb7u14. COPYRIGHT
Copyright (C) 1997-2010 The PHP Group This source file is subject to version 3.01 of the PHP license, that is bundled with this package in the file LICENSE, and is available through the world-wide-web at the following url: http://www.php.net/license/3_01.txt If you did not receive a copy of the PHP license and are unable to obtain it through the world-wide-web, please send a note to license@php.net so we can mail you a copy immediately. The PHP Group 2010 PHP(1)
All times are GMT -4. The time now is 11:36 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy