Help to edit a large file

jxh461 · #1 (**permalink**) 05-01-2003

I am trying to edit a file that has 33k+ records. In this file I need to edit each record that has a 'Y' in the 107th position and change the 10 fields before the 'Y' to blanks. Not all records have a 'Y' in the 107th field.

ex:

.....................................2222222222Y..................

to

..................................... Y..................

Thanks in advance

oombera · #2 (**permalink**) 05-01-2003

What kind of file is this?

Code:

sed 's/..........Y/          Y/g' someFile > TMP_00
mv TMP_00 someFile

will do it except that it doesn't check for the 107th position.. could there be a Y at other places on each line too, or just in the 107th position?

oombera · #3 (**permalink**) 05-01-2003

I figured out a solution..

Code:

while read LINE
do
  theTest=`echo $LINE | awk '{print substr ($0, 107, 1)}'`
  if [[ $theTest == "Y" ]]; then
    echo $LINE | awk '{print substr ($0, 1, 96) "          Y" substr ($0, 108, length ($0) - 107)}' >> newFile
  else
    echo $LINE >> newFile
  fi
done < someFile

mv newFile someFile

criglerj · #4 (**permalink**) 05-02-2003

Perl's extensions to the RE vocabulary are helpful here ([b]WARNING: untested code[/b):

Code:

perl -pe '/^(.{96}).{10}(Y.*)/ && do { $_ = $1 . " " x 10 . $2 }'

This takes adavntage of the -p option, which prints $_ after processing each record.

Or you can combine oombera's separate invocations of awk into one:

Code:

awk 'substr($0,107,1) == "Y" { $0 = substr($0,1,96) "          " substr($0,107,length($0) - 107) }
{print}' someFile > newFile
mv newFile someFile

Awk is still record length limited AFIAK, so if your records are very long (> 8K ?), you may not be able to read entire lines (I was bitten by this earlier this week).

criglerj · #5 (**permalink**) 05-02-2003

In oombera's solution, each place you see $LINE (lines 3, 5, 7) you should probably change it to "$LINE" since otherwise, consecutive blanks in your records will change the records in an unwanted way.

oombera · #6 (**permalink**) 05-02-2003

criglerj, that's cool - i can't believe the code can be that shortened! And I should've remembered the quotes.. they protect spaces, but I suppose potentially other special characters too..

Just a couple typos..

Code:

"          " should be "          Y"
substr($0,107,len... should be substr($0,108,len...

criglerj · #7 (**permalink**) 05-08-2003

I think I got it right the first time. When the OP wrote "edit each record that has a 'Y' in the 107th position", I guessed the positions started from one, which is the way awk does it. But I stand by my blank string: You want to leave the Y where it is, i.e., you keep characters 1..96, change 97..106 to blanks, then keep 107..end. Char 107 is the "Y" which is left intact. But if the OP's count starts from zero, then all the positions get adjusted by one.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Split large file and add header and footer to each file	ashish4422	Shell Programming and Scripting	1	04-15-2008 06:12 AM
Edit a large file in place	mvijayv	Shell Programming and Scripting	6	09-11-2007 08:31 PM
how to edit large file in unix	balireddy_77	Shell Programming and Scripting	3	12-14-2006 07:40 AM
Strange difference in file size when copying LARGE file..	0ktalmagik	Filesystems, Disks and Memory	1	06-03-2006 07:34 PM
how to edit large files using vi	nazri	UNIX for Dummies Questions & Answers	3	06-15-2001 09:18 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread: