Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Edit a large file in place
Top Forums Shell Programming and Scripting Edit a large file in place Post 302134733 by fazliturk on Monday 3rd of September 2007 08:24:03 AM
Old 09-03-2007
I tried the following code in aix,in ksh
code is long but there is no while etc.
let say your original file origfile

step 1.

sed s/"|"/" "/g origfile >tempfile

/** if you dont have sed ,you must change "|" with blank with someting */
/after this your original file looks like this 12123 12345678 */

grep -n "^$" tempfile >origfile,rm tempfile

/*after this your original file looks like this ;
1 12123 12345678
2 42142 53988989

step 2.
/**produce 1 million random numbers and save to the RandNumbersFile**/

step 3.
/**produce 1 million random numbers and save to the RandRecordsFile**/
sort -u RandRecordsFile>tempfile
mv tempfile RandRecordsFile

/*you can produce 1 million numbers but if you sort it unically it can be less than 1 million. you must be sure that every line in this file is unique, the above command arranges this*/

let "NeededLine=1000000-`wc -l RandRecordsFile |awk '{print $1}'`"

/*this line shows you how many new records do you need after sort */

counter=0
while [ $counter -lt $NeededLine ]
do
/**produce random RandomRecord(means random number).I mean you must add your code here **/
grep $RandomRecord RandomRecordsFile >/dev/null
if [ $? -ne 0 ]
then
echo $RandomRecord >>RandomRecordsFile
let "counter=$counter+1"
fi
done
sort -u RandomRecordsFile>tempfile
paste tempfile RandNumbersFile >RandomRecordsFile
rm tempfile

/** after this your RandomRecordsFile looks like this ;
1 12345678
27 53988989
first one stands for record num and the second rundom field (orig second field) **/

join -v1 origfile RandomRecordsFile >tempfile /** unmatched lines **/
join -o 1.1,1.2,2.2 origfile RandomRecordsFile >>tempfile /*matched lines */
sort -u tempfile >origfile /*sort on field1 */
/**if you need add these lines
cut -f2,f3 origfile >tempfile
sed s/" "/"|"/g tempfile>origfile **/
rm tempfile

so the code is;
/**produce 1 million random numbers and save to the RandNumbersFile**/
/**produce 1 million random numbers and save to the RandRecordsFile**/

cp yourfile origfile
sed s/"|"/" "/g origfile >tempfile
grep -n "^$" tempfile >origfile
sort -u RandRecordsFile>tempfile
mv tempfile RandRecordsFile
let "NeededLine=1000000-`wc -l RandRecordsFile |awk '{print $1}'`"
while [ $counter -lt $NeededLine ]
do
/**produce random RandomRecord(means random number).I mean you must add your code here **/
grep $RandomRecord RandomRecordsFile >/dev/null
if [ $? -ne 0 ]
then
echo $RandomRecord >>RandomRecordsFile
let "counter=$counter+1"
fi
done
sort -u RandomRecordsFile>tempfile
paste tempfile RandNumbersFile >RandomRecordsFile
join -v1 origfile RandomRecordsFile >tempfile
join -o 1.1,1.2,2.2 origfile RandomRecordsFile
sort -u tempfile >origfile
rm tempfile
Last edited by fazliturk; 09-03-2007 at 09:41 AM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

how to edit large files using vi

How to edit large file using vi where you can't increase /usr/var/tmp anymore? (3 Replies)
Discussion started by: nazri
3 Replies

2. Shell Programming and Scripting

Help to edit a large file

I am trying to edit a file that has 33k+ records. In this file I need to edit each record that has a 'Y' in the 107th position and change the 10 fields before the 'Y' to blanks. Not all records have a 'Y' in the 107th field. ex: ... (8 Replies)
Discussion started by: jxh461
8 Replies

3. Shell Programming and Scripting

how to edit large file in unix

hi All, Plz let me know how to edit a file with 2000000 records. each record contains with 40 field seperated by |. i want modify 455487 record, but i am uable to edit this large file using vi editor in unix. plz let me know how to modify this file. Thanks in advance. -Bali Reddy (3 Replies)
Discussion started by: balireddy_77
3 Replies

4. Shell Programming and Scripting

Scripting the process to edit a large file

Hi, I need to make a script to edit a file. File is a large file in below format Version: 2008120101 ;$INCLUDE ./abc/xyz/Delhi ;$INCLUDE ./abc/xyz/London $INCLUDE ./abc/xyz/New York First line in the file is version number which is in year,month,date and serial number format. Each... (5 Replies)
Discussion started by: makkar4u
5 Replies

5. Shell Programming and Scripting

sed edit in place -i issues

Hello, I am attempting to create a command that I can eventually put into a loop so I can edit 1file on many servers. I would like to edit the file in place with sed -i. If not I will take any suggestions on how to use a temp file. I need to remove a email address from the configuration file... (4 Replies)
Discussion started by: abacus
4 Replies

6. Solaris

What is the best way to copy data from place to another place?

Dear Gurus, I need you to advice or suggestion about the best solution to copy data around 200-300G from serverA(location A) to serverB(location B). Normally, I will share folder and then copy but it takes too long time(about 2 days). Do you have any suggestion or which way should be... (9 Replies)
Discussion started by: unitipon
9 Replies

7. Shell Programming and Scripting

Read from file specific place in file using inode

Hello, I am using tcsh on AIX. I would like to write a script that does the following: 1. given an inode, how do I find exactly the name of the file? I know I could do this using ls -i | grep <inode> but it returns: <inode> <filename>. I need some string manipulation or something to... (1 Reply)
Discussion started by: lastZenMaster
1 Replies

8. Shell Programming and Scripting

How to get awk to edit in place and join all lines in text file

Hi, I lack the utter fundamentals on how to craft an awk script. I have hundreds of text files that were mangled by .doc format so all the lines are broken up so I need to join all of the lines of text into a single line. Normally I use vim command "ggVGJ" to join all lines but with so many... (3 Replies)
Discussion started by: n00ti
3 Replies

9. Shell Programming and Scripting

How to edit a large file

Whenever I am trying to edit a file in unix with vi editor, I am getting the following error: <data> :Tmp file too large Is there any way that I can edit the file other than vi. Any help is really appreciated. Thanks (10 Replies)
Discussion started by: bobby1015
10 Replies

10. Shell Programming and Scripting

Search for a file in all directories and place the file in that directory

Hi All, Daily I am getting the updated file. I have to search for this file in all directories and sub directories. If the file existed in a particular directory then move this updated file to that particular directory. If the file is not existed in any of the directories then place this... (4 Replies)
Discussion started by: ROCK_PLSQL
4 Replies

LEARN ABOUT SUNOS

sag

sag(1)								   User Commands							    sag(1)

NAME

       sag - system activity graph

SYNOPSIS

       sag [-e time] [-f file] [-i sec] [-s time] [-T term] [-x spec] [-y spec]

DESCRIPTION

       The  sag  utility  graphically displays the system activity data stored in a binary data file by a previous sar(1) run. Any of the sar data
       items may be plotted singly or in combination, as cross plots or versus time. Simple arithmetic combinations of data may be specified.  sag
       invokes	 sar  and  finds  the  desired data by string-matching the data column header (run  sar to see what is available). The sag utility
       requires a graphic terminal to draw the graph, and uses tplot(1) to produce its output. When running Solaris 2.x and  OpenWindows,  perform
       the following steps:

       1.  Run an "xterm" as a Tektronics terminal: prompt# xterm -t

       2.  In the "xterm" window, run  sag specifying a  tek terminal: prompt# sag -T tek options

OPTIONS

       The following options are supported and passed through to  sar (see sar(1)):

       -e time	Select data up to time. Default is 18:00.

       -f file	Use file as the data source for sar. Default is the current daily data file /usr/adm/sa/sadd.

       -i sec	Select data at intervals as close as possible to sec seconds.

       -s time	Select data later than time in the form hh[:mm]. Default is 08:00.

       -T term	Produce output suitable for terminal term. See	tplot(1) for known terminals. Default for term is $TERM.

       -x spec	x axis specification with spec in the form:

		name[op name]...[lo hi]

		name  is  either a string that will match a column header in the sar  report, with an optional device name in square brackets, for
		example, r+w/s[dsk-1], or an integer value. op	is  + - * or /	surrounded by blank spaces. Up to five	names  may  be	specified.
		Parentheses  are  not  recognized.  Contrary  to  custom, + and - have precedence over * and /. Evaluation is left to right. Thus,
		A/A+B*100 is evaluated as (A/(A+B))*100, and A+B/C+D is (A+B)/(C+D). lo and hi are optional numeric scale limits. If  unspecified,
		they are deduced from the data.

		Enclose  spec in double-quotes ("") if it includes white space.

		A single spec is permitted for the x axis. If unspecified, time is used.

       -y spec	y  axis  specification with spec in the same form as for -x. Up to 5 spec arguments separated by a semi-colon (;) may be given for
		-y. The -y default is:

		-y"%usr0100;%usr+%sys0100;%usr+%sys+%wio0100"

EXAMPLES

       Example 1: Examples of the sag command.

       To see today's CPU utilization:

       example$ sag

       To see activity over 15 minutes of all disk drives:

       example$ TS=`date +%H:%M`
       example$ sar -o /tmp/tempfile 60 15
       example$ TE=`date +%H:%M`
       example$ sag -f /tmp/tempfile -s $TS -e $TE -y "r+w/s[dsk]"

FILES

       /usr/adm/sa/sadd        daily data file for day dd

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Availability		     |SUNWaccu			   |
       +-----------------------------+-----------------------------+

SEE ALSO

       sar(1), tplot(1), attributes(5)

SunOS 5.10							    4 Mar 1998								    sag(1)
All times are GMT -4. The time now is 08:24 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy