insert a header in a huge data file without using an intermediate file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting insert a header in a huge data file without using an intermediate file
# 1  
Old 02-10-2009
Question insert a header in a huge data file without using an intermediate file

I have a file with data extracted, and need to insert a header with a constant string, say: H|PayerDataExtract

if i use sed, i have to redirect the output to a seperate file like

Code:
sed ' sed commands' ExtractDataFile.dat > ExtractDataFileWithHeader.dat

the same is true for awk

and in its simplist form i could say
Code:
echo 'H|PayerDataExtract' > ExtractDataFileWithHeader.dat
 cat ExtractDataFile.dat >> ExtractDataFileWithHeader.dat
 mv ExtractDataFileWithHeader.dat ExtractDataFile.dat

but in all of the above an extra file is created. If i were to do this in vi manually the extra file could be avoided.

Is there a way to avoid the extra file while still not having to manually use vi in a interactive manner? This is even more neccessary if the file uses over 50% filespace and an extra file will just double my usage , though temporarily

Last edited by rbatte1; 12-13-2016 at 09:46 AM.. Reason: Added CODE tags
# 2  
Old 02-10-2009
This is crazy, I agree.

Typically, in situations like this, I employ an "original_file_name.info" file instead.

In the .info file I'll put all the information necessary for another program/user
to verify the contents of the huge file i created.

There are other options . . . . like creating a dummy header record in
the file creation program . . . and then using fseek() to hop back to
the beginning and overwrite the header info....

but you may not have this much control over how this file is created.
# 3  
Old 02-10-2009
There are 'edit in place' features for GNU sed and for perl (perl -pie ' perl script here').
However they do use tmp files behind the scenes
# 4  
Old 02-10-2009
You can even use the sed 'insert' command along with -i (inline) option to place the the "header" at a specific line number (like line 1).

Something like this should work:
Code:
sed -i '1 i \Some Header Text Here' ExtractDataFile.dat

# 5  
Old 02-11-2009
Why not create an ed script instead? And then use 'patch'? But that would still require a temp file in the background though you wouldn't notice.
# 6  
Old 02-18-2009
Data

hi ddreggors/jim

i tried this - and it fails with the below error

Code:
sed -i '1 i \HeaderText' ExtractDataFile.txt
sed: illegal option -- i

The system is a SunOS ussun1l 5.8 Generic_117350-60 sun4u sparc SUNW,Sun-Fire-15000

the perl option is out for me as the client may not want to install any piece of software without business cause, however as long as the tmp file is created in the /tmp space and not my file-dir i think i'm ok - any more ideas anyone?

Last edited by rbatte1; 12-13-2016 at 09:49 AM.. Reason: Added code tags & ICODE tags for clarity
# 7  
Old 02-18-2009
try this:
Code:
sed 1'i\HeaderText' ExtractDataFile.txt

If this outputs to the screen as expected (this does not update the file) then the inner 'i' (insert command) is working fine.

Next try:

Code:
sed -i 1'i\HeaderText' ExtractDataFile.txt

If either gives an error then you may have to upgrade sed to get the use of the 'inline' flag (-i) or the insert command ('i\text') to be able to do this without writing to another file first and specifying a line number.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

File comaprsons for the Huge data files ( around 60G) - Need optimized and teh best way to do this

I have 2 large file (.dat) around 70 g, 12 columns but the data not sorted in both the files.. need your inputs in giving the best optimized method/command to achieve this and redirect the not macthing lines to the thrid file ( diff.dat) File 1 - 15 columns File 2 - 15 columns Data is... (9 Replies)
Discussion started by: kartikirans
9 Replies

2. UNIX for Advanced & Expert Users

Need Optimization shell/awk script to aggreagte (sum) for all the columns of Huge data file

Optimization shell/awk script to aggregate (sum) for all the columns of Huge data file File delimiter "|" Need to have Sum of all columns, with column number : aggregation (summation) for each column File not having the header Like below - Column 1 "Total Column 2 : "Total ... ...... (2 Replies)
Discussion started by: kartikirans
2 Replies

3. Shell Programming and Scripting

Insert date/time header at top of file

I'm trying to take mrt output and put it at the top of a file along with the date and time. I was able to do it at the bottom of the file with the following printf "********** $(date) **********\n\n" >> $OUTPUT_PATH/$HOSTNAME mtr -r -w -c 10 $HOSTADDRESS >> $OUTPUT_PATH/$HOSTNAME printf... (2 Replies)
Discussion started by: kramer65
2 Replies

4. Shell Programming and Scripting

Extract header data from one file and combine it with data from another file

Hi, Great minds, I have some files, in fact header files, of CTD profiler, I tried a lot C programming, could not get output as I was expected, because my programming skills are very poor, finally, joined unix forum with the hope that, I may get what I want, from you people, Here I have attached... (17 Replies)
Discussion started by: nex_asp
17 Replies

5. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

6. Shell Programming and Scripting

Three Difference File Huge Data Comparison Problem.

I got three different file: Part of File 1 ARTPHDFGAA . . Part of File 2 ARTGHHYESA . . Part of File 3 ARTPOLYWEA . . (4 Replies)
Discussion started by: patrick87
4 Replies

7. Programming

to find header in Mp3 file and retrieve data

hi all, In an mp3 file , data is arranged in sequence of header and data ,how to retrieve data between two headers. Is the data between two headers fixed? because as per theory it says 1152 samples will be there , but dont knw how many bits one sample correspond to? it would help if any c... (2 Replies)
Discussion started by: shashi
2 Replies

8. Shell Programming and Scripting

How to extract data from a huge file?

Hi, I have a huge file of bibliographic records in some standard format.I need a script to do some repeatable task as follows: 1. Needs to create folders as the strings starts with "item_*" from the input file 2. Create a file "contents" in each folders having "license.txt(tab... (5 Replies)
Discussion started by: srsahu75
5 Replies

9. UNIX for Dummies Questions & Answers

search and grab data from a huge file

folks, In my working directory, there a multiple large files which only contain one line in the file. The line is too long to use "grep", so any help? For example, if I want to find if these files contain a string like "93849", what command I should use? Also, there is oder_id number... (1 Reply)
Discussion started by: ting123
1 Replies

10. UNIX for Advanced & Expert Users

Insert a line as the first line into a very huge file

Hello, I need to insert a line (like a header) as the first line of a very huge file (about 3 ml rows). I am able to do it with sed, but redirecting the output and creating a new file takes quite some time. I was wondering if there was a more efficient way of doing it? Any help would be... (3 Replies)
Discussion started by: shriek
3 Replies
Login or Register to Ask a Question