Inserting Header to another file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Inserting Header to another file
# 8  
Old 11-13-2014
Quote:
Originally Posted by gvkumar25
Instead what I taught is adding a single line to 20GB data is much faster.
Files do not work that way, you can't insert in the middle. That's why we have text editors to do it for us.

If whatever's using this huge file can read from standard input, you can skip editing the file, just add it to the stream -- print it first, then the rest of the file.

Code:
(       echo "header line"
        exec cat hugefile ) | programwhichuseshugefile


Last edited by Corona688; 11-13-2014 at 02:53 PM..
This User Gave Thanks to Corona688 For This Post:
# 9  
Old 11-13-2014
Quote:
Originally Posted by shamrock
AFAIK ed and ex would use an internal buffer instead of a temporary file...
This is implementation dependent.

Code:
$ strace ed file.txt 2>trace
q
$ grep "/tmp/" strace
open("/tmp/tmpfpHh7ZE", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
unlink("/tmp/tmpfpHh7ZE")               = 0
$

Personally I would expect it to at least sometimes use a temp file -- it's effectively memory if the system is unloaded, and it's either that or crash on large files...
This User Gave Thanks to Corona688 For This Post:
# 10  
Old 11-13-2014
Quote:
Originally Posted by Corona688
This is implementation dependent.

Code:
$ strace ed file.txt 2>trace
q
$ grep "/tmp/" strace
open("/tmp/tmpfpHh7ZE", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
unlink("/tmp/tmpfpHh7ZE")               = 0
$

Personally I would expect it to at least sometimes use a temp file -- it's effectively memory if the system is unloaded, and it's either that or crash on large files...
Yes that might be implementation dependent depending on the file size...big files might need an on disk scratchpad whereas small files could be loaded into one of the internal buffers...
# 11  
Old 11-13-2014
Quote:
Originally Posted by shamrock
How about our good ole ex utility...
Code:
ex -sc '0r file1 | x' file2

---------- Post updated at 12:01 PM ---------- Previous update was at 11:57 AM ----------


AFAIK ed and ex would use an internal buffer instead of a temporary file...
In some implementations they use a temp file, in others they use memory if the file will fit in memory and a temp file if it won't, and in others they use memory and die on an ENOSPC error if the file won't fit into memory.

Even in cases where ed and ex edit a file entirely in memory, the entire 20GB file being discussed in this thread has to be read into memory and the updated file has to be written back to the file. There just plain is absolutely no way to add text to the start of a file without reading the entire file at least once and writing the entire contents of the updated file at least once with current BSD, UNIX, Linux, or Windows filesystems.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Find header in a text file and prepend it to all lines until another header is found

I've been struggling with this one for quite a while and cannot seem to find a solution for this find/replace scenario. Perhaps I'm getting rusty. I have a file that contains a number of metrics (exactly 3 fields per line) from a few appliances that are collected in parallel. To identify the... (3 Replies)
Discussion started by: verdepollo
3 Replies

2. Shell Programming and Scripting

Inserting a header with special character

Hi, I am trying to insert header row with a special character delimiter with Unicode u0109 into a file with ‘echo’, header looks like below echo –e “header1\u0109header\u0109header3\u0109header4” It just inserting as it is in the quotes but not the special character, Please suggest if am... (2 Replies)
Discussion started by: oom
2 Replies

3. UNIX for Beginners Questions & Answers

Inserting Header at different position in a file

I would like to hear your directions on how to Insert theses tag </TITLE> and <TEXT> at a given position in 1000 of text files. My Files look like as samplefile1.txt <DOC> <DOCNO>3_September_2012</DOCNO> <TITLE> ... ... ... .... ... .. .. .. ... .. .. .... </TITLE> <TEXT> .... (1 Reply)
Discussion started by: imranrasheedamu
1 Replies

4. Shell Programming and Scripting

Inserting header after every nth line

Hi Experts, I have a file which contain hundreds of records/lines. I want to insert the below header in the file after every 60 lines. #Header FirstName LastName Address --------- ---------- --------- Let say I saved the... (6 Replies)
Discussion started by: brichigo
6 Replies

5. UNIX for Dummies Questions & Answers

Merge all csv files in one folder considering only 1 header row and ignoring header of all others

Friends, I need help with the following in UNIX. Merge all csv files in one folder considering only 1 header row and ignoring header of all other files. FYI - All files are in same format and contains same headers. Thank you (4 Replies)
Discussion started by: Shiny_Roy
4 Replies

6. Shell Programming and Scripting

Comparing one file header with another file header

Hi Experts, In our project we have requirement where in we have to compare header of one file with header in the parameter file. There are 20 files which we ftp from one site. All this files have different header. We are comapring this file with our parameter file(which is having the header... (2 Replies)
Discussion started by: Amey Joshi
2 Replies

7. Ubuntu

Inserting a header with column number to a 1.6 GB file with special spacing

Hi; I've been searching posts to find a solution to what I'm trying to do, but I've have NOT found anything yet. I have a file (file1) with 300K columns and 1411 rows, the columns don't have a column no. header (No header at all) and I'm trying to fetch the information from specific columns.... (3 Replies)
Discussion started by: sogi
3 Replies

8. Shell Programming and Scripting

Inserting Header and footer

Hi All, I have several txt files i need to enter specific header and footer (both are separate) to all these files how can i do this? plz help.. Regards, Raghav (4 Replies)
Discussion started by: digitalrg
4 Replies

9. Shell Programming and Scripting

Inserting a String in a file header.

Dear all, I have a file created in the name sample.txt in UNIX with header and footer. How to insert a required string (for example "FILE1") in the header part after the file has been created. What kind of command can i use to do the same. Thanks in advance Hari (3 Replies)
Discussion started by: Hari123
3 Replies

10. Linux

Reading the header of a tar file(posix header)

say i have these many file in a directory named exam. 1)/exam/newfolder/link.txt. 2)/exam/newfolder1/ and i create a tar say exam.tar well the problem is, when i read the tar file i dont find any metadata about the directories,as you cannot create a tar containig empty directories. on the... (2 Replies)
Discussion started by: Tanvirk
2 Replies
Login or Register to Ask a Question