Unfortunately, any solution is going to use a temp file, either explicitly or behind the scenes.
You might as well go with:
What I typically like to do though in situations like these is create an associated
file_name.info file which contains all the header and other useful information like
create-dt, created-by-user, number-of-records, number-of-bytes and check-sums.
This also works for using ftp when a polling program is trying to figure out when a
large file is finished being created so it can snatch it up right away. Instead of
polling for the size to stop growing, it just waits til the info file is created.
In fact one of the other methods i would think works is something like this
1) Create a file that has vi commands
CommandsOfVi.script
2) Open a file using vi and non-interactive mode by using the above commands file as input
I had done something a few years back, but don't remember the syntax etc. so just put it in a very simplified manner here - this approach, I believe, wont use a temp file
Last edited by rbatte1; 12-13-2016 at 09:50 AM..
Reason: Added CODE tags
I have 2 large file (.dat) around 70 g, 12 columns but the data not sorted in both the files.. need your inputs in giving the best optimized method/command to achieve this and redirect the not macthing lines to the thrid file ( diff.dat)
File 1 - 15 columns
File 2 - 15 columns
Data is... (9 Replies)
Optimization shell/awk script to aggregate (sum) for all the columns of Huge data file
File delimiter "|"
Need to have Sum of all columns, with column number : aggregation (summation) for each column
File not having the header
Like below -
Column 1 "Total
Column 2 : "Total
...
...... (2 Replies)
I'm trying to take mrt output and put it at the top of a file along with the date and time. I was able to do it at the bottom of the file with the following
printf "********** $(date) **********\n\n" >> $OUTPUT_PATH/$HOSTNAME
mtr -r -w -c 10 $HOSTADDRESS >> $OUTPUT_PATH/$HOSTNAME
printf... (2 Replies)
Hi, Great minds, I have some files, in fact header files, of CTD profiler, I tried a lot C programming, could not get output as I was expected, because my programming skills are very poor, finally, joined unix forum with the hope that, I may get what I want, from you people,
Here I have attached... (17 Replies)
I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;”
Here is the sample of 5 lines in the file:
Name1;phone1;address1;city1;state1;zipcode1
Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
hi all,
In an mp3 file , data is arranged in sequence of header and data ,how to retrieve data between two headers. Is the data between two headers fixed? because as per theory it says 1152 samples will be there , but dont knw how many bits one sample correspond to?
it would help if any c... (2 Replies)
Hi,
I have a huge file of bibliographic records in some standard format.I need a script to do some repeatable task as follows:
1. Needs to create folders as the strings starts with "item_*" from the input file
2. Create a file "contents" in each folders having "license.txt(tab... (5 Replies)
folks,
In my working directory, there a multiple large files which only contain one line in the file. The line is too long to use "grep", so any help?
For example, if I want to find if these files contain a string like "93849", what command I should use?
Also, there is oder_id number... (1 Reply)
Hello,
I need to insert a line (like a header) as the first line of a very huge file (about 3 ml rows). I am able to do it with sed, but redirecting the output and creating a new file takes quite some time. I was wondering if there was a more efficient way of doing it?
Any help would be... (3 Replies)