Sponsored Content
Top Forums Shell Programming and Scripting insert a header in a huge data file without using an intermediate file Post 302286265 by jim mcnamara on Tuesday 10th of February 2009 03:51:42 PM
Old 02-10-2009
There are 'edit in place' features for GNU sed and for perl (perl -pie ' perl script here').
However they do use tmp files behind the scenes
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Insert a line as the first line into a very huge file

Hello, I need to insert a line (like a header) as the first line of a very huge file (about 3 ml rows). I am able to do it with sed, but redirecting the output and creating a new file takes quite some time. I was wondering if there was a more efficient way of doing it? Any help would be... (3 Replies)
Discussion started by: shriek
3 Replies

2. UNIX for Dummies Questions & Answers

search and grab data from a huge file

folks, In my working directory, there a multiple large files which only contain one line in the file. The line is too long to use "grep", so any help? For example, if I want to find if these files contain a string like "93849", what command I should use? Also, there is oder_id number... (1 Reply)
Discussion started by: ting123
1 Replies

3. Shell Programming and Scripting

How to extract data from a huge file?

Hi, I have a huge file of bibliographic records in some standard format.I need a script to do some repeatable task as follows: 1. Needs to create folders as the strings starts with "item_*" from the input file 2. Create a file "contents" in each folders having "license.txt(tab... (5 Replies)
Discussion started by: srsahu75
5 Replies

4. Programming

to find header in Mp3 file and retrieve data

hi all, In an mp3 file , data is arranged in sequence of header and data ,how to retrieve data between two headers. Is the data between two headers fixed? because as per theory it says 1152 samples will be there , but dont knw how many bits one sample correspond to? it would help if any c... (2 Replies)
Discussion started by: shashi
2 Replies

5. Shell Programming and Scripting

Three Difference File Huge Data Comparison Problem.

I got three different file: Part of File 1 ARTPHDFGAA . . Part of File 2 ARTGHHYESA . . Part of File 3 ARTPOLYWEA . . (4 Replies)
Discussion started by: patrick87
4 Replies

6. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

7. Shell Programming and Scripting

Extract header data from one file and combine it with data from another file

Hi, Great minds, I have some files, in fact header files, of CTD profiler, I tried a lot C programming, could not get output as I was expected, because my programming skills are very poor, finally, joined unix forum with the hope that, I may get what I want, from you people, Here I have attached... (17 Replies)
Discussion started by: nex_asp
17 Replies

8. Shell Programming and Scripting

Insert date/time header at top of file

I'm trying to take mrt output and put it at the top of a file along with the date and time. I was able to do it at the bottom of the file with the following printf "********** $(date) **********\n\n" >> $OUTPUT_PATH/$HOSTNAME mtr -r -w -c 10 $HOSTADDRESS >> $OUTPUT_PATH/$HOSTNAME printf... (2 Replies)
Discussion started by: kramer65
2 Replies

9. UNIX for Advanced & Expert Users

Need Optimization shell/awk script to aggreagte (sum) for all the columns of Huge data file

Optimization shell/awk script to aggregate (sum) for all the columns of Huge data file File delimiter "|" Need to have Sum of all columns, with column number : aggregation (summation) for each column File not having the header Like below - Column 1 "Total Column 2 : "Total ... ...... (2 Replies)
Discussion started by: kartikirans
2 Replies

10. UNIX for Advanced & Expert Users

File comaprsons for the Huge data files ( around 60G) - Need optimized and teh best way to do this

I have 2 large file (.dat) around 70 g, 12 columns but the data not sorted in both the files.. need your inputs in giving the best optimized method/command to achieve this and redirect the not macthing lines to the thrid file ( diff.dat) File 1 - 15 columns File 2 - 15 columns Data is... (9 Replies)
Discussion started by: kartikirans
9 Replies
patterns(5int)															    patterns(5int)

Name
       patterns - patterns for use with internationalization tools

Syntax
       See the Description section.

Description
       The patterns file contains the patterns that must be matched for the internationalization tools and

       The pattern file in the following example is the default patterns file located in

       # This is the header to insert at the beginning of the first new
       # source file

       $SRCHEAD1(1)
       #include <nl_types.h>
       nl_catd _m_catd;
       

       # The header to insert at the beginning of the rest of the new
       # source files

       $SRCHEAD2(2)
       #include <nl_types.h>
       extern nl_catd _m_catd;
       

       # This is the header to insert at the beginning of the message
       # catalogues

       $CATHEAD(3)
       $ /*
       $  * X/OPEN message catalogue
       $  */
       
       $quote "

       # This is how patterns that are matched will get rewritten.

       $REWRITE(4)
       catgets(_m_catd, %s, %n, %t)

       # Following is a list of the sort of strings we are looking for.
       # The regular expression syntax is based on regex(3).

       $MATCH(5)

       # Match on strings containing an escaped "
       "[^\]*\"[^"]*"

       # Match on general strings
       "[^"]*"

       # Now reject some special C constructs.

       $REJECT(6)
       # the empty string
       ""0

       # string with just one format descriptor
       "%."
       "%.."

       # string with just line control in
       "\."

       # string with just line control and one format descriptor in
       "%.\."
       "\.%."

       # ignore cpp include lines
       #[  ]*include[	]*".*"
       #[  ]*ident[  ]*".*"

       # reject some common C functions and expressions with quoted
       # strings
       [sS][cC][cC][sS][iI][dD][][  ]*=[  ]*".*"
       open[  ]*([^,]*,[^)]*)
       creat[  ]*([^,]*,[^)]*)
       access[	]*([^,]*,[^)]*)
       chdir[  ]*([^,]*,[^)]*)
       chmod[  ]*([^,]*,[^)]*)
       chown[  ]*([^,]*,[^)]*)

       # Reject any strings in single line comments
       /*.**/

       # Print a warning for initialised strings.

       $ERROR initialised strings cannot be replaced(7)
       char[^=]*=[  ]*"[^"]*"
       char[^=]*=[  ]*"[^\]*\"[^"]*"
       char[ ]***[A-Za-z][A-Za-z0-9]*[[^]*][ ]*=[  {]*"[^"]*"
       char[ ]***[A-Za-z][A-Za-z0-9]*[[^]*][ ]*=[  {]*"[^\]*\"[^"]*"

       The default patterns file is divided into the following sections:(1)  In	the $SRCHEAD1 section, the and commands place text in this section at the beginning of the first new source program, which is pre-
	    fixed by These commands define the native language file descriptors that point to the message catalog.(2)  In the $SRCHEAD2 section, the and commands place text in this section at the beginning of the second and  remaining  source  programs.
	    These  commands  also  define  the native language file descriptors that point to the message catalog. $SRCHEAD2 contains the external
	    declaration of the nl file descriptor.(3)  In the $CATHEAD section, the and commands place text in this section at the beginning of the message catalog.(4)  In the $REWRITE section, you specify how the and commands should replace the extracted strings in the new source program. You can sup-
	    ply three options to the command:

	    %s	 This  option increments the set number for each source. This option applies only if you are using the command.  For more informa-
		 tion on set numbers, see the reference page.

	    %n	 This option increments the message number for each string extracted. This option applies if you are using either the or commands.

	    %t	 This option expands the text from the string extracted. The string can be a error message or the  default  string  extracted  and
		 printed  by the command. For example, if you want an error message to appear when is unable to retrieve the message from the mes-
		 sage catalog, you would include the following line:
		 catgets(_m_catd, %s, %n, "BAD STRING")

		 When fails, it returns the message BAD STRING.(5)  In the $MATCH section, you specify the patterns in the form of a regular expression that you want the and commands to find and  match.
	    The regular expression follows the same syntax rules as defined in reference page.(6)  In	the $REJECT section, you specify the matched strings that you do not want the and commands to replace in your source program.  The
	    regular expression follows the same syntax rules as defined in reference page.(7)  In the $ERROR section, the and commands look for bad matches and notify you with a warning message. The regular expression follows the
	    same syntax rules as defined in the reference page.

See Also
       intro(3int), extract(1int), strextract(1int), strmerge(1int), trans(1int), regex(3)
       Guide to Developing International Software

																    patterns(5int)
All times are GMT -4. The time now is 08:47 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy