Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: use of sed over cat to merge files
Top Forums UNIX for Advanced & Expert Users use of sed over cat to merge files Post 302147767 by miwinter on Wednesday 28th of November 2007 11:25:33 AM
Old 11-28-2007
use of sed over cat to merge files
I have three files, basically:

file 1 - one line header
file 2 - big data (approx 80GB)
file 3 - a one line trailer

the existing process cats these together i.e cat file 1 file 2 file 3

however... I was thinking, surely it could be more efficient to insert the header (file 1) on the data file (file 2) and then just append the trailer (file 3) on the end.

I wondered if this could be achieved some way via sed instead, i.e. something like:

#!/bin/sh

export HEAD=`cat $1`
export TRAIL=`cat $3`

sed -e '1i\
$HEAD
$a\
$TRAIL' $2 > complete_file.txt

...but it seems this won't substitute in the variable values setup within the shell script already

Thanks in advance for any replies on this one.
Last edited by miwinter; 11-28-2007 at 01:24 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

tr, sed, awk, cat or scripting

I need to change all Newline caracters (\12) to Fieldseparator(\34). tr -A '\12' '\34' <file1> file2 Replace all delete (\177) with Newline (\12) tr -A '\177' '\12' <file2> file3 Put the name of the file first in all rows. awk '{printf "%s\34%s\n", FILENAME,$0} file3 > file4 So far no... (6 Replies)
Discussion started by: MrKlint
6 Replies

2. Shell Programming and Scripting

Help on cat with sed!

Hi All! I have a script having one statement like. cat xfile | sed 's/A/a/g' > xfile I have two boxes which have similar version of linux running but one is greater in speed compared to other. Lets say A is faster than B. A which is faster writes output as NULL B which... (6 Replies)
Discussion started by: kbalaji_uk
6 Replies

3. Shell Programming and Scripting

want to pass sentence with SED and CAT

I have to pass a sentence in a file, the specs are as: cat run | sed 's/SRT/'$8'/g' | sed 's/plength/68/g' | sed 's/stcol/'$5'/g' | sed 's/encol/'$6'/g' | sed 's/brdtype/'$1'/g' | sed 's/brdtxt/'$3'/g' | sed 's/demotxt/Total '$2'/g' | sed 's/bantxt/ban_'$7'/g' | sed 's/validcodes/'$4'/g' > runx... (1 Reply)
Discussion started by: patilrakesh1984
1 Replies

4. Shell Programming and Scripting

grep or cat using sed

Is there a way using grep or cat a file to create a new file based on whether the first 9 positions of each record is less than 399999999? This is a fixed file format. (3 Replies)
Discussion started by: ski
3 Replies

5. Shell Programming and Scripting

Merge 2 CSV files using sed

Help in writing a script using sed which updates fileOne with the contents from fileTwo Example: Contents of fileOne 1,111111 2,897823 3,235473 4,222222 Contents of fileTwo 1,111111,A,1,2 4,222222,A,2,2 5,374632,A,3,2 6,374654,A,4,2 Final File should be: 1,111111,A,1,2... (9 Replies)
Discussion started by: NewToSed
9 Replies

6. UNIX for Advanced & Expert Users

cat / sed process weird characters

Hi everyone, I'm trying to write a shell script that process a log file. The log format is generally: (8 digit hex of unix time),(system ID),(state)\n My shell script gets the file from the web, saves it in a local text directory. I then want to change the hex to decimal, convert from unix time... (7 Replies)
Discussion started by: bencpeters
7 Replies

7. UNIX for Dummies Questions & Answers

emulating cat in sed with q option

Hi, I am aware that the below are the equivalent in sed for cat command. sed ':' sed -n 'p' Is there any way to emulate the same using "q" option in sed? Thanks (8 Replies)
Discussion started by: pandeesh
8 Replies

8. UNIX for Dummies Questions & Answers

How to cat via ssh and sed at the same time.?

I am trying to write a script to automatically create conf files and remote servers. I would like to do all this without creating files locally and copying them . Here is what I have tried. sitename=$1 prodserver=$2 ssh $prodserver "cat > /data/$sitename.conf" << cat |sed... (5 Replies)
Discussion started by: macrossm
5 Replies

9. Shell Programming and Scripting

Merge files and copy some data using sed or awk

Copy data from other file to paste cat file1: Name: server1.data.com data1 server1 running Name: server3.data.com data3 server3 running cat file2: server1 good server2 bad network not ok server3 good Output: (10 Replies)
Discussion started by: kenshinhimura
10 Replies

10. UNIX for Beginners Questions & Answers

Merge multiple columns into one using cat

I will like to merge several files using 'cat', but I observe the output is not consistent. the merge begins at the last line of the first file. file1.txt: 1234 1234 1234 file2.txt: aaaa bbbb cccc dddd cat file1.txt file2.txt > file3.txt file3.txt: 1234 1234 1234aaaa bbbb cccc... (13 Replies)
Discussion started by: geomarine
13 Replies

LEARN ABOUT DEBIAN

git-merge-index

GIT-MERGE-INDEX(1)						    Git Manual							GIT-MERGE-INDEX(1)

NAME

       git-merge-index - Run a merge for files needing merging

SYNOPSIS

       git merge-index [-o] [-q] <merge-program> (-a | [--] <file>*)

DESCRIPTION

       This looks up the <file>(s) in the index and, if there are any merge entries, passes the SHA1 hash for those files as arguments 1, 2, 3
       (empty argument if no file), and <file> as argument 4. File modes for the three files are passed as arguments 5, 6 and 7.

OPTIONS

       --
	   Do not interpret any more arguments as options.

       -a
	   Run merge against all files in the index that need merging.

       -o
	   Instead of stopping at the first failed merge, do all of them in one shot - continue with merging even when previous merges returned
	   errors, and only return the error code after all the merges.

       -q
	   Do not complain about a failed merge program (a merge program failure usually indicates conflicts during the merge). This is for
	   porcelains which might want to emit custom messages.

       If git merge-index is called with multiple <file>s (or -a) then it processes them in turn only stopping if merge returns a non-zero exit
       code.

       Typically this is run with a script calling git's imitation of the merge command from the RCS package.

       A sample script called git merge-one-file is included in the distribution.

       ALERT ALERT ALERT! The git "merge object order" is different from the RCS merge program merge object order. In the above ordering, the
       original is first. But the argument order to the 3-way merge program merge is to have the original in the middle. Don't ask me why.

       Examples:

	   torvalds@ppc970:~/merge-test> git merge-index cat MM
	   This is MM from the original tree.			 # original
	   This is modified MM in the branch A. 		 # merge1
	   This is modified MM in the branch B. 		 # merge2
	   This is modified MM in the branch B. 		 # current contents

       or

	   torvalds@ppc970:~/merge-test> git merge-index cat AA MM
	   cat: : No such file or directory
	   This is added AA in the branch A.
	   This is added AA in the branch B.
	   This is added AA in the branch B.
	   fatal: merge program failed

       where the latter example shows how git merge-index will stop trying to merge once anything has returned an error (i.e., cat returned an
       error for the AA file, because it didn't exist in the original, and thus git merge-index didn't even try to merge the MM thing).

GIT

       Part of the git(1) suite

Git 1.7.10.4							    11/24/2012							GIT-MERGE-INDEX(1)
All times are GMT -4. The time now is 03:33 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy