I have three files, basically:

file 1 - one line header
file 2 - big data (approx 80GB)
file 3 - a one line trailer

the existing process cats these together i.e cat file 1 file 2 file 3

however... I was thinking, surely it could be more efficient to insert the header (file 1) on the data file (file 2) and then just append the trailer (file 3) on the end.

I wondered if this could be achieved some way via sed instead, i.e. something like:

#!/bin/sh

export HEAD=`cat $1`
export TRAIL=`cat $3`

sed -e '1i\
$HEAD
$a\
$TRAIL' $2 > complete_file.txt

...but it seems this won't substitute in the variable values setup within the shell script already

Thanks in advance for any replies on this one.

You could use sed, but

it's not really more efficient. Sed still needs to read and write all of file 2 and needs to make checks on every line. Cat just needs to read and write, and can do so block by block rather than line by line.

I agree that there's little you can do about making a 80GB edit more efficient.

The reason your "sed" command did not work is because you used single-quotes around the expression, which prevents the variables from being evaluated:

So if you need to reference a variable, you can't use the single-quotes. But if you use double-quotes, you'll need to escape anything that the shell will evaluate but that you need to be literal.