11-28-2007
use of sed over cat to merge files
I have three files, basically:
file 1 - one line header
file 2 - big data (approx 80GB)
file 3 - a one line trailer
the existing process cats these together i.e cat file 1 file 2 file 3
however... I was thinking, surely it could be more efficient to insert the header (file 1) on the data file (file 2) and then just append the trailer (file 3) on the end.
I wondered if this could be achieved some way via sed instead, i.e. something like:
#!/bin/sh
export HEAD=`cat $1`
export TRAIL=`cat $3`
sed -e '1i\
$HEAD
$a\
$TRAIL' $2 > complete_file.txt
...but it seems this won't substitute in the variable values setup within the shell script already
Thanks in advance for any replies on this one.
Last edited by miwinter; 11-28-2007 at 01:24 PM..
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
I need to change all Newline caracters (\12) to Fieldseparator(\34).
tr -A '\12' '\34' <file1> file2
Replace all delete (\177) with Newline (\12)
tr -A '\177' '\12' <file2> file3
Put the name of the file first in all rows.
awk '{printf "%s\34%s\n", FILENAME,$0} file3 > file4
So far no... (6 Replies)
Discussion started by: MrKlint
6 Replies
2. Shell Programming and Scripting
Hi All!
I have a script having one statement like.
cat xfile | sed 's/A/a/g' > xfile
I have two boxes which have similar version of linux running but one is greater in speed compared to other. Lets say A is faster than B.
A which is faster writes output as NULL
B which... (6 Replies)
Discussion started by: kbalaji_uk
6 Replies
3. Shell Programming and Scripting
I have to pass a sentence in a file, the specs are as:
cat run | sed 's/SRT/'$8'/g' | sed 's/plength/68/g' | sed 's/stcol/'$5'/g' | sed 's/encol/'$6'/g' | sed 's/brdtype/'$1'/g' | sed 's/brdtxt/'$3'/g' | sed 's/demotxt/Total '$2'/g' | sed 's/bantxt/ban_'$7'/g' | sed 's/validcodes/'$4'/g' > runx... (1 Reply)
Discussion started by: patilrakesh1984
1 Replies
4. Shell Programming and Scripting
Is there a way using grep or cat a file to create a new file based on whether the first 9 positions of each record is less than 399999999?
This is a fixed file format. (3 Replies)
Discussion started by: ski
3 Replies
5. Shell Programming and Scripting
Help in writing a script using sed which updates fileOne with the contents from fileTwo
Example:
Contents of fileOne
1,111111
2,897823
3,235473
4,222222
Contents of fileTwo
1,111111,A,1,2
4,222222,A,2,2
5,374632,A,3,2
6,374654,A,4,2
Final File should be:
1,111111,A,1,2... (9 Replies)
Discussion started by: NewToSed
9 Replies
6. UNIX for Advanced & Expert Users
Hi everyone,
I'm trying to write a shell script that process a log file. The log format is generally:
(8 digit hex of unix time),(system ID),(state)\n
My shell script gets the file from the web, saves it in a local text directory. I then want to change the hex to decimal, convert from unix time... (7 Replies)
Discussion started by: bencpeters
7 Replies
7. UNIX for Dummies Questions & Answers
Hi,
I am aware that the below are the equivalent in sed for cat command.
sed ':'
sed -n 'p'
Is there any way to emulate the same using "q" option in sed?
Thanks (8 Replies)
Discussion started by: pandeesh
8 Replies
8. UNIX for Dummies Questions & Answers
I am trying to write a script to automatically create conf files and remote servers. I would like to do all this without creating files locally and copying them .
Here is what I have tried.
sitename=$1
prodserver=$2
ssh $prodserver "cat > /data/$sitename.conf" << cat |sed... (5 Replies)
Discussion started by: macrossm
5 Replies
9. Shell Programming and Scripting
Copy data from other file to paste
cat file1:
Name: server1.data.com
data1 server1 running
Name: server3.data.com
data3 server3 running
cat file2:
server1 good
server2 bad
network not ok
server3 good
Output: (10 Replies)
Discussion started by: kenshinhimura
10 Replies
10. UNIX for Beginners Questions & Answers
I will like to merge several files using 'cat', but I observe the output is not consistent. the merge begins at the last line of the first file.
file1.txt:
1234
1234
1234
file2.txt:
aaaa
bbbb
cccc
dddd
cat file1.txt file2.txt > file3.txt
file3.txt:
1234
1234
1234aaaa
bbbb
cccc... (13 Replies)
Discussion started by: geomarine
13 Replies
LEARN ABOUT DEBIAN
flow-merge
flow-merge(1) General Commands Manual flow-merge(1)
NAME
flow-merge -- Merge flow files.
SYNOPSIS
flow-merge [-aghm] [-b big|little] [-C comment] [-d debug_level] [-o filename] [-z z_level] [file|directory ...]
DESCRIPTION
The flow-merge utility processes files and/or directories of files in the flow-tools format. The resulting merged data set is written to
the standard output or file specified by -o. If file is a single dash (`-') or absent, flow-merge will read from the standard input.
Unlike flow-cat, flow-merge interleaves flow records preserving the relative chronological order.
OPTIONS
-a Do not ignore filenames that begin with tmp.
-b big|little
Byte order of output.
-C Comment
Add a comment.
-d debug_level
Enable debugging.
-g Sort file list by capture start time before processing.
-h Display help.
-m Disable the use of mmap().
-p Preload headers. Use to preserve meta information such as lost flows.
-o file Write to file instead of the standard out.
-z z_level
Configure compression level to z_level. 0 is disabled (no compression), 9 is highest compression.
file|directory...
Process the files and/or directory.
EXAMPLES
Merge all flow files begining with ft-v05.2001-05.01, use flow-print to display the results.
flow-merge ft-v05.2001-05-01.* | flow-print
BUGS
None known.
AUTHOR
Larry Lidz ellidz@eridu.uchicago.edu
SEE ALSO
flow-tools(1)
flow-merge(1)