Best performance to merge two files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Best performance to merge two files
# 1  
Old 12-13-2018
Best performance to merge two files

Hi Gurus,


I need to merge two files.


file1 (small file, only one line)
Code:
this is first line

file2 (large file)
Code:
 abc
def
ghi
...

I use below command to merge the file, since the file2 is really large file, the command read whole file2, the performance is not good.
Code:
cat file1 > file3
cat file2 >> file2

I searched internet, there is suggestion to use
Code:
sed -i '1 i\anything' file or 
sed -i '1i text' filename

the command thrown out error
Code:
sed: illegal option -- i

please provide your valuable suggestion.

my os is SunOS 5.10 Generic_150400-04 sun4v sparc sun4v


thanks in advance

Last edited by green_k; 12-13-2018 at 01:05 PM..
# 2  
Old 12-13-2018
Please define what you mean by merge.
Maybe providing a desired output given your sample inputs.
# 3  
Old 12-13-2018
Quote:
Originally Posted by vgersh99
Please define what you mean by merge.
Maybe providing a desired output given your sample inputs.
thanks vgersh99.
Actually, it just add one line (from file1) at begin of file2
in put file:
file1
Code:
this is first line

file2
Code:
abc
def
ghi
...

the desired output is:
Code:
this is first line
abc
def
ghi
...


Last edited by green_k; 12-13-2018 at 01:35 PM..
# 4  
Old 12-13-2018
cat file1 file2 > file3
Or something else?
# 5  
Old 12-13-2018
Quote:
Originally Posted by vgersh99
cat file1 file2 > file3
Or something else?
Thanks vgersh99.
my current code is using cat command as you suggested. because file2 is really large file, so, it takes long time to run, I am wondering if there is better way to do it to get better performance
# 6  
Old 12-13-2018
For this operation there is no faster way. Even "sed -i" would create a temporary new file.
If you want to do this operation multiple times, then there are ways for optimization.
For example
Code:
cat file1a file2 > file3; mv file3 file2
cat file1b file2 > file3; mv file3 file2

can be optimized
Code:
cat file1b file1a file2 > file3; mv file3 file2

# 7  
Old 12-14-2018
There's not really a go-faster button for file operations.

How do you need to use this file? If you can use it as a stream, you can avoid that delay by just not creating the file at all.

Code:
cat file1 file2 > file3
command file3

could conceivably be done as

Code:
cat file1 file2 | command /dev/stdin

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

DB2 - Performance Issue using MERGE option

Dear Team, I am using DB2 v10.5 and trying to load huge data using MERGE option ( 10-12 Million) using below query. Basically it loads data from staging to target . Staging table schemaname.Customer_Staging has 12 Million records . Runs for 25 Mins for just select query. But while doing merge... (2 Replies)
Discussion started by: Perlbaby
2 Replies

2. Shell Programming and Scripting

Merge files and generate a resume in two files

Dear Gents, Please I need your help... I need small script :) to do the following. I have a thousand of files in a folder produced daily. I need first to merge all files called. txt (0009.txt, 0010.txt, 0011.txt) and and to output a resume of all information on 2 separate files in csv... (14 Replies)
Discussion started by: jiam912
14 Replies

3. Shell Programming and Scripting

Checking in a directory how many files are present and basing on that merge all the files

Hi, My requirement is,there is a directory location like: :camp/current/ In this location there can be different flat files that are generated in a single day with same header and the data will be different, differentiated by timestamp, so i need to verify how many files are generated... (10 Replies)
Discussion started by: srikanth_sagi
10 Replies

4. UNIX for Dummies Questions & Answers

Merge files

Hi, I would like to know how can I merge files based on their coordinates, but mantaining the score of each file in the output file like: Note: 1st column is for chromosome, 2nd for start, 3rd for end of segment, 4th for score file1: 1 200 300 20 1 400 500 30 file2: 1 200 350 30 1... (1 Reply)
Discussion started by: fadista
1 Replies

5. Shell Programming and Scripting

How can i merge these two files into several...

Given are File A and File B File A has for example 5 lines: AAA BBB CCC DDD EEE File B has 3 lines: 111 222 333 How can i merge A and B into: 111 222 333 AAA (first line from A) then a new file: (4 Replies)
Discussion started by: Y-T
4 Replies

6. Shell Programming and Scripting

merge files

hi i have two files file1 1234567 1234678 1234679 file2 98765|jjkskk|9393|iyutr 98765|kkooo|9393|hjjjd 98765|abcvfg|9393|sskds output should be 1234567|jjkskk|9393|iyutr 1234678|kkooo|9393|hjjjd 1234679|abcvfg|9393|sskds (5 Replies)
Discussion started by: mad_man12
5 Replies

7. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies

8. Shell Programming and Scripting

merge files

Hi, i have the files f1 and f2 like: files f1: c1 a1 c2 a2 c3 a3 file f2: c1 b1 c2 b2 c3 b3 i want merge the f1 and f2 file to f3 file like: c1 a1 b1 c2 a2 b3 c3 a3 b3........ .... . . please help me onthis..... (5 Replies)
Discussion started by: koti_rama
5 Replies

9. Shell Programming and Scripting

How to merge files

Hello guys, I gotta question, i have a lot of log files (simple text) and i need to merge them in group of 10 files, one next to the other, that have sense? For example, i have the files: File1 File2 File3 File4 . . File100 I need to merge the contents of each file into a new file... (3 Replies)
Discussion started by: lestat_ecuador
3 Replies

10. Shell Programming and Scripting

help in merge files

I have created these files in a particular directory a_1.txt a_2.txt ... .. a_n.txt Each file has a single line. I want to write a output file a.txt, which will have concantated value of string from all the files. which utility should i use. copy, cat or paste???? Can anyone help... (5 Replies)
Discussion started by: u263066
5 Replies
Login or Register to Ask a Question