Merging two files each contain 16 lakh lines on HP-UX 11.11 system


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merging two files each contain 16 lakh lines on HP-UX 11.11 system
# 1  
Old 03-13-2016
Merging two files each contain 16 lakh lines on HP-UX 11.11 system

Hello All ,

I am trying to merge two files each contain 16 lakh lines ..My requirement is i have merge after every 14 lines of each file .

Like from file1 14 lines then after after 14 lines form file2 ..so i wrote below script .

It is working for small files ,but large files script not running and PID of script get killed automatically after merging 90000 files ..

Please anyone help me inthis ,anyway that i can merge it in quick time .

Code:
#!/bin/sh
touch test
size=`ls -lrt file1 | awk '{print $5}'`
while [ "$size" -gt 0 ]
do
    sed -n 1,14 file1 >> test
    sed  '1,14d' file1 >testing1.tmp
    cat testing1.tmp >file1
    rm testing1.tmp
    sed -n 1,14p file2 >> test
    sed  '1,14d' file2 >testing2.tmp
    cat testing2.tmp >file2
    rm testing2.tmp
    size=`ls -lrt file1 | awk '{print $5}'`
done
echo " Files merged successfully "

# 2  
Old 03-13-2016
How about
Code:
awk '1; !(NR%14) {for (i=1; i<=14; i++)  {getline < F2; print}}' F2=file2 file1

This User Gave Thanks to RudiC For This Post:
# 3  
Old 03-14-2016
Code:
#!/usr/bin/awk -f
BEGIN {
File2 = ARGV[2]
    --ARGC
    }
    {
        print
            if((FNR % 14) == 0) {
                for(n=1; n<=14; ++n) {
                            getline <File2
                                    print
                                        }
                                            }
                                            }
                                            END {
                                                while (getline <File2)  print
                                                }

Above code is working for small sized files but coming to big size files it merging in zigzag manner .. Can anyone suggest which script works here
# 4  
Old 03-14-2016
What is "zigzag manner"?
# 5  
Old 03-14-2016
lines are repeated so many times by suppressing trailing lines.. like 0 1 2 3 4 5 5 5 5 5 5 10 11 12 12 12 ..like this i am receiving output ..
# 6  
Old 03-14-2016
I neglected error checking. Might be file2 is at its end, and getline failed repeatedly. This doesn't depend on the files' sizes but stems from file length unbalance. Try
Code:
awk '
function FLOK() {return (1 == getline < F2)}
1
!(NR%14)        {for (; (++i)%15 && FLOK();) print}
END             {while (FLOK()) print}
' F2=file2 file1

This User Gave Thanks to RudiC For This Post:
# 7  
Old 03-14-2016
I am receiving same error .. it is merging good upto 1000 lines then it is starting wrong merging .. Lines are getting repeated and some line getting supressedSmilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Merging specifc lines of three files into one?

Recall that nc-county-pop.dat has 100 lines, and each line corresponds to a county. The files girls.dat and boys.dat each has 50 lines. Assume that the girls come from the first 50 counties and each county has only one girl. That is the first girl comes from the first county, the second girl comes... (1 Reply)
Discussion started by: novicep11
1 Replies

2. Shell Programming and Scripting

Merging multiple lines to columns with awk, while inserting commas for missing lines

Hello all, I have a large csv file where there are four types of rows I need to merge into one row per person, where there is a column for each possible code / type of row, even if that code/row isn't there for that person. In the csv, a person may be listed from one to four times... (9 Replies)
Discussion started by: RalphNY
9 Replies

3. Shell Programming and Scripting

Merging 2 lines together

I have a small problem, which due to my lack of knowledge, has left me unable to decipher some of the solutions that I looked at on these forums. So below is a piece of text, which I ran via cat -vet, which comes from within a program file. I have many such programs to process and repeatable,... (4 Replies)
Discussion started by: skarnm
4 Replies

4. Shell Programming and Scripting

Merging multiple files using lines from one file

I have been working of this script for a very long time and I have searched the internet for direction but I am stuck here. I have about 3000 files with two columns each. The length of each file is 50000. Each of these files is named this way b.4, b.5, b.6, b.7, b.8, b.9, b.10, b.11, b.12... (10 Replies)
Discussion started by: iconig
10 Replies

5. Shell Programming and Scripting

Merging lines

Thanks it worked for me. I have one more question on top of that. We had few records which were splitted in 2 lines instead of one. Now i identified those lines. The file is too big to open via vi and edit it. How can i do it without opening the file. Suppose, I want line number 1001 & 1002 to... (2 Replies)
Discussion started by: Gangadhar Reddy
2 Replies

6. Shell Programming and Scripting

merging two .txt files by alternating x lines from file 1 and y lines from file2

Hi everyone, I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines). Does anybody have an idea how to do that using perl, awk or sed?... (6 Replies)
Discussion started by: ink_LE
6 Replies

7. Shell Programming and Scripting

Merging lines

Hi folks. Could somebody help me write a script or command that will look through a file and for every line that doesn't contain a certain value, merge it with the one above? For example, the file contains: SCOTLAND|123|ABC|yes SCOTLAND|456|DEF|yes SCOTLAND|78 9|GHI|yes ... (3 Replies)
Discussion started by: MDM
3 Replies

8. Shell Programming and Scripting

Matching lines across multiple csv files and merging a particular field

I have about 20 CSV's that all look like this: "","","","","","","","","","","","","","","",""What I've been told I need to produce is the exact same thing, but with each file now containing the start_code from every other file where the email matches. It doesn't matter if any of the other... (1 Reply)
Discussion started by: Demosthenes
1 Replies

9. Shell Programming and Scripting

Merging files with AWK filtering and counting lines

Hi there, I have a couple of files I need to merge. I can do a simple merge by concatenating them into one larger file. But then I need to filter the file to get a desired result. The output looks like this: TRNH 0000000010941 ORDH OADR OADR ORDL ENDT 1116399 000000003... (2 Replies)
Discussion started by: Meert
2 Replies

10. UNIX for Dummies Questions & Answers

Merging lines into one

Hello. I would be very pleased if sb. help me to solve my problem. I've got a file with many non blank lines and I want to merge all lines into one not destroy the informations on them. I've tryed it with split and paste, tr, sed , but everything I've done has been wrong. I know about crazy... (8 Replies)
Discussion started by: Foxgard
8 Replies
Login or Register to Ask a Question