Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Merging two files each contain 16 lakh lines on HP-UX 11.11 system

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merging two files each contain 16 lakh lines on HP-UX 11.11 system
# 8  
Old 03-14-2016
How many lines in file1, and in file2?

---------- Post updated at 21:33 ---------- Previous update was at 21:23 ----------

I did a test with roundabout 3000 lines, and one file 100 lines longer than the other. It worked.
# 9  
Old 03-15-2016
It is working fine upto 5000 lines (not sure) ,but in my case each file contain more than 15000 lines.

File =16739
File2 = 17512

Lenght of each line is below 100 characters

Can you suggest me for this much large file sizeSmilie
# 10  
Old 03-15-2016
Try this shell script:
Code:
#!/bin/ksh
while :
do
  for descriptor in 3 4
  do
    for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14
    do
      IFS= read -r line || break 3
      printf "%s\n" "$line"
    done <&$descriptor
  done
done 3< file1 4< file2

The outer loop opens the two files via descriptors 3 and 4 and does never end.
The 2nd loop toggles between the the two descriptors.
The 3rd loop reads 14 lines from the current descriptor.
The break 3 breaks out from the 3 nested loops
If the output looks good, you can redirect the whole stuff to a file
Code:
...
done 3< file1 4< file2 > file3

Last edited by MadeInGermany; 03-15-2016 at 06:16 PM..
# 11  
Old 03-16-2016
Hi Phani,

can we split src file in 16 files each with 1 lakh records and then process each one by one with codes suggested here..?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Merging specifc lines of three files into one?

Recall that nc-county-pop.dat has 100 lines, and each line corresponds to a county. The files girls.dat and boys.dat each has 50 lines. Assume that the girls come from the first 50 counties and each county has only one girl. That is the first girl comes from the first county, the second girl comes... (1 Reply)
Discussion started by: novicep11
1 Replies

2. Shell Programming and Scripting

Merging multiple lines to columns with awk, while inserting commas for missing lines

Hello all, I have a large csv file where there are four types of rows I need to merge into one row per person, where there is a column for each possible code / type of row, even if that code/row isn't there for that person. In the csv, a person may be listed from one to four times... (9 Replies)
Discussion started by: RalphNY
9 Replies

3. Shell Programming and Scripting

Merging 2 lines together

I have a small problem, which due to my lack of knowledge, has left me unable to decipher some of the solutions that I looked at on these forums. So below is a piece of text, which I ran via cat -vet, which comes from within a program file. I have many such programs to process and repeatable,... (4 Replies)
Discussion started by: skarnm
4 Replies

4. Shell Programming and Scripting

Merging multiple files using lines from one file

I have been working of this script for a very long time and I have searched the internet for direction but I am stuck here. I have about 3000 files with two columns each. The length of each file is 50000. Each of these files is named this way b.4, b.5, b.6, b.7, b.8, b.9, b.10, b.11, b.12... (10 Replies)
Discussion started by: iconig
10 Replies

5. Shell Programming and Scripting

Merging lines

Thanks it worked for me. I have one more question on top of that. We had few records which were splitted in 2 lines instead of one. Now i identified those lines. The file is too big to open via vi and edit it. How can i do it without opening the file. Suppose, I want line number 1001 & 1002 to... (2 Replies)
Discussion started by: Gangadhar Reddy
2 Replies

6. Shell Programming and Scripting

merging two .txt files by alternating x lines from file 1 and y lines from file2

Hi everyone, I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines). Does anybody have an idea how to do that using perl, awk or sed?... (6 Replies)
Discussion started by: ink_LE
6 Replies

7. Shell Programming and Scripting

Merging lines

Hi folks. Could somebody help me write a script or command that will look through a file and for every line that doesn't contain a certain value, merge it with the one above? For example, the file contains: SCOTLAND|123|ABC|yes SCOTLAND|456|DEF|yes SCOTLAND|78 9|GHI|yes ... (3 Replies)
Discussion started by: MDM
3 Replies

8. Shell Programming and Scripting

Matching lines across multiple csv files and merging a particular field

I have about 20 CSV's that all look like this: "","","","","","","","","","","","","","","",""What I've been told I need to produce is the exact same thing, but with each file now containing the start_code from every other file where the email matches. It doesn't matter if any of the other... (1 Reply)
Discussion started by: Demosthenes
1 Replies

9. Shell Programming and Scripting

Merging files with AWK filtering and counting lines

Hi there, I have a couple of files I need to merge. I can do a simple merge by concatenating them into one larger file. But then I need to filter the file to get a desired result. The output looks like this: TRNH 0000000010941 ORDH OADR OADR ORDL ENDT 1116399 000000003... (2 Replies)
Discussion started by: Meert
2 Replies

10. UNIX for Dummies Questions & Answers

Merging lines into one

Hello. I would be very pleased if sb. help me to solve my problem. I've got a file with many non blank lines and I want to merge all lines into one not destroy the informations on them. I've tryed it with split and paste, tr, sed , but everything I've done has been wrong. I know about crazy... (8 Replies)
Discussion started by: Foxgard
8 Replies
Login or Register to Ask a Question