Merging multiple files using lines from one file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merging multiple files using lines from one file
# 8  
Old 09-24-2012
I have no idea how long it will take. As I said before, I don't have any idea whether it will even complete successfully. Try it and see what happens. If you run out of memory, hit argument limits, hit line length limits, etc. you can let us know exactly what errors are reported and we may then be able to help split up the calls to awk into chunk sizes that will work for your environment. Then we can use paste to join the chunks together.

I would suggest trying it with just b.? (instead of b.? b.?? b.??? and b.????) as a test to see that you're getting what you want and to get a an extremely rough estimate of how long it will take to process an input file. (However, each file processed will probably tale longer than the last file before it.)
# 9  
Old 09-24-2012
hi,
it did not take up to 5 minutes. It seems to give me what I want but I would have to check to make sure.
Thanks

---------- Post updated at 08:06 AM ---------- Previous update was at 08:04 AM ----------

I had to do a little adjustment to b.?, I used b* instead because it could not read the first one
# 10  
Old 09-24-2012
I am not sure what you want..

as per my understanding you want your output file as per the list file.

have you tried with join..?

try this..

Code:
cat list > Final_out
for i in b.*
do
join Final_out $i > temp_file
mv temp_file Final_out    
done

# 11  
Old 09-24-2012
Quote:
Originally Posted by iconig
hi,
it did not take up to 5 minutes. It seems to give me what I want but I would have to check to make sure.
Thanks

---------- Post updated at 08:06 AM ---------- Previous update was at 08:04 AM ----------

I had to do a little adjustment to b.?, I used b* instead because it could not read the first one
Hi iconig,
In your first message on this thread, you said:
Code:
Each of these files is named this way b.4, b.5, b.6, b.7, b.8, b.9, b.10, b.11, b.12 .......b.3000

If that statement was true, then b.? would match files b.4 through b.9. If you use:
Code:
list b.* b.?? b.??? b.????

instead of:
Code:
list b.? b.?? b.??? b.????

you will include output from files b.10, b.11, b.12 ... b.3000 in your output twice. If you use:
Code:
list b.*

instead of:
Code:
list b.? b.?? b.??? b.????

you'll get each file once, but the 1st few columns in the output will be from files list, b.10, b.100, b.1000, b.1001, and b.1002 instead of from files list, b.4, b.5, b.6, and b.7.

If you use list b* instead of list b.* and you have other files with first character "b" and a 2nd character that is not ".", you may also get output from some of them throwing off columns in some rows if they are not in the same format as the format you specified for the b.* files.

Last edited by Don Cragun; 09-24-2012 at 10:08 AM.. Reason: Missed a period.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Merging multiple lines into single line based on one column

I Want to merge multiple lines based on the 1st field and keep into single record. SRC File: AAA_POC_DB.TAB1 AAA_POC_DB.TAB2 AAA_POC_DB.TAB3 AAA_POC_DB.TAB4 BBB_POC_DB.TAB1 BBB_POC_DB.TAB2 CCC_POC_DB.TAB6 OUTPUT ----------------- 'AAA_POC_DB','TAB1','TAB2','TAB3','TAB4'... (10 Replies)
Discussion started by: raju2016
10 Replies

2. Shell Programming and Scripting

Merging multiple lines to columns with awk, while inserting commas for missing lines

Hello all, I have a large csv file where there are four types of rows I need to merge into one row per person, where there is a column for each possible code / type of row, even if that code/row isn't there for that person. In the csv, a person may be listed from one to four times... (9 Replies)
Discussion started by: RalphNY
9 Replies

3. Shell Programming and Scripting

Merging multiple lines

I do have a text file with multiple lines on it. I want to put the lines of text into a single line where ever there is ";" for example ert, ryt, yvig, fgr; rtyu, hjk, uio, hyu, hjo; ghj, tyu, gho, hjp, jklo, kol; The resultant file I would like to have is ert, ryt, yvig, fgr;... (2 Replies)
Discussion started by: Kanja
2 Replies

4. Shell Programming and Scripting

awk Merging multiple files with symbol representing new file

I just tried following ls *.dat|sort -t"_" -k2n,2|while read f1 && read f2; do awk '{print}' $f1 awk FNR==1'{print $1,$2,$3,$4,$5,"*","*","*" }' OFS="\t" $f2 awk '{print}' $f2 donegot following result 18-Dec-1983 11:45:00 AM 18.692 84.672 0 25.4 24 18-Dec-1983 ... (3 Replies)
Discussion started by: Akshay Hegde
3 Replies

5. Shell Programming and Scripting

Merging multiple files from multiple columns

Hi guys, I have very basic linux experience so I need some help with a problem. I have 3 files from which I want to extract columns based on common fields between them. File1: --- rs74078040 NA 51288690 T G 461652 0.99223 0.53611 3 --- rs77209296 NA 51303525 T G 461843 0.98973 0.60837 3... (10 Replies)
Discussion started by: bartman2099
10 Replies

6. Shell Programming and Scripting

merging multiple lines into single line

Hi, 1. Each message starts with date 2. There is blank line between each message 3. Each message does not contain same number of lines. Any help in merging multiple lines in each message to a single line is much appreciated. AIX: Korn Shell Error log file looks like below. ... (5 Replies)
Discussion started by: bala123
5 Replies

7. Shell Programming and Scripting

merging two .txt files by alternating x lines from file 1 and y lines from file2

Hi everyone, I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines). Does anybody have an idea how to do that using perl, awk or sed?... (6 Replies)
Discussion started by: ink_LE
6 Replies

8. Shell Programming and Scripting

Merging information from multiple files to a single file

Hello, I am new to unix and need help with a problem. I have 2 files each containing multiple columns of information ie; File 1 : A B C D E 1 2 3 4 5 File 2 : F G 6 7 I would like to merge the information from File 2 to File 1 so that the data reads as follows; File 1: A... (4 Replies)
Discussion started by: crunchie
4 Replies

9. Shell Programming and Scripting

Matching lines across multiple csv files and merging a particular field

I have about 20 CSV's that all look like this: "","","","","","","","","","","","","","","",""What I've been told I need to produce is the exact same thing, but with each file now containing the start_code from every other file where the email matches. It doesn't matter if any of the other... (1 Reply)
Discussion started by: Demosthenes
1 Replies

10. Shell Programming and Scripting

Merging columns from multiple files in one file

Hi, I want to select columns from multiple files and combine them in one file. The files are simulation-data-files with 23 columns each and about 50 rows. I now use: cut -f 11 Sweep?wing-30?scale=0.?0?fan2?.txt | pr -3 | awk '{printf("\n%s\t%s\t%s",$1,$2,$3)}' > ../Data_Processed/output.txtI... (1 Reply)
Discussion started by: isgoed
1 Replies
Login or Register to Ask a Question