building output file from multiple input files

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting building output file from multiple input files
# 1  
Old 08-03-2009
building output file from multiple input files

Hi there,

I am trying to figure out a way to combine multiple sources with different data on a single file, and I am trying to find the best way to do it.

I have multiple files, let's say A, B, C and D. A has a field in common with B, B has a field in common with C, and C has a field in common with D.

I want the output file to have all records from file A, and to pull the fields from the other sources, for example

file A
file B (field 1 is common to A field 1)
1 10
2 20
3 30
4 40
file C (field 1 is common to B field 2)
10 abc
20 def
30 ghi
40 jkl
file D (field 1 is common to C field 2)
abc Cat
def Bird
ghi Dog
xyz Fish

The desired output file would contain

2 20 def Bird
3 30 ghi Dog

As I am new to this, I was first thing about running a mix of nested while read's and greps and echoing the output to a file. Is there any cleaner way to do this? Each file can be really big, and I think nesting while read's can make it slow.

Any tips will be highly appreciated! Smilie
# 2  
Old 08-04-2009
It's complicated to do this in one awk command with many arrays, it's more clarifying to split this in more steps:

awk 'NR==FNR{a[$1];next}
$1 in a
' fileA fileB > temp

awk 'NR==FNR{a[$2]=$0;next}
$1 in a {print a[$1], $2}
' temp fileC > temp1

awk 'NR==FNR{a[$3]=$0;next}
$1 in a {print a[$1], $2}
' temp1 fileD

rm temp temp1

# 3  
Old 08-04-2009
Originally Posted by Franklin52
It's complicated to do this in one awk command with many arrays, it's more clarifying to split this in more steps:
Thank you for your idea! Really it is not necessary to be a one-liner... just need something simple.

So far I was trying something like this:

while read field1; do
    B2='cat fileB | nawk -v fi=$field1 '{ if ( $1 == fi ) print $2 }''
    C2='cat fileC | nawk -v fi=$B2 ' { if ( $2 == fi ) print $1 }''
    D2='cat fileD | nawk -v fi=$C2 ' { if ( $2 == fi ) print $1 }''
    echo "$field1 $B2 $C2 $D2" >> output
done < fileA

(may not be actual commands, typing from memory)

but something was broken... and I had to sleep, so I'll go back at it again today. Anyway, this is not very good as it will parse all files several times, one for each record in fileA.

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Building a dynamic UNIX menu with input files

Hi! I am looking to build dynamic menu (named: that reads a pipe delimited file for input. for example, contents of input.txt could be: user1|srv1 user3|srv1 user4|srv1 user2|srv2 I want the menu look like: 1) get password for user1 on srv1 2) get password for user3 on... (7 Replies)
Discussion started by: cpolikowsky
7 Replies

2. Shell Programming and Scripting

Copying a file to multiple other files using a text file as input

Hello, I have a file called COMPLIST as follows that contains 4 digit numbers.0002 0003 0010 0013 0015 0016 0022 0023 0024 0025 0027 0030 0031 0032 0033 0035 0038 0041 (3 Replies)
Discussion started by: sph90457
3 Replies

3. Shell Programming and Scripting

[Solved] Multiple input files and output files

Hi, I have many test*.ft1 files to which I want to read as input for a script called pipe2txt.tcl and print the output in each separate file. For example, pipe2txt.tcl < test001.ft1 > test001.txt How can I read many files in this maner? thank you very much, Best, Pahuja (5 Replies)
Discussion started by: Pahuja
5 Replies

4. Shell Programming and Scripting

Search & Replace in Multiple Files by reading a input file

I have a environment property file which contains: Input file: value1 = url1 value2 = url2 value3 = url3 and so on. I need to search all *.xml files under directory for value1 and replace it with url1. Same thing I have to do for all values mentioned in input file. I need script in unix bash... (7 Replies)
Discussion started by: Shamkamde
7 Replies

5. Shell Programming and Scripting

Split: File into multiple and keeping the same 3 lines from input into all output files

The following code will split the infile into multiple files. However, I need it to insert the same first 3 lines from the original input file into each splitted file. How do I modify my script below to do so: print -n "Enter file name to split? " ; read infile if then echo "Invalid file... (4 Replies)
Discussion started by: mrn6430
4 Replies

6. Shell Programming and Scripting

awk, multiple files input and multiple files output

Hi! I'm new in awk and I need some help. I have a folder with a lot of files and I need that awk do something in each file and print a new file with the output. The input file name should be modified when I print the outpu files. Thanks in advance for help! :-) ciao (5 Replies)
Discussion started by: gabrysfe
5 Replies

7. Shell Programming and Scripting

How to ftp multiple files by taking the file name from a input file.

Hi, I'm working on a script which has to copy multiple files from one server to another server. The list of files that are to be copied is present in a file say input.txt. vi input.txt abc.c welcome.c new.c welcome1.c for ftp'ing a single file say 'new.c' the following code... (2 Replies)
Discussion started by: i.srini89
2 Replies

8. Shell Programming and Scripting

Split a file into multiple files based on the input pattern

I have a file with lines something like. ...... 123_start ...... ....... 123_end .... ..... 456_start ...... ..... 456_end .... ..... 789_start .... .... 789_end (6 Replies)
Discussion started by: abinash
6 Replies

9. Shell Programming and Scripting

How to add data from 2 input files and save it in 1 output file

Hi, i have 2 input files which are file1.txt and file2.txt. I need to extract data from file1.txt and file2.txt and save it in file3.txt like example below:- File1.txt ID scrap1 Name scrap1 start 1 end 10 ID scrap2 Name scrap2 start 11 end ... (4 Replies)
Discussion started by: redse171
4 Replies

10. Shell Programming and Scripting

Create Multiple files by reading a input file and changing the contents

Being new to this area .I have been assigned a task which i am unable to do . Can any one please help me . Hi I have requirement where i have input file XYZ_111_999_YYYYMMDD_1.TXT and with header and series of Numbers and Footer. I want to create a mutiple output files with each file having a... (2 Replies)
Discussion started by: bhargavkr
2 Replies
Login or Register to Ask a Question