Merge values from multiple directories into one file in awk or bash


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merge values from multiple directories into one file in awk or bash
# 1  
Old 10-05-2016
Merge values from multiple directories into one file in awk or bash

I am trying to merge or combine all $1 values in validation.txt from multiple directories into one new file and output it here tab-delimited:/home/cmccabe/Desktop/20x/total/total.txt. Each $2 value and the header would then be a new field in total.txt. I am not sure how to go about this as cat is not the right tool. My real data has 15 paths and 70 .txt files. Thank you Smilie.

/home/cmccabe/Desktop/20x/panel/cardiomyopathy/coverage/validation.txt path1
Code:
Sample    20xCARDIOMYOPATHY
NA00449    98.8
NA02782    98.9

/home/cmccabe/Desktop/20x/panel/idp/coverage/validation.txt path2
Code:
Sample    20xIDP
NA00449    98.6
NA02782    98.7

/home/cmccabe/Desktop/20x/panel/pid/coverage/validation.txt path3
Code:
Sample    20xPID
NA00449    98.9
NA02782    99.0

desired output of total.txt (tab-delimited)
Code:
Sample    20xCARDIOMYOPATHY 20xIDP 20xPID
NA00449    98.8              98.6   98.9
NA02782    98.9              98.7   99.0


Last edited by cmccabe; 10-05-2016 at 03:10 PM.. Reason: added details, fixed format
# 2  
Old 10-05-2016
Hello cmccabe,

Could you please try following and let me know if this helps you.
Code:
awk 'BEGIN{print "Sample 20xCARDIOMYOPATHY 20xIDP 20xPID"} FNR==NR && FNR>1{A[$1]=$2;next} ($1 in A){A[$1]=A[$1]?A[$1] OFS $2:$2} END{for(i in A){if(num=split(A[i],X," ")>1){print i,A[i]}}}'  Input_file1  Input_file2  Input_file3

Output will be as follows.
Code:
Sample 20xCARDIOMYOPATHY 20xIDP 20xPID
NA00449 98.8 98.6 98.9
NA02782 98.9 98.7 99.0

So above will give you output into a random sequence only. Please do let me know if this helps you.

NOTE: Also in above code put path1, path2 and path3 Input_files respectively while running the code.

Thanks,
R. Singh
# 3  
Old 10-05-2016
That helps thank you very much Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to extract multiple values from file and add two additional fields

In the attached file I am trying to use awk to extract multiple values and create the tab-delimited desired output. In the output R_Index is a the sequential # and Pre_Enrichment is defaulted to .. I can extract from the values to the side of the keywords, but most are above and I can not... (2 Replies)
Discussion started by: cmccabe
2 Replies

2. Shell Programming and Scripting

[bash] wanted: function with a clean way for multiple return values

Hi, I have a small part of a project which is done as a bash script. bash was selected as an portability issue that works out of the box. In this script I have an exec shell-function, a wrapper around arbitrary commands. I want to have STDOUT, as an addon STDERR and the EXIT-CODE of a specified... (5 Replies)
Discussion started by: stomp
5 Replies

3. Shell Programming and Scripting

Get multiple values from an xml file using one of the following commands or together awk/perl/script

Hello, I have a requirement to extract the value from multiple xml node and print out the values to new file to compare. Would be done using either awk/perl or some unix script. For example sample input file: ..... ..... <factories xmi:type="resources.jdbc:DataSource"... (2 Replies)
Discussion started by: slbmind
2 Replies

4. Shell Programming and Scripting

Read record from the text file contain multiple separated values & assign those values to variables

I have a file containing multiple values, some of them are pipe separated which are to be read as separate values and some of them are single value all are these need to store in variables. I need to read this file which is an input to my script Config.txt file name, first path, second... (7 Replies)
Discussion started by: ketanraut
7 Replies

5. Shell Programming and Scripting

Merge multiple lines in same file with common key using awk

I've been a Unix admin for nearly 30 years and never learned AWK. I've seen several similar posts here, but haven't been able to adapt the answers to my situation. AWK is so damn cryptic! ;) I have a single file with ~900 lines (CSV list). Each line starts with an ID, but with different stuff... (6 Replies)
Discussion started by: protosd
6 Replies

6. UNIX for Dummies Questions & Answers

Create multiple directories with awk

Hello all. Newbie here. In a directory, I have 50 files and one additional file that is a list of the names of the 50 files. I would like to create a directory for each of the 50 files, and I need the 50 directory names to correspond to the 50 file names. I know this can be done by running... (6 Replies)
Discussion started by: Zeckendorff
6 Replies

7. Shell Programming and Scripting

perform 3 awk commands to multiple files in multiple directories

Hi, I have a directory /home/datasets/ which contains a bunch (720) of subdirectories called hour_1/ hour_2/ etc..etc.. in each of these there is a single text file called (hour_1.txt in hour_1/ , hour_2.txt for hour_2/ etc..etc..) and i would like to do some text processing in them. Each of... (20 Replies)
Discussion started by: amarn
20 Replies

8. Shell Programming and Scripting

BASH: extracting values from multiple lines after a match

Hi there, I have the following output, # raidctl -l RAID Volume RAID RAID Disk Volume Type Status Disk Status ------------------------------------------------------ c0t1d0 IM OK c0t1d0 OK ... (4 Replies)
Discussion started by: rethink
4 Replies

9. Shell Programming and Scripting

bash script to rename multiple directories

Hello I have a directory structure with year in format 4 digits, e.g 2009, below which is month format 1 or 2 digits, e.g 1 or 12, blow which is day format 1 or 2 digits, e.g 1 or 31. I want to change the names of lots of directories to the be Year - 4 digits , e.g 2009 - No change here... (4 Replies)
Discussion started by: garethsays
4 Replies
Login or Register to Ask a Question