Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Merge and Sort tabular data from different text files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merge and Sort tabular data from different text files
# 1  
Old 10-15-2015
Computer Merge and Sort tabular data from different text files
I have 42 text files; each containing up to 34 lines with following structure;

file1
Code:
H-01 23
H-03 5
H-05 9
H-02 14
.
.

file2
Code:
H-01 17
H-02 43
H-04 7
H-05 8
H-03 7
.
.

file3
Code:
H-01 11
H-05 14
H-04 2
H-02 8
H-06 7
.
.

My desired output;
Code:
     file1 file2 file3 . . . file42
H-01 23 17 11
H-02 14 43 8
H-03 5 7 -
H-04 - 7 2
H-05 9 8 14
H-06 - - 7
.
.

The resulting file will contain a data matrix of 42 columns and 34 fields (excluding the headers). I have tried grep, but it prints the first occurrence only. Kindly help me.
Last edited by Syeda Sumayya; 10-15-2015 at 04:57 AM.. Reason: Description was not so clear.
# 2  
Old 10-15-2015
Hello Syeda,

For file1 and file2, could you please try following and let me know if this helps.
Code:
awk 'BEGIN{print "     file1 file2"}FNR==NR{;A[$1]=$NF;next} ($1 in A){print $1 OFS A[$1] OFS $NF} !($1 in A){print $1 OFS "-" OFS $NF}' file1 file2 | sort -k1,3

Output will be as follows.
Code:
     file1 file2
H-01 23 17
H-02 14 43
H-03 5 7
H-04 - 7
H-05 9 8
H-224 - 8

Please do let us know if you have any queries on same.

Thanks,
R. Singh
Last edited by RavinderSingh13; 10-15-2015 at 02:52 AM..
This User Gave Thanks to RavinderSingh13 For This Post:
Syeda Sumayya
# 3  
Old 10-15-2015
Thank you for the script @RavinderSingh13.

There are two queries with the current script;
1. The output lacks the line which should contain a dash (-).
2. The headers are appearing at the bottom of each column.

The current output is;
Code:
H-01 23 17
H-02 14 43
H-03 5 7
H-05 9 8
     file1 file2

# 4  
Old 10-15-2015
Hello Syeda,

Not sure which command you have used, following and post#2 command provided be me works fine for me as follows.
Code:
awk 'BEGIN{print "     file1 file2"}FNR==NR{;A[$1]=$NF;next} ($1 in A){print $1 OFS A[$1] OFS $NF} !($1 in A){print $1 OFS "-" OFS $NF}' file1 file2 | sort -k1,3
     file1 file2
H-01 23 17
H-02 14 43
H-03 5 7
H-04 - 7
H-05 9 8

NOTE: It checks those contents which are present in file2 and print - for those which are present in file2 and NOT in file1.

Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
Syeda Sumayya
# 5  
Old 10-15-2015
Thank you RavinderSingh13 for the comment. Query 1 is resolved if I place the file with less lines as file1 and that with more lines as file 2. But I need a script irrespective of the file order. And it should be applied to many files (42 in my case).

Query 2 still persists, but it is not a major issue. I will overcome it.
# 6  
Old 10-15-2015
Perhaps something more like:
Code:
awk '
BEGIN {	printf("    ")
	s = "sort"
}
FNR == 1 {
	printf(" %s", files[++fc] = FILENAME)
}
{	d[$1, fc] = $2
	c1[$1]
}
END {	print ""
	for(i in c1) {
		printf("%s", i) | s
		for(j = 1; j <= fc; j++)
			printf(" %s", ((i, j) in d) ? d[i, j] : "-") | s
		print "" | s
	}
	close(s)
}' file1 file2 file3 ... file42

As always, if someone wants to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.

When run with just your sample file1, file2, and file3 with the trailing lines containing periods removed, it produces the output:
Code:
     file1 file2 file3
H-01 23 17 11
H-02 14 43 8
H-03 5 7 -
H-04 - 7 2
H-05 9 8 14
H-06 - - 7

These 3 Users Gave Thanks to Don Cragun For This Post:
giuliangiuseppe RavinderSingh13 Syeda Sumayya
# 7  
Old 10-16-2015
Thank you Mr. Don Cragun for the correct answer.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Sort data in text file in particular format

I have to sort below output in text file in unix bash 20170308 DA,I,113 20170308 PM,I,123 20170308 DA,U,22 20170308 PM,U,123 20170309 DA,I,11 20170309 PM,I,23 20170309 DA,U,123 20170309 PM,U,233 (8 Replies)
Discussion started by: Adfire
8 Replies

2. UNIX for Dummies Questions & Answers

Merge two text files (oh no, not again!)

Hello, I'm new to this forum. I have always made good use of all the wise hints shown here. But this time I'm struggling with an issue that is driving me crazy. I have two text files, I have to merge them based on the first column, resulting file must contain all record from the first file... (4 Replies)
Discussion started by: emare
4 Replies

3. Shell Programming and Scripting

Extract data in tabular format from multiple files

Hi, I have directory with multiple files from which i need to extract portion of specif lines and insert it in a new file, the new file will contain a separate columns for each file data. Example: I need to extract Value_1 & Value_3 from all files and insert in output file as below: ... (2 Replies)
Discussion started by: belalr
2 Replies

4. Shell Programming and Scripting

Generate tabular data based on a column value from an existing data file

Hi, I have a data file with : 01/28/2012,1,1,98995 01/28/2012,1,2,7195 01/29/2012,1,1,98995 01/29/2012,1,2,7195 01/30/2012,1,1,98896 01/30/2012,1,2,7083 01/31/2012,1,1,98896 01/31/2012,1,2,7083 02/01/2012,1,1,98896 02/01/2012,1,2,7083 02/02/2012,1,1,98899 02/02/2012,1,2,7083 I... (1 Reply)
Discussion started by: himanish
1 Replies

5. Shell Programming and Scripting

AWK to match and merge data from 2 files into 1.

Hello, hopefully this is an easy on for the AWK guru's out there. I'm having some trouble figuring out how to match+merge data in 2 files into 1 single report. I've got my 2 files filtered and delimited, just need to MATCH $3 in file1 to $1 in file2, then put $0 from File1 and $2+$3 from File2... (6 Replies)
Discussion started by: right_coaster
6 Replies

6. UNIX for Dummies Questions & Answers

Merge files and sort them

hi all, i have three files naming file1 , file2, file3 . i want to merge the content of these three files, sort them and display the sorted output on the screen page by page. (1 Reply)
Discussion started by: sonu_pal
1 Replies

7. AIX

merge text files

Hello. Could you please help to know the command to merge multiple text files into one? I am thinking to use: cat f1.txt f2.txt f3.txt > f4.txt Is it okay to use cat command for same purpose - Or could there be any disadvantage in using it? Thank you (4 Replies)
Discussion started by: panchpan
4 Replies

8. Shell Programming and Scripting

Merge 70 files into one data matrix

Hi, I have a list of 70 files in a directory and I need to merge the content of each file into one big matrix file (71 columns x 3060 rows). Each file has the following format only two columns per file: unique identifier1 randomtext1 randomtext1 a 5 b 3 c 6 d 3 e 2... (11 Replies)
Discussion started by: labrazil
11 Replies

9. UNIX for Dummies Questions & Answers

converting a tabular format data to comma seperated data in KSH

Hi, Could anyone help me in changing a tabular format output to comma seperated file pls in K-sh. Its very urgent. E.g : username empid ------------------------ sri 123 to username,empid sri,123 Thanks, Hema:confused: (2 Replies)
Discussion started by: Hemamalini
2 Replies
Login or Register to Ask a Question