Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Merge two files and add the count

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merge two files and add the count
# 1  
Old 09-08-2014
Merge two files and add the count
I have two files with same layout which i want to merge, removing the duplicate header and adding the count in footer

Code:
 
HD|09052014|09052014
data1|data2|data3
data1|data2|data3
data1|data2|data3
data1|data2|data3
TR|09052014|09052014|4

Code:
 
HD|09052014|09052014
dt1|dt2|dt3
dt1|dt2|dt3
TR|09052014|09052014|2

the output file should look like
Code:
 
HD|09052014|09052014
data1|data2|data3
data1|data2|data3
data1|data2|data3
data1|data2|data3
dt1|dt2|dt3
dt1|dt2|dt3
TR|09052014|09052014|6

How do i do this in shell script?
Moderator's Comments:
Mod Comment Please use CODE tags (not QUOTE) tags for sample input, output, and code segments.
Last edited by Don Cragun; 09-08-2014 at 04:18 PM.. Reason: Fix tags.
# 2  
Old 09-08-2014
You've been asking for our help for well over 5 years.

What have you tried to solve this problem on your own?

What utility (or utilities) do you think would be well suited to solve this problem?
# 3  
Old 09-08-2014
i have tried to do this, remove header and footer , merge both files and then add header footer back

Code:
sed '1d;$d' file1.txt > outfile1.txt
sed '1d;$d' file2.txt > outfile2.txt

cat outfile1.txt outfile2.txt > file.txt

Add header
Code:
sed -i '1s/^/HD|`date +%m%d%Y`|`date +%m%d%Y`\n/' file.txt

And stuck here, date is not displaying rahter the actual text is displaying, after i figure out how to add header , i got to add footer with row count -2
Last edited by Don Cragun; 09-08-2014 at 05:21 PM.. Reason: Add CODE tags.
# 4  
Old 09-08-2014
Adapting your approach, how about
Code:
echo "HD|$(date +%m%d%Y)|$(date +%m%d%Y)" > outfile
sed '1d;$d' file1 >> outfile 
sed '1d;$d' file2 >> outfile 
echo "TR|$(date +%m%d%Y)|$(date +%m%d%Y)|$(($(wc -l < outfile) -1))" >> outfile 
cat outfile 
HD|09082014|09082014
data1|data2|data3
data1|data2|data3
data1|data2|data3
data1|data2|data3
dt1|dt2|dt3
dt1|dt2|dt3
TR|09082014|09082014|6

To ensure the wc -l < outfile doesn't collide with the redirection, you could assign it to a variable upfront and then echo that variable. And, you could of course use a var for the date result...
This User Gave Thanks to RudiC For This Post:
depakjan
# 5  
Old 09-08-2014
The sed utility isn't very good for arithmetic. Try something like:
Code:
awk '
BEGIN {		FS = OFS = "|" }
NR==1 {		print } 
$1=="HD" {	next }
$1=="TR" {	$NF = count; t = $0; next }
{		print; count++ }
END {		print t }
' file1.txt file2.txt > file.txt

As long as you aren't trying to change the dates found in the headers and trailers in your input files, this will produce a single output file from one or more input files with the count set correctly in the trailer line.

If you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk.
This User Gave Thanks to Don Cragun For This Post:
depakjan
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Error files count while coping files from source to destination locaton as well count success full

hi All, Any one answer my requirement. I have source location src_dir="/home/oracle/arun/IRMS-CM" My Target location dest_dir="/home/oracle/arun/LiveLink/IRMS-CM/$dc/$pc/$ct" my source text files check with below example.text file content $fn "\t" $dc "\t" $pc "\t" ... (3 Replies)
Discussion started by: sravanreddy
3 Replies

2. Shell Programming and Scripting

Merge files and generate a resume in two files

Dear Gents, Please I need your help... I need small script :) to do the following. I have a thousand of files in a folder produced daily. I need first to merge all files called. txt (0009.txt, 0010.txt, 0011.txt) and and to output a resume of all information on 2 separate files in csv... (14 Replies)
Discussion started by: jiam912
14 Replies

3. Shell Programming and Scripting

Checking in a directory how many files are present and basing on that merge all the files

Hi, My requirement is,there is a directory location like: :camp/current/ In this location there can be different flat files that are generated in a single day with same header and the data will be different, differentiated by timestamp, so i need to verify how many files are generated... (10 Replies)
Discussion started by: srikanth_sagi
10 Replies

4. Shell Programming and Scripting

Count and merge using common column

I have the following records from multiple files. 415 A G 415 A G 415 A T 415 A . 415 A . 421 G A 421 G A,C 421 G A 421 G A 421 G A,C 421 G . 427 A C 427 A ... (3 Replies)
Discussion started by: empyrean
3 Replies

5. UNIX for Dummies Questions & Answers

How to count specific columns and merge with unique ones?

Hi. I am not sure the title gives an optimal description of what I want to do. I have several text files that contain data in many columns. All the files are organized the same way, but the data in the columns might differ. I want to count the number of times data occur in specific columns,... (0 Replies)
Discussion started by: JamesT
0 Replies

6. UNIX for Dummies Questions & Answers

Merge files and add file name to the end of each line

Hello everybody, I'm trying to merge a lot of files, but I want to include the filename to the end of each line. I've tried to use cat, but I got stuck. My files are for example: file01.001 123456 aaa ddd ee 458741 eee fff ee file02.003 478596 uuu ddd ee 145269 ttt fff ee ... (4 Replies)
Discussion started by: ernesto561
4 Replies

7. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies

8. Shell Programming and Scripting

How to merge and add?? Two Files

Sorry noob here...... I have 2 files like this: File A 8080000001 400 8080000002 300 8080000003 200 File B 8080000001 600 8080000002 200 8080000004 200 I want one file where File OUT 8080000001 1000 (6 Replies)
Discussion started by: Unice
6 Replies

9. Shell Programming and Scripting

MERGE 13 files and add the file name at the end of each record

Hi Gurus, I have 13 comma(,) seperated files that i have to merge and create a single file which has file name attached at th end of each record in the out put file. Can any one please help me with writing a unix script with this issue? test1.dat BIG ID,Local ID,Bond... (2 Replies)
Discussion started by: vkr
2 Replies
Login or Register to Ask a Question