Merging Files in UNIX shell script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merging Files in UNIX shell script
# 1  
Old 02-12-2013
Merging Files in UNIX shell script

I have the urge to merge some files using unix shell script but I'm very new using this language and I haven't succeeded yet.

The requirement is to merge the header, body and footer into one file with the name "ANY-NAME" in below example. To identify which files should be merged, I have flagged them using a number, see below example

Code:
1-BODY-ANY-NAME.txt
1-FTR-ANY-NAME.txt
1-HDR-ANY-NAME.txt
2-BODY-ANY-NAME.txt
2-FTR-ANY-NAME.txt
2-HDR-ANY-NAME.txt
3-BODY-ANY-NAME.txt
3-FTR-ANY-NAME.txt
3-HDR-ANY-NAME.txt

The files starting with the same number should be merged into one file. See below the ouput for the script.

Code:
1-ANY-NAME.txt
2-ANY-NAME.txt
3-ANY-NAME.txt

Please let me know how can I achieve this

Last edited by Scrutinizer; 02-12-2013 at 07:43 PM.. Reason: code tags
# 2  
Old 02-12-2013
Code:
ls | awk -F- '{ print $1 }' | sort -u | while read X
do
      echo cat $X-HDR-* $X-BODY-* $X-FTR-*
done

See if this finds the files you wanted. It should print "1-HDR-whatever 1-BODY-whatever 1-FTR-whatever".

If it's assembling them in the right order, change that line to this one to do the job for real:

Code:
 cat $X-HDR-* $X-BODY-* $X-FTR-* > output-$X.txt

This User Gave Thanks to Corona688 For This Post:
# 3  
Old 02-12-2013
Code:
#!/bin/bash

for i in {1..3}
do
    cat ${i}-HDR-ANY-NAME.txt ${i}-BODY-ANY-NAME.txt ${i}-FTR-ANY-NAME.txt > ${i}-ANY-NAME.txt
done

This User Gave Thanks to Yoda For This Post:
# 4  
Old 02-12-2013
try also:
Code:
for section in HDR BODY FTR
do
   ls -1 *$section* | while read file
   do
      new_file="${file%-${section}*}"-"${file#*${section}-}"
      cat "$file" >> "$new_file"
   done
done

This User Gave Thanks to rdrtx1 For This Post:
# 5  
Old 02-12-2013
Is there a way to generate the file with part of the first file name?
i.e.

Code:
1-HDR-FLASH_1-1H1-1305_1_20120618_20120624.txt
1-BODY-1-1H1-1305-201225
1-FTR

The output name should be:
Code:
FLASH_1-1H1-1305_1_20120618_20120624.txt


Last edited by Scrutinizer; 02-12-2013 at 07:42 PM.. Reason: code tags
# 6  
Old 02-12-2013
Can you post the exact file naming convention of all 3 files (Header,Body,Footer)?
# 7  
Old 02-12-2013
try:
Code:
for hfile in *HDR*
do
   new_file=${hfile#*HDR-}
   n=${hfile%%HDR*}
   cat "${n}HDR"*  >  $new_file 2>/dev/null
   cat "${n}BODY"* >> $new_file 2>/dev/null
   cat "${n}FTR"*  >> $new_file 2>/dev/null
done

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help getting a UNIX shell script to look at multiple files.

Hey everyone! I made a shell script that would go through a file and replace any phrase or letter with another phrase or letter. Helps update variable names or values. The following code is this: #!/bin/sh Word1="$1" Replace1="$2" File1="$3" arg=$( echo "$Word1" | sed 's:\:\\&:g' )... (3 Replies)
Discussion started by: rebmonk
3 Replies

2. Shell Programming and Scripting

Merging two files in UNIX

Hi Experts, Need urgent solution for a problem. I have two files file1 and file2. file1 is tab separated and file2 is comma separated. I need to merge both the files into single file based on CUST_ID by retaining the headers of file1 Matching CUST_IDs should be placed one below the other in... (11 Replies)
Discussion started by: bharathbangalor
11 Replies

3. Shell Programming and Scripting

Merging Very large CSV files in Unix

Hi, I have two very large CSV files, which I want to merge (equi-join) based on a key (column). One of the file (say F1) would have ~30 MM records and 700 columns. The other file (~f2) would have same # of records and lesser columns (say 50). I want to create an output file joining on a... (3 Replies)
Discussion started by: student_007
3 Replies

4. Shell Programming and Scripting

Script for merging six files into one

Hi all Need little help from you my aim is like merging the output of 6 files (log_files) And merge it into the one master file tell me the script for this please thanks in advance Please reply in mail if possible <email removed> Regards: zimmy (1 Reply)
Discussion started by: zimmyyash
1 Replies

5. Shell Programming and Scripting

AWK Script For Merging Text Files

Hello, I am trying to merge data from two text files. One file (File1) contains a listing of data which includes the trial number in Column 5, while the other text file (File2) contains what category the trial belongs to. Here is a snippet of what File1 looks like. 1 Arrow_ST 9.738 0.905... (2 Replies)
Discussion started by: Jahn
2 Replies

6. Shell Programming and Scripting

Script for merging files

I am facing a strange situation where I need to collect logs and consolidate them. All logs are under say /tmp/useless and all logs are under subdir of the form IPaddresses_x_y eg: /tmp/useless/105.52.34.246_3_1 /tmp/useless/105.52.1.118_0_10 There are several logs under each... (2 Replies)
Discussion started by: zpn
2 Replies

7. Shell Programming and Scripting

Shell script for merging lines in a loop

Dear All, I need a script to merge lines of an input file in a loop, please guide me for the script or one liner(awk, sed, tr, shell, perl). I/P File------------------------- APaul,,,,SDH,,23,,,PPH,,2 ,,,,KKH,,19,,,MMH,,12, ,,,,CCH,,22,,,MNH,,19, ,,,,TCH,,55,,,NNH,,67,... (3 Replies)
Discussion started by: ashis.tewari
3 Replies

8. Shell Programming and Scripting

merging of 2 files AWK, SHELL or something else

I have 2 files pipe delimted and want to merge them based on a key e.g file 1 123|xxx|yyy|zzz 345|xab|yzy|zyz 456|sss|ttt|foo file 2 123|hhh|ggg|xxx 345|ddd|www|ddd|fff 456|ddd|sss|sss|eee so if the key is the first field, and the result should be file 1 with field 2 from file 2... (24 Replies)
Discussion started by: klut
24 Replies

9. UNIX for Dummies Questions & Answers

Merging 2 .CSV files in Unix

I need a little help as I am a complete novice at scripting in unix. However, i am posed with an issue...:eek: i have two csv files in the following format@ FILE1.CSV: HEADER HEADER Header , , HEADER 001X ,,200 002X ,,300 003X ... (6 Replies)
Discussion started by: chachabronson
6 Replies

10. Shell Programming and Scripting

How to delete files in UNIX using shell script

Hi, I have the following task to perform using shell script. The user will provide a directory name along with a date. The script will delete all the files in the specified directory that was created earlier to that date. Also it should display the number of files that has been deleted. ... (7 Replies)
Discussion started by: theguy16
7 Replies
Login or Register to Ask a Question