Combine multiple Files into one big file

Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
Homework and Emergencies Emergency UNIX and Linux Support Combine multiple Files into one big file
# 1  
Power Combine multiple Files into one big file

Hi Ppl,

I have a requirement like i will be getting files of huge size daily and if the file size is so huge ,the files will be split into many parts and sent.The first file will have the header details followed by detail records and the consecutive files will have detail records and the last file will have the trailer details.I need to merge all these files into one single file in the order Header record then detail records then trailer record.

Example,
Filename pattern is Daily_report_1111_20111115 & Daily_report_2222_20111115.
If the file size is huge it will be split and sent as

Daily_report_1111_20111115_1
Daily_report_1111_20111115_2
Daily_report_1111_20111115_3
Daily_report_1111_20111115_4

Daily_report_2222_20111115_1
Daily_report_2222_20111115_2
Daily_report_2222_20111115_3
Daily_report_2222_20111115_4.

I need to concatenate all the Daily_report_1111_20111115_1,2,3,4 files into one big file Daily_report_1111_20111115 with contents of Daily_report_1111_20111115_1 in the top followed by the contents of the files Daily_report_1111_20111115_2,3,4.So the final output file should be Daily_report_2222_20111115 with contents of all 4 files in the order received(the order of the files will be present in the filename like _ 1,2,3,4)

Similarly for Daily_report_2222_20111115 files.

The requirement is the date 20111115(which is in the filename) will keep changing and we might get multiple days files on the same day.so i need to write and automate the script so that it looks out for the files with the same date and check the filename pattern(either Daily_report_1111_YYYYMMDD or Daily_report_2222_YYYYMMDD and concatenate them into one single file based on the filenames like _1,2,3 etc.

Please help me out!
# 2  
Code:
ls |
        awk -F_ '{ STR=$1; for(N=2; N<NF; N++) STR=STR"_"$N;  T[STR]=1 } END { for(K in T) print K; }' |
        while read LINE
        do
                cat ${LINE}_* > $LINE
        done

The awk groups the filenames, feeding them into a while loop which globs them and sticks them together into the output filename.
These 2 Users Gave Thanks to Corona688 For This Post:
# 3  
Hi Corona,

I am sorry but i do not understand the code.Can you explain me each part of the code as i am new to unix.Thanks!
# 4  
Code:
# List files in the current directory
ls |
# Feed it into the AWK programming language.
# splits each line apart $1,$2,...$NF on the _ character.
# Then puts it back together into STR, minus the _1,_2, etc on the end.
# Then stores it in T.
# Once it's read all lines, prints all unique combinations it found.
        awk -F_ '{ STR=$1; for(N=2; N<NF; N++) STR=STR"_"$N;  T[STR]=1 } END { for(K in T) print K; }' |
# Reads unique combinations from awk, like Daily_report_2222_20111115
# and puts them in the variable LINE.
# feed all files matching ${LINE}_* into cat, writing to the file $LINE.
        while read LINE
        do
                cat ${LINE}_* > $LINE
        done

These 2 Users Gave Thanks to Corona688 For This Post:
# 5  
Hi Corona,

There might be multiple files of other filename patter and i do not want to touch those.I need to combine only the file names which are of the pattern Daily_report_NNNN_YYYYMMDD in that particular folder.Not all days we might get a single big file split into multiple parts and sent.So i will have to write script and automate it so that if at all i get a large file split and sent into multiple parts i need to concatenate them into one and if its sent as one big file without being split into multiple parts then i need to use the one big file as is for processing.Thanks for your understanding
# 6  
Okay.

Code:
# List relevant files in the current directory
ls Daily_report_[0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_[0-9] |
# Feed it into the AWK programming language.
# splits each line apart $1,$2,...$NF on the _ character.
# Then puts it back together into STR, minus the _1,_2, etc on the end.
# Then stores it in T.
# Once it's read all lines, prints all unique combinations it found.
        awk -F_ '{ STR=$1; for(N=2; N<NF; N++) STR=STR"_"$N;  T[STR]=1 } END { for(K in T) print K; }' |
# Reads unique combinations from awk, like Daily_report_2222_20111115
# and puts them in the variable LINE.
# feed all files matching ${LINE}_* into cat, writing to the file $LINE.
        while read LINE
        do
                cat ${LINE}_* > $LINE
        done

This User Gave Thanks to Corona688 For This Post:
# 7  
Hi Corona,

thanks a lot for your time and patience in explaining me things.
Il try out the script and come back to you if i have any concerns.
Thanks once again Corona!!!SmilieSmilie

---------- Post updated at 12:32 AM ---------- Previous update was at 12:16 AM ----------

Hi Corona,

Your code worked like a charm.thanks for that.
Also i need to remove the old files after merging the multiple files into one single file.I mean for files that have come as one big file without getting split into multiple parts,i need to keep them untouched while the other big files which had come as multiple parts ,i need to delete these files after merging them into one single big file and i need to keep only the final big file for further processing and i need to delete the other files that came as parts.!
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #730
Difficulty: Medium
The first version BASIC language was released on 1 May 1964.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Combine and complete multiple CSV files based on 1 parameter

I have to create a new CSV file based on the value listed on the 3rd column from different CSV files. This is what I need: 1. I should substitute the first column from each file, excluding the headers, with the file name InputXX. 2. Then, I need to look for rows with 0 on the third column in... (7 Replies)
Discussion started by: Xterra
7 Replies

2. Shell Programming and Scripting

Split a big file into multiple files using awk

this thread is a continuation from previous thread https://www.unix.com/shell-programming-and-scripting/223901-split-big-file-into-multiple-files-based-first-four-characters.html ..I am using awk to split file and I have a syntax error while executing the below code I am using AIX 7.2... (4 Replies)
Discussion started by: etldev
4 Replies

3. Shell Programming and Scripting

Compare multiple files, identify common records and combine unique values into one file

Good morning all, I have a problem that is one step beyond a standard awk compare. I would like to compare three files which have several thousand records against a fourth file. All of them have a value in each row that is identical, and one value in each of those rows which may be duplicated... (1 Reply)
Discussion started by: nashton
1 Replies

4. Shell Programming and Scripting

Split a big file into multiple files based on first four characters

I have a requirement to split a huge file to smaller text files based on first four characters which look like ABCD 1234 DFGH RREX : : : : : 0000 Each of these records are OF EQUAL bytes with a different internal layout based on the above first digit identifier.. Any help to start... (5 Replies)
Discussion started by: etldev
5 Replies

5. Shell Programming and Scripting

Combine Multiple Files into Single One File One after other

I am trying to combine 4 .dat files into one single Output file Inputs are:- file123.dat, file256.dat, file378.dat & file490 Expected Output:- FileName=file1 {text from file1} EOF {blank line} FileName=file2 {text from file2} EOF {blank line} FileName=file3 {text from file3} EOF... (4 Replies)
Discussion started by: lancesunny
4 Replies

6. Shell Programming and Scripting

Combine columns from multiple files

Can anybody help on the script to combine/concatenate columns from multiple files input1 4 135 5 185 6 85 11 30 16 72 17 30 21 52 22 76 input2 2 50 4 50 6 33 8 62 10 25 12 46 14 42 15 46output (2 Replies)
Discussion started by: sdf
2 Replies

7. Shell Programming and Scripting

combine multiple files by column into one files already sorted!

I have multiple files; each file contains a certain data in a column view simply i want to combine all those files into one file in columns example file1: a b c d file 2: 1 2 3 4 file 3: G (4 Replies)
Discussion started by: ahmedamro
4 Replies

8. UNIX for Dummies Questions & Answers

Combine multiple files with common string into one new file.

I need to compile a large amount of data with a common string from individual text files throughout many directories. An example data file is below. I want to search for the following string, "cc_sectors_1" and combine all the data from each file which contains this string, into one new... (2 Replies)
Discussion started by: GradStudent2010
2 Replies

9. Shell Programming and Scripting

Combine multiple columns from multiple files

Hi there, I was wondering if someone can help me with this. I am trying the combine multiple columns from multiple files into one file. Example file 1: c0t0d0 c0t2d0 # hostname vgname c0t0d1 c0t2d1 # hostname vgname c0t0d2 c0t2d2 # hostname vgname c0t1d0 c0t3d0 # hostname vgname1... (5 Replies)
Discussion started by: martva
5 Replies

10. Shell Programming and Scripting

Help Needed : Split one big file to multiple files

Hi friends, I have data in flat file as following, first filed is the customer number. We have almost 50-100 customers in the system 100 ABC A123 100 BVC D234 100 BNC N324 200 CBC A122 200 AVC D294 200 HNC N324 300 GBC A173 300 FVC D234 300 DNC N344 I want to split the file and... (5 Replies)
Discussion started by: monicasgupta
5 Replies