Shell Programming and Scripting

Hi,

My requirement is,there is a directory location like:

:camp/current/

In this location there can be different flat files that are generated in a single day with same header and the data will be different, differentiated by timestamp, so i need to verify how many files are generated in a single day and merge them all into a single file.

Note:header should come in the single file.

Thank you!

Pls be more specific, post an example of your constellation, where to find the header, and rules how to merge the files (e.g. specially ordered, by contents, no order).

Hi,

Thank you!

There is a location like:

scripts/new/projectdir/

in which there is a chance of generating a two or three flat files per day,
they will be with name as flatfile1_v_timestamp,flatfile2_v_timestamp.

I need to check for this pattern in this location for these files as my job runs once in a day i need to pick all these flat files combine them into a single file, header will be same for all these files.

Note: records should come in ordered way and the values should accurately go into particular columns, even if we merge the files into a single file, field separator is '|'

Example:

File1

Code:

header :col1_name1|col2_name2|col3_name3
              1              2                3
              1                                1

File2

Code:

header: col1_name1|col2_name2|col3_name3
               4              5               6
               1

mergefile:

Header:

Code:

col1_name1|col2_name2|col3_name3
                1                 2           3
                1                              1
                4                 5            6
                1

Header will be the same but values may come or may not come for some of the columns.

Please let me know if anything needed on this

Moderator's Comments:

Please use code tags next time for your code and data.

Pls use code tags as advised.
So - the headers are identical across all files, and all except first should be removed. Data should be ordered according to their files' creation time sequence. Try this:

Code:

awk 'NR=1 FNR>1' $(ls -rt flatfile1_v_*)

Remove the -r option to ls if you want newest files first.


Pls note: the field separator | that you mention I can see only in the headers given, not in the data lines.

sorry i forgot to mention, there is separator for the data as well.

---------- Post updated at 03:20 AM ---------- Previous update was at 03:14 AM ----------

can you explain me what this code does.

Code:

awk 'NR=1 FNR>1' $(ls -rt flatfile1_v_*)

where it is checking the directory, whether more than one file was present?

Code:

awk 'NR=1 FNR>1' $(ls -rt flatfile1_v_*)

when i use this command for merging the flat files into a single file header is coming two times in a single file, but i wanted it only one what i need to do here....

Example:
Flat file1:

Code:

column1|column2|column3
1|2|3
4||4

Flatfile2:

Code:

column1|column2|column3
4|5|6
1|1|

Mergedfile:

Code:

column1|column2|column3
1|2|3
4||4 
4|5|6
1|1|

In the merged file as well particular column value should go into particular column.

Please reply to me at the earliest..

Code:

awk 'NR==1 || FNR>1' flatfile1_v_*