Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Merge files according date

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merge files according date
# 1  
Old 05-21-2017
Merge files according date
Gents,
I have a big list of files and i would like to merge some of them using a select date

In my case i would like to merge all files which are bigger that 2017-05-15

List of files
Code:
SG-10_2017-05-10de.csv
SG-10_2017-05-11de.csv
SG-10_2017-05-12de.csv
SG-10_2017-05-13de.csv
SG-10_2017-05-14de.csv
SG-10_2017-05-15de.csv
SG-10_2017-05-16de.csv
SG-10_2017-05-17de.csv
SG-10_2017-05-18de.csv
SG-10_2017-05-19de.csv
SG-10_2017-05-20de.csv

I have try with this code
Code:
for i in {15..30}
do
cat "SG-10_2017-05-${i}de.csv" >> merged
done

the problem is that in the loop i have files that are not present.. Then, I will like to use the date 2017-05-15 as the first file to concatenate to the last file present.

Thanks for your help
# 2  
Old 05-21-2017
Quote:
Originally Posted by jiam912
Gents,
I have a big list of files and i would like to merge some of them using a select date

In my case i would like to merge all files which are bigger that 2017-05-15
[...]
What does it mean for you "bigger tha[n] 2017-05-15". Is 2017-08-10 a possibility?
In that particular example, what would be the ceiling or last file name? Any files you are not showing?
Last edited by Aia; 05-21-2017 at 06:13 PM..
This User Gave Thanks to Aia For This Post:
jiam912
# 3  
Old 05-21-2017
Hello jiam912,

Considering that the day your file gets created has the same date in it's name, if this condition is TRUE then could you please try following and let me know if this helps you.
Code:
find -type f -mtime -$((($(date +%s) - $(date --date "2017-05-15" +%s))/86400)) -exec cat >> merged {} \+

So explanation in short of above code(as it is too late in IST now, will add full explanation in morning) is getting the days difference and searching only those files which are older than(or same day 2017-05-15) and concatenating their values then into merged file name.

I hope this helps you, let me know if you have any queries on same.

NOTE: If your files are always .csv then you could add a condition like -name *.csv into above command and could test with it too.
Code:
find -type f -name "*.csv" -mtime -$((($(date +%s) - $(date --date "2017-05-15" +%s))/86400)) -exec cat >> merged {} \+

Thanks,
R. Singh
Last edited by RavinderSingh13; 05-21-2017 at 06:17 PM..
This User Gave Thanks to RavinderSingh13 For This Post:
jiam912
# 4  
Old 05-21-2017
Quote:
Originally Posted by RavinderSingh13
Hello jiam912,

Considering that the day your file gets created has the same date in it's name, if this condition is TRUE then could you please try following and let me know if this helps you.
Code:
find -type f -mtime -$((($(date +%s) - $(date --date "2017-05-15" +%s))/86400)) -exec cat >> merged {} \+

So explanation in short of above code(as it is too late in IST now, will add full explanation in morning) is getting the days difference and searching only those files which are older than(or same day 2017-05-15) and concatenating their values then into merged file name.

I hope this helps you, let me know if you have any queries on same.

NOTE: If your files are always .csv then you could add a condition like -name *.csv into above command and could test with it too.

Thanks,
R. Singh
Hi Singh,

Any timestamp is not guarantee to have a correlation with the name of the file.
This User Gave Thanks to Aia For This Post:
jiam912
# 5  
Old 05-22-2017
Dear RaviderSingh13 and Aia.

Mr Alia is correct I would like to merge the files using the name file not the timestamp

The purpose is to select all files with name more than 2017-05-15 and merge all of them.

Input files
Code:
SG-10_2017-05-10de.csv
SG-10_2017-05-11de.csv
SG-10_2017-05-12de.csv
SG-10_2017-05-13de.csv
SG-10_2017-05-14de.csv
SG-10_2017-05-15de.csv
SG-10_2017-05-16de.csv
SG-10_2017-05-17de.csv
SG-10_2017-05-18de.csv
SG-10_2017-05-19de.csv
SG-10_2017-05-20de.csv

Merged files
Code:
SG-10_2017-05-15de.csv
SG-10_2017-05-16de.csv
SG-10_2017-05-17de.csv
SG-10_2017-05-18de.csv
SG-10_2017-05-19de.csv
SG-10_2017-05-20de.csv

Appreciate your help
# 6  
Old 05-22-2017
Please, try the following and see if produces the result you want.
Code:
cat SG-10_2017-05-1[6-9]de.csv SG-10_2017-05-[2-9][0-9]de.csv >> merge

or if you know before hand you want up to SG-10_2017-05-20de.csv
Code:
cat SG-10_2017-05-1[6-9]de.csv SG-10_2017-05-20de.csv >> merge

Last edited by Aia; 05-22-2017 at 11:04 AM.. Reason: icode for file *-20dev.csv
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to list files that are not first two files date & last file date for every month-year?

Hi All, I need to find all files other than first two files dates & last file date for month and month/year wise list. lets say there are following files in directory Mar 19 2012 c.txt Mar 19 2012 cc.txt Mar 21 2012 d.txt Mar 22 2012 f.txt Mar 24 2012 h.txt Mar 25 2012 w.txt Feb 12... (2 Replies)
Discussion started by: Makarand Dodmis
2 Replies

2. Shell Programming and Scripting

Merge files and generate a resume in two files

Dear Gents, Please I need your help... I need small script :) to do the following. I have a thousand of files in a folder produced daily. I need first to merge all files called. txt (0009.txt, 0010.txt, 0011.txt) and and to output a resume of all information on 2 separate files in csv... (14 Replies)
Discussion started by: jiam912
14 Replies

3. Shell Programming and Scripting

Merge files

a.txt id name subject 12 aaa History 23 bbb Science 45 ccc Zoology b.txt id layer LayerNo 12 xxx12 1 23 yyy23 2 23 lll23 3 45 xxx45 1 45 yyy45 2 45 lll45 3 i have file a.txt which is parent file and another children file b.txt . Both files are linked together by common field... (2 Replies)
Discussion started by: manas_ranjan
2 Replies

4. Shell Programming and Scripting

Checking in a directory how many files are present and basing on that merge all the files

Hi, My requirement is,there is a directory location like: :camp/current/ In this location there can be different flat files that are generated in a single day with same header and the data will be different, differentiated by timestamp, so i need to verify how many files are generated... (10 Replies)
Discussion started by: srikanth_sagi
10 Replies

5. Shell Programming and Scripting

merge files

hi, i've two files whcih i need to merge in below format. File1- DEFINE lc_driver CHAR(3) INITIALIZE gr_XXX_MAIN_TABLE_XXX.* TO NULL { EXECUTE p_XXX_MAIN_TABLE_XXX_ins USING gr_XXX_MAIN_TABLE_XXX.* } File2 - # field1 LET table1.field1= (8 Replies)
Discussion started by: dvah
8 Replies

6. Shell Programming and Scripting

MERGE FILES

Hi all! How could I merge all the text files (in format xml) of a single folder, after having deleted from each of them all the text from its beginning up to a specific string: "<body>" ? Thanks a lot! mjomba (4 Replies)
Discussion started by: mjomba
4 Replies

7. Shell Programming and Scripting

Merge 2 files

Hello, i'd like a bash script to merge 2 files without duplicate lines. Example : file1 : toto titi file2 : toto tata Expected result, file3 : toto (5 Replies)
Discussion started by: Celmar
5 Replies

8. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies

9. Shell Programming and Scripting

Sorting Files by date and moving files in date order

I need to build a k shell script that will sort files in a directory where files appear like this "XXXX_2008021213.DAT. I need to sort by date in the filename and then move files by individual date to a working folder. concatenate the files in the working folder then start a process once... (2 Replies)
Discussion started by: rebel64
2 Replies

10. Shell Programming and Scripting

help in merge files

I have created these files in a particular directory a_1.txt a_2.txt ... .. a_n.txt Each file has a single line. I want to write a output file a.txt, which will have concantated value of string from all the files. which utility should i use. copy, cat or paste???? Can anyone help... (5 Replies)
Discussion started by: u263066
5 Replies
Login or Register to Ask a Question