Concatenate many files which contents the same date as part of name file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Concatenate many files which contents the same date as part of name file
# 1  
Old 11-22-2012
Concatenate many files which contents the same date as part of name file

Gents,

I have lot of files in a folder where each file name includes the date of generation, then I would like to merge
all the files for each date in a complete file.

list of files in forder.

Code:
dsd01_121104.txt
dsd01_121105.txt
dsd01_121106.txt
dsd03_121104.txt
dsd03_121105.txt
dsd03_121106.txt
dsd04_121104.txt
dsd04_121105.txt
dsd04_121106.txt
dsd05_121104.txt
dsd05_121105.txt
dsd05_121106.txt
dsd06_121104.txt
dsd06_121105.txt
dsd06_121106.txt
dsd07_121104.txt
dsd07_121105.txt
dsd07_121106.txt
dsd08_121104.txt
dsd08_121105.txt
dsd08_121106.txt
dsd09_121104.txt
dsd09_121105.txt
dsd09_121106.txt
dsd10_121104.txt
dsd10_121105.txt
dsd10_121106.txt
dsd11_121104.txt
dsd11_121105.txt
dsd11_121106.txt
dsd12_121104.txt
dsd12_121105.txt
dsd12_121106.txt

output desire files concatenated.

Code:
dsd_121104.txt " all files for day 121104 concatenated "
dsd_121105.txt " all files for day 121105 concatenated "
dsd_121106.txt " all files for day 121106 concatenated "

Also, Please how to make a script ,, to make a copy for all the files from one date to a new folder,
I mean to create a new folders.. example folder dsd_121104..
and copy files for day 121104. and the same for the rest of days...

new folders

Code:
dsd_121104 "inside all files for day 121104"
dsd_121105 "inside all files for day 121105"
dsd_121106 "inside all files for day 121106"

thanks in advance.
# 2  
Old 11-22-2012
Code:
#!/bin/ksh

ls dsd*.txt | awk -F_ ' { print substr($2,1,6); } ' | sort | uniq > dates.list

while read file_dt
do
        mkdir -p dsd_${file_dt}
        for file in dsd*${file_dt}.txt
        do
                cat ${file} >> dsd_${file_dt}/dsd_${file_dt}.txt
        done
done < dates.list

exit 0

# 3  
Old 11-22-2012
If you're running cat 9 times to read 9 files, you don't know what cat's for. You also don't need ls's help to use *. And don't need to use a temp file.


Code:
#!/bin/ksh

printf "%s\n" dsd*.txt | awk -F_ ' { print substr($2,1,6); }'  | sort | uniq |
while read file_dt
do
        mkdir -p dsd_${file_dt}
        cat dsd*${file_dt}.txt >> dsd_${file_dt}/dsd_${file_dt}.txt
done

exit 0

[edit] Slight changes to how the filenames are fed to awk.

Last edited by Corona688; 11-22-2012 at 02:16 PM..
These 2 Users Gave Thanks to Corona688 For This Post:
# 4  
Old 11-22-2012
Corona688, I think ls is required because we don't want awk to read the file, but to list the files and get sub string of file names. Am I correct?

EDIT: never mind saw your modification. Thank you.
# 5  
Old 11-22-2012
I just noted and fixed the difference myself, but used the printf shell builtin instead of the ls external utility.
# 6  
Old 11-23-2012
Thanks Corona688 and Image bipinajith

I notice that the concatenated file is saved in the created folder.. The main point is to make a copy of all the files for the same date in a new folder... And the concatenated file should be saved in the original folder where are all the files.. Thanks for your help

---------- Post updated 11-23-12 at 05:46 AM ---------- Previous update was 11-22-12 at 01:51 PM ----------

Thanks guys

I have done a little change in your script to like that,, and now is working as I want. thanks again.

Code:
printf "%s\n" dsd*.txt | awk -F_ ' { print substr($2,1,6); }' | sort | uniq |
while read file_dt
do
        
mkdir -p folder/dsd_${file_dt}
       cp dsd*${file_dt}.txt folder/dsd_${file_dt}
      
      cat dsd*${file_dt}.txt >> folder/dsd_${file_dt}.txt

done

exit 0

# 7  
Old 11-23-2012
A possible alternative:

Code:
printf "%s\n" /tmp/dsd*.txt |
awk -F"dsd" '{FS="_";$0=$0;a[$2]++}END{for (i in a) {cmd="cat /tmp/dsd*_"i" > /tmp/dsd_"i"";system(cmd)}}'

This User Gave Thanks to ripat For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Linux

Filter log file contents between date

Hi, Could you please provide me command to filter contents between date in a log file? Say for example, in a log file I want to capture contents between date May 01 from 5am to 9 am. OS -- Linux Regards, Maddy (1 Reply)
Discussion started by: Maddy123
1 Replies

2. Shell Programming and Scripting

How to remove the date part of the file?

Hi Gurus, I have file name like: abcd.20131005.dat I need to remove mid part of data final name should be abcd.dat thanks in advance (2 Replies)
Discussion started by: ken6503
2 Replies

3. Shell Programming and Scripting

How to concatenate the files based upon the file name?

Hi Experts, I am trying to merge multiple files into one file based upon the file name. Testreport_Server1.txt ============ MonitoringReport_Server1.txt============ CentralReport_Server1 Here two files containing server1 should be merged into one file? How can i do... (16 Replies)
Discussion started by: sharsour
16 Replies

4. Shell Programming and Scripting

Getting the date part from the log file.

Hi, I have a script that runs to generate a log file which is in this format: Start: Friday, April 29, 2011 18:30 User : Host : Database : I need to write a script that reads this log file (RUN.LOG) and rename the existing log file... (7 Replies)
Discussion started by: ashok@119
7 Replies

5. Shell Programming and Scripting

Fetch contents of file greater than date

I have an input file which looks like something mentioned below: ====== 25/07/2011 05:06:59 : test ====== 25/07/2011 05:06:59 : test1 ====== 25/07/2011 05:08:09 : test2 ====== I need to write a script which will take date in the format "25/07/2011 05:07:01" as input and compare it... (1 Reply)
Discussion started by: bhallarandeep
1 Replies

6. Shell Programming and Scripting

Concatenate files to one file with naming convention

Hi , i have below files in wrk folder. file names are 1102090001.CLT 1102090003.CLT 1102100019.CLT 1102100020.CLT the above files are concatenate to one file but that concatenate file name must be same naming convention. (date +%y%m%d)and 0001 count. example : concatenate file... (9 Replies)
Discussion started by: krbala1985
9 Replies

7. Shell Programming and Scripting

concatenate files sorted by date

I am a beginner in script writing, i tried to do the following I have a set of files sorted by date in the format YYMMDD.s and .x and .r I need to concatenate a header file to these sets of files so I used the following code echo "enter Swath number" read s echo "please enter first date and... (2 Replies)
Discussion started by: docaia
2 Replies

8. HP-UX

How do I take out(remove) the date part in the file name?

Hi Guys here I am again, I have two files in a specified location. Location ex: /opt/xdm/input/ input file names: 1. abc_app.yyyymmdd.dtd 2. abd_app.yyyymmdd.dtd I need to build a code that reads the files from the location based on the oldest date and cuts the date part... (5 Replies)
Discussion started by: ruthless
5 Replies

9. Shell Programming and Scripting

Add Date as part of file name

How can I add date in format yyyymmdd to my existing file name ? For example my file name is test.dat then i want it to be test.datyyyymmdd (3 Replies)
Discussion started by: amsh76
3 Replies

10. UNIX for Dummies Questions & Answers

Concatenate date to file name

Hi, I have written a script to rename a file, but I can not add the current date (in YYYYMMDD-HHMM format) can you please look at this cript, and help? thanks, #!/usr/local/bin/tcsh -f set server = "$1" set user = "$2" if (-f $server) then \mv $server $server.Saar. endif (4 Replies)
Discussion started by: sierra_aar
4 Replies
Login or Register to Ask a Question