Combining files(every 15 min) as one file(hourly)


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Combining files(every 15 min) as one file(hourly)
# 1  
Old 09-25-2013
Combining files(every 15 min) as one file(hourly)

Hello,

My system is generating two files every 15 minutes and file names are given automatically as below. (98,99,89,90 are the sequence numbers)
Code:
File1_09242013131016_000000098
File1_09242013131516_000000099
File2_09242013124212_000000089
File2_09242013124712_000000090

I want to combine these files and generate only one file in every one hour as below.
Code:
File3_09242013120000_00000001
File3_09242013130000_00000002

How can i achieve this?

Last edited by Franklin52; 09-25-2013 at 03:45 AM.. Reason: Please use code tags
# 2  
Old 09-25-2013
Try this... Test it first and then try on actual files...
Code:
ls | awk '{x=$0;sub(/...._[0-9]+$/,"",x); if(!f[x]){f[x]=x"0000_"sprintf("%08d", ++seq)} print "cat "$0">>"f[x]}' | bash

This will not delete the files, if you do want to delete it, append the rm command to the print statement.

--ahamed
# 3  
Old 09-25-2013
Thanks Ahamed,

That script didnt combine the files. This created seperate files by hourly based.
I prefer to combine "15 minutes generated two different files" "hourly in one file"
# 4  
Old 09-25-2013
You wanted File1_09242013131016_000000098 File1_09242013131516_000000099 to be combined to File3_09242013130000_00000002 and so on right?

--ahamed
# 5  
Old 09-25-2013
There are two types of files named File1 and File2. File1 and File2 are being generated every 15 minutes interval.

Hourly, I want to combine them and generate only one file like named File3.
# 6  
Old 09-25-2013
Just adding my 2 cents here.
Say you have the files a1.txt, b1.txt, c1.txt, d1.txt.
How about you try writing a script that performs the following operations:
1) Create a new file, say 1 and append the contents of each file into it:
Code:
cat a1 b1 c1 d1 > 1

2) Leave the 1 file intact and delete the other files:
Code:
rm *.txt

(or you can move them to a different directory)
Repeat the process for as long as you need, creating files with .txt extension every 15 minutes, then combining them into a file without extension, and then deleting the .txt files.
Hope it helps.
# 7  
Old 09-25-2013
Try
Code:
for i in 1 2 ; do for j in File$i*; do TGT=${j%????_*}0000_0000000$i; TGT=${TGT/[12]/3}; done; cat File$i* > $TGT; done

If files come in regularly, and you run this regularly, you'll need to either remove the quarter hourly files after catting, or make the j for loop somewhat more specific.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Combining certain columns of multiple files into one file

Hello Unix gurus, I have a large number of files (say X) each containing two columns of data and the same number of rows. I would like to combine these files to create a unique merged file containing X columns corresponding to the second column of each file (with a bonus of having the first... (3 Replies)
Discussion started by: ksennin
3 Replies

2. Shell Programming and Scripting

Combining columns from multiple files into one single output file

Hi, I have 3 files with one column value as shown File: a.txt ------------ Data_a1 Data_a2 File2: b.txt ------------ Data_b1 Data_b2 Data_b3 Data_b4 File3: c.txt ------------ Data_c1 Data_c2 Data_c3 Data_c4 Data_c5 (6 Replies)
Discussion started by: vfrg
6 Replies

3. Shell Programming and Scripting

Combining multiple column files into one with file name as first row

Hello All, I have several column files like this $cat a_b_s1.xls 1wert 2tg 3asd 4asdf 5asdf $cat c_d_s2.xls 1wert 2tg 3asd 4asdf 5asdf desired put put $cat combined.txt s1 s2 (2 Replies)
Discussion started by: avatar_007
2 Replies

4. Shell Programming and Scripting

Combining columns from multiple files to one file

I'm trying to combine colums from multiple file to a single file but having some issues, appreciate your help. The filenames are the same except for the extension, path1.m0 --------- a b c d e f g h i path1.m1 --------- m n o p q r s t u File names are path1.m The... (3 Replies)
Discussion started by: rkmca
3 Replies

5. UNIX for Dummies Questions & Answers

Assistance with combining, sorting and saving multi files into one new file

Good morning. I have a piece of code that is currently taking multiple files and using the CAT.exe command to combine into one file that is then sorted in reverse order based on the 3rd field of the file, then displayed on screen. I am trying to change this so that the files are being combined into... (4 Replies)
Discussion started by: jaacmmason
4 Replies

6. Shell Programming and Scripting

how to grep string from hourly basis files

dear all, pls help on this script.. i have many files which will be created every mins in particular directory. i want to grep a particular string from only for unique hour files. from the below code i want to grep a string from only 9th hour files . Ex files: -rw-r--r-- 1 root ... (5 Replies)
Discussion started by: steve2216
5 Replies

7. UNIX for Dummies Questions & Answers

Combining lines of files to new file

Hi everybody. I have a number of files that i would like to combine. however not concatenating, but rather extract lines from the files. Example: File1 ------ File2 ------File3 ... line11 ---- line21 ---- line31 ... line12 ---- line22 ---- line32 ... line13 ... (3 Replies)
Discussion started by: kabbo
3 Replies

8. Shell Programming and Scripting

Shell script to check the files hourly and purge

I'm new to shell scripting... i have been given a task.. can any one help in this regard.... 1) Check hourly for files in <destination-path><destination-file-template><destination-file-suffix> for files older than <destination-file-retention> days and purge. It should then check... (1 Reply)
Discussion started by: satishpabba
1 Replies

9. Shell Programming and Scripting

combining 2 files with more than one match in second file

Hello, I am attempting to combine two files where the second file can have more than one match with the lookup field (node) in the first file, onto one line of the output file. Also alerting if a lookup was not found in file2 =-=-=-=-=-=-= Example of file1 node,type =-=-=-=-=-=-= bob,232... (5 Replies)
Discussion started by: johnes42
5 Replies
Login or Register to Ask a Question