Merge all the files in a folder in FIFO order


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merge all the files in a folder in FIFO order
# 8  
Old 06-14-2012
Thanks a lot guys..but frankly speaking I did not get what will be the right code to use :-(..I am new to UNIX and this is part of my first assignment in UNIX..and please let me know how to append the timestamp of the file at the end of each record while merging the all files..Thanks again.

---------- Post updated at 09:45 AM ---------- Previous update was at 09:34 AM ----------

I am using the code:
Code:
ls -1rt | while IFS= read -r f; do [[ -f $item ]] && cat $item >> /work/scripts/acu/outputfile.txt ; done

but it is not working

---------- Post updated at 10:33 AM ---------- Previous update was at 09:45 AM ----------

I have used:
Code:
for f in $(ls -tr); do cat "$f" >> /work/scripts/acu/my_accumulated_data_file; done

This is merging all the files in order to timestamp of the file but sometime the first line in the new file is starting from the new line and sometime from the end of the last line of the previous file

I have tried this as well:
Code:
ls | while IFS= read -r f; do cat "$f" >> /work/scripts/acu/outputfile.txt ; done

The new line issue is here also and it is not following any order in mergeing the file it seems.
Please advise as it is urgent. And how to append the timestamp (the timestamp when the file placed in the folder) at the end of the each record of that perticular file.

Thanks

Last edited by Franklin52; 06-15-2012 at 04:31 AM.. Reason: Please use code tags for data and code samples
# 9  
Old 06-14-2012
Code:
ls -1rt | while IFS= read -r item; do [[ -f $item ]] && { cat $item; stat -c %y $item; } >> /work/scripts/acu/outputfile.txt; done

# 10  
Old 06-14-2012
Hi Lem,

i am getting error that "stat: not found." for all the files in the folder.
The first line of all the files are not starting from the new line as well.
# 11  
Old 06-14-2012
It probably means what it says. It checks in the current folder for a file named "a.txt" or whatever, and it's not there.

I'm guessing you're not listing the current, hence why it can't find those files -- it's looking in the wrong place.

Prepend the path that ls removed.

Code:
ls -1rt folder | while IFS= read -r item; do [[ -f $item ]] && { cat folder/$item; stat -c %y folder/$item; } >> /work/scripts/acu/outputfile.txt; done

# 12  
Old 06-14-2012
ohh..is it for .txt only??can't we do it for any kind of extension?(.dat,.trg etc)..and what about the new line :-(
# 13  
Old 06-14-2012
Why on earth would you want the file contents dumped into the logfile if they weren't text? Smilie You won't be able to read them...

What about the newline? I don't understand the problem you're having. Show an example of the bad output.
# 14  
Old 06-14-2012
ok..I should have explained the scenario at the very beginning itself.
Different users can put some information in pre-defined format (180 bytes) in a file which may end in .dat or .trg or .txt. We have to merge all this data in one file and then load them in teradata.(to know what record came when we want to append the file timestamp at the end of each record which will be then inserted to teradata).
If this code will work for .txt only we can ask users to put the files in .txt format only but it will be better if there will be no bar on extension.
The new line character issue is: After merge,the first record of the new file is starting from the end of the last record of the earlier file. Can't we have all the records starting from the new line?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Request for Shell script to move files from Subfolder to Parent folder and delete sub folder

Hi Team, I am new to shell script and there is a requirement where files should be moved from Subfolder to parent folder. Eg: parent folder --> /Interface/data/test/IN Sub folder -->/Interface/data/test/IN/Invoice20180607233338 Subfolder will be always with timestamp... (6 Replies)
Discussion started by: srivarun15
6 Replies

2. Shell Programming and Scripting

Shell scripting for moving folder specific files into target directory of that country folder.

I need help to write shell script to copy files from one server to another server. Source Directory UAE(inside i have another folder Misc with files inside UAE folder).I have to copy this to another server UAE folder( Files should be copied to UAE folder and Misc files should be copied in target... (3 Replies)
Discussion started by: naresh2389
3 Replies

3. Shell Programming and Scripting

Read input files and merge them in given order and write them to input one param or one file

Dear Friends, I am looking for a shell script to merge input files into one file .. here is my idea: 1st paramter would be outfile file (all input files content) read all input files and merge them to input param 1 ex: if I pass 6 file names to the script then 1st file name as output file... (4 Replies)
Discussion started by: hyd1234
4 Replies

4. UNIX for Dummies Questions & Answers

Folder Merge

Hi, I had a folder ABC . Was trying to move it . But the command was randomly stopped by mistake. Now , I have folder ABC in 2 location with different half data at each path . How do I merge both the folders and sub folders into a single one ? Any help is appreciated. (2 Replies)
Discussion started by: Abhayman
2 Replies

5. UNIX for Dummies Questions & Answers

Merge all csv files in one folder considering only 1 header row and ignoring header of all others

Friends, I need help with the following in UNIX. Merge all csv files in one folder considering only 1 header row and ignoring header of all other files. FYI - All files are in same format and contains same headers. Thank you (4 Replies)
Discussion started by: Shiny_Roy
4 Replies

6. Shell Programming and Scripting

check how many files in folder or total files in folder

Hi all, I have copied all my files to one folder.and i want to check how many files (count) in the folder recently moved or total files in my folder? please send me the query asap. (3 Replies)
Discussion started by: durgaprasad
3 Replies

7. Solaris

Program execution order like FIFO queue

Hi all: I have a problem with a C++ service runing on solaris 10. This service only set a signal on oracle table. When the service detect a cut off on the line (tcp/ip), trigger a cobol program for set the signal OFF. When the line is on again, the service trigger the same cobol program for set... (0 Replies)
Discussion started by: hcastellani
0 Replies

8. UNIX for Advanced & Expert Users

merge two files in ascending order

Hello Friends, I want to merge two files in ascending order on the first field. And if the first field matches sort on 3rd field i.e, TXADDR should come ahead of RXADDR . file1 9 : TXADDR : 00000000 65 : TXDATA 0000000000000011 83 : TXDATA 0000000000000012 453 :... (10 Replies)
Discussion started by: user_prady
10 Replies

9. UNIX for Advanced & Expert Users

Auto copy for files from folder to folder upon instant writing

Hello all, I'm trying to accomplish that if a file gets written to folder /path/to/a/ it gets automatically copied into /path/to/b/ the moment its get written. I thought of writing a shell script and cron it that every X amount of minutes it copies these files over but this will not help me... (2 Replies)
Discussion started by: Bashar
2 Replies

10. UNIX for Dummies Questions & Answers

need to find fifo files

I need to be able to recursively find fifo files in a users home directory. What is the best way to do that. (3 Replies)
Discussion started by: frankkahle
3 Replies
Login or Register to Ask a Question