[Solved] How to concatenate files in subdirectories?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting [Solved] How to concatenate files in subdirectories?
# 1  
Old 10-14-2010
[Solved] How to concatenate files in subdirectories?

I have many log files in various subdirectories of a parent directory.

The directory structure e something like this:

Parent directory:

/home/linus/logfiles/

This parent directory has many subdirectories, like this:

.../logfiles/jack
.../logfiles/helen
.../logfiles/sue
.../loffiles/jack
etc...

New subdirectories are being constantly, and automatically, created. Of course new logs files are also being constantly, and automatically, created.

All log files are ascii files and have the .txt extension.

Well, what I need is a bash script which creates, on every and each subdirectory, a new ascii file with the contents of all log files existing on that specific subdirectory.
So I want the bash script to create a new concatenated file on every and each subdirectory.

Every concatenated file must be created with the contents of all ascii file located at the specific subdirectory.
After each of these concatenated files is created, all log files should be deleted.

It would be good if each concatenated file would be created with the name of the subdirectory on it resides plus the extension .txt. So, for instance, on a subdirectory named "james", the concatenated file should be created with the name "james.txt".

I would be very grateful if someone could post a bash script which solves this problem.

Last edited by zetetic; 10-14-2010 at 09:02 PM..
# 2  
Old 10-15-2010
Code:
logdir=$HOME/tmp/logfiles ## adjust to taste

for dir in "$logdir"/*/
do
  (
    cd "$dir"
    files=( *.txt )
    cat "${files[@]}" > "${PWD##*/}.txt"
    rm  "${files[@]}"
  )
done

These 2 Users Gave Thanks to cfajohnson For This Post:
# 3  
Old 10-15-2010
Thank you for your kind help, cfajohnson !
You are great.
It works like a charm!
# 4  
Old 01-11-2011
@cfajohnson:

What is "${PWD##*/}" ?
# 5  
Old 01-11-2011
Quote:
What is "${PWD##*/}" ?

POSIX parameter expansion.

It removes the longest matching pattern (*/) from the beginning of the value of the variable.

In practice, it removes the directory portion of a filepath, leaving only the filename.


Last edited by cfajohnson; 01-11-2011 at 07:51 PM..
# 6  
Old 01-11-2011
Off topic. Welcome back cfajohnson. Happy New Year.
# 7  
Old 02-07-2011
@cfajohnson

Thanks a lot cfajohnson. I do not know much about shell scripting. I tried to run your code to concatenate all the files in the subdirectories. But that didn't work. Will appreciate if you please send me the script to concatenate all the files in several subdirectories in linux. Thanks once again!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Append string to all the files inside a directory excluding subdirectories and .zip files

Hii, Could someone help me to append string to the starting of all the filenames inside a directory but it should exclude .zip files and subdirectories. Eg. file1: test1.log file2: test2.log file3 test.zip After running the script file1: string_test1.log file2: string_test2.log file3:... (4 Replies)
Discussion started by: Ravi Kishore
4 Replies

2. Shell Programming and Scripting

Unzip all the files with subdirectories present and append a part of string from the main .zip files

Hi frnds, My requirement is I have a zip file with name say eg: test_ABC_UH_ccde2a_awdeaea_20150422.zip within that there are subdirectories on each directory we again have .zip files and in that we have files like mama20150422.gz and so on. Iam in need of a bash script so that it unzips... (0 Replies)
Discussion started by: Ravi Kishore
0 Replies

3. Shell Programming and Scripting

Bash script deleting my files, and editing files in subdirectories question

#!/bin/bash # name=$1 type=$2 number=1 for file in ./** do if then filenumber=00$number elif then filenumber=0$number fi tempname="$name""$filenumber"."$type" if (4 Replies)
Discussion started by: TheGreatGizmo
4 Replies

4. Shell Programming and Scripting

[Solved] Script to concatenate 2 files with the same number of lines

Hi everyone, I have two files, namely: file1: file1Col1Row1;file1Col2Row1;file1Col3Row1 file1Col1Row2;file1Col2Row2;file1Col3Row2 file1Col1Row3;file1Col2Row3;file1Col3Row3file2: file2Col1Row1;file2Col2Row1;file2Col3Row1 file2Col1Row2;file2Col2Row2;file2Col3Row2... (0 Replies)
Discussion started by: gacanepa
0 Replies

5. UNIX for Dummies Questions & Answers

Concatenate files and delete source files. Also have to add a comment.

- Concatenate files and delete source files. Also have to add a comment. - I need to concatenate 3 files which have the same characters in the beginning and have to remove those files and add a comment and the end. Example: cat REJ_FILE_ABC.txt REJ_FILE_XYZ.txt REJ_FILE_PQR.txt >... (0 Replies)
Discussion started by: eskay
0 Replies

6. Shell Programming and Scripting

[solved] merging two files and writing to another file- solved

i have two files as file1: 1 2 3 file2: a b c and the output should be: file3: 1~a 2~b 3~c (1 Reply)
Discussion started by: mlpathir
1 Replies

7. UNIX for Advanced & Expert Users

Recursively concatenate files in subdirectories with the same folder name

I'm trying to concatenate files in subdirectories with the same folder name. Say concatenate all the files in the 'current' subdirectories in 'Literature' parent directory. Literature/USA/current/ Literature/Europe/current/ Can anyone help with it? Thanks a lot! (2 Replies)
Discussion started by: joyce007
2 Replies

8. UNIX for Dummies Questions & Answers

recursively concatenate files in subdirectories with same folder name

I'm trying to concatenate files in subdirectories with the same folder name. Say concatenate all the files in the 'current' subdirectories in 'Literature' parent directory. Literature/USA/current/ Literature/Europe/current/ Can anyone help with it? Thanks a lot! (2 Replies)
Discussion started by: joyce007
2 Replies

9. UNIX for Dummies Questions & Answers

list the files but exclude the files in subdirectories

If I execute the command "ls -l /export/home/abcde/dev/proj/code/* | awk -F' ' '{print $9}' | cut -d'/' -f6-8" it will list all the files in /export/home/abcde/dev/proj/code/ directory as well as the files in subdirectories also proj/code/test.sh proj/code/test1.c proj/code/unix... (8 Replies)
Discussion started by: shyjuezy
8 Replies

10. UNIX for Dummies Questions & Answers

read files from subdirectories

hello there the problem i got: i need to list .rrd files in each sub-directory from the parent directory, then create .xml files for each rrd files, the xml file should be in the same subdirectoryas rrd file. i have tried ls |awk '{print... (3 Replies)
Discussion started by: binbintriangel
3 Replies
Login or Register to Ask a Question