Looping inside directories based on a file which contains file directory list


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Looping inside directories based on a file which contains file directory list
# 1  
Old 10-16-2012
Looping inside directories based on a file which contains file directory list

Hi All,

Please help.
I have got a file which contains a listing of a file and some directories after it, one by one. I am calling this file xyz.txt here
Code:
file1
dir1
dir2
dir3
dir4
file2
dir5
dir6
dir7
dir8
file3
dir9
dir10
dir11
dir12

Requirement is that I have to pick the files one by one from xyz.txt file i.e. file1 then file2 then file3 ... for file1 I have to go inside directories listed below file1 i.e. dir1, dir2,dir3, dir4. for file2 I have to go inside directories listed below file2 i.e. dir5,dir6, dir7, dir8. means like a loop. Actually, inside these directories there are some log files. i have to search for a pattern (some records)from these log files and grep these records from the files mentioned in the file xyz.txt ( this part looks easy and i can try for it).
Only thing is I am not able to loop through file xyz.txt

thanks.
# 2  
Old 10-16-2012
Not clear. How do you tell dirs from files? Always four dirs? Where do you get the patterns to grep for? Be more specific. What have you tried so far?
# 3  
Old 10-16-2012
clarification, let me know if anything is missing

Thanks for the reply. Sorry if i am not clear here.
suppose file1 is as follow ( and other files file2, file3 etc also similar to this, only data is different)

Code:
00020856820000      00544     00000000000001028
00068826707655      00155     00000000000006013
00003150631309      00166     00000000000002027
00004800121348      00088     00000000000004042
00068826704470      00645     00000000000001101
00001380055665      06005     00000000000001045
00078616200151      00603     00000000000004014
00002880015010      06111     00000000000001099
00004400002795      00603     00000000000006142
00002073511018      06078     00000000000004044
00002073509273      06078     00000000000001024
00000000004329      00144     00000000000002133
00004152087031      00666     00000000000001080
00004470000927      00777     00000000000001106
00002310010266      00066     00000000000002129
00001251140600      00650     00000000000001116
00004180050130      00666     00000000000002036
00007218056750      00444     00000000000002049
00068826712549      00650     00000000000001027
00004140900410      00767     00000000000002134

dir1, dir2, dir3,dir4 contain the logs, log1.txt,log2.txt,log3.txt
log1.txt is inside dir1
log2.txt is inside dir2
log3.txt is inside dir3
log4.txt is inside dir4

dir1,dir2,dir3 etc names are always like this, i.e always word "dir" comes at the starting in directory name.

suppose log1.txt is
Code:
00068826707655 sfdfsdfsf
00003150631309 gdgdgdgdg
00004800121348 sgsgsgsgsg

log2.txt is
Code:
00004400002795 gsgsgsgsgsg
00002073511018 gsgsggsgsgg

log3.txt is
Code:
00007218056750 gsgsgsgsg
00068826712549 gsgsgsgsg
00004140900410 gsgggsgsg

log4.txt is
Code:
00001251140600 fasdfasfs

so i am thinking like:

1) Traverse file xyz.txt
2) First, read file1, store the filename(file1) in some variable
3) Traverse through dir1,dir2,dir3,dir4 and cut the first field from 1-14 ( only the numbers) from the logs log1.txt, log2.txt, log3.txt, log4.txt i.e.
Code:
00068826707655
00003150631309
00004800121348
00004400002795
00002073511018
00007218056750
00068826712549
00004140900410
00001251140600

4) grep these numbers from the file1 and make a new file.
Code:
00068826707655      00155     00000000000006013
00003150631309      00166     00000000000002027
00004800121348      00088     00000000000004042
00004400002795      00603     00000000000006142
00002073511018      06078     00000000000004044
00007218056750      00444     00000000000002049
00068826712549      00650     00000000000001027
00004140900410      00767     00000000000002134

5) Then, repeat the same steps from 2) to 4) for file2

6) Then, repeat the same steps from 2) to 4) for file3
# 4  
Old 10-16-2012
Well, not knowing your system nor shell (which you did not mention), based on your examples and requirements, and hoping I understood and interpreted everything correctly, I've come up with this, which works on linux and bash:
Code:
#!/bin/bash
# set -vx
function grepit () {
                  eval $DIRS" | cut -d\" \" -f1 > pattfile"
                  grep -f pattfile $fn >$fn.grep
                 }

FIRST=1
DIRS="cat "
while read line
     do if [ "${line:0:3}" != "dir" ]
            then    [ "$FIRST" -eq "0" ] && grepit
                    fn=$line
                    FIRST=0
                    DIRS="cat "
            else    DIRS=$DIRS" "$line"/log*.txt"
       fi
     done <xyz.txt
grepit
rm pattfile

Results will finish up in respective "filename.grep". Pls. test and come back with results.

Last edited by RudiC; 10-16-2012 at 11:10 AM..
# 5  
Old 10-17-2012
Thanks

Thank You RudiC for all your efforts.
Sorry I forgot to mention system and shell. Smilie
system is AIX 6.1 and shell is ksh
I will try your code.

in your code
Code:
 do if [ "${line:0:3}" != "dir" ]

i am not sure if this line of code will work in ksh.
i guess you are looking for 4 directories always. In my case it could be any number of directories and not 4 always ... You mentioned already " Always four dirs?" i missed it. very sorry for that Smilie

---------- Post updated 10-17-12 at 01:35 PM ---------- Previous update was 10-16-12 at 09:26 PM ----------

I tried to run your script by changing the code from
Code:
#!/bin/bash

to
Code:
#!/bin/ksh

and removing that "function" keyword, but it failed in this line
Code:
"${line:0:3}": bad substitution

I guess this doesn't work in ksh.

Last edited by Piyush Jakra; 10-17-2012 at 04:14 AM..
# 6  
Old 10-17-2012
My little meek script will work for more than four directories. In that if clause you'll need to tell files from directories, that's all. Sorry I don't have access to a ksh shell. In bash you could do a check for directory like [ -d "$line" ] or file like [ -f "$line" ]; according to the man page, this should exist in ksh, or, try to find a ksh counterpart.
# 7  
Old 10-17-2012
thank you very much Rudic. I will check.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Search for a file in all directories and place the file in that directory

Hi All, Daily I am getting the updated file. I have to search for this file in all directories and sub directories. If the file existed in a particular directory then move this updated file to that particular directory. If the file is not existed in any of the directories then place this... (4 Replies)
Discussion started by: ROCK_PLSQL
4 Replies

2. Shell Programming and Scripting

How to delete all the files and folders inside all the directories except some specific directory?

hi, i have a requirement to delete all the files from all the directories except some specific directories like archive and log. for example: there are following directories such as A B C D Archive E Log F which contains some sub directories and files. The requirement is to delete all the... (7 Replies)
Discussion started by: Little
7 Replies

3. Shell Programming and Scripting

Getting all the subdirectories inside directories in excel file

Can anyone help me with a short command or script for the below scenario there is a path, /a/b/c/home?? Inside the above path there are number of subdirectories such as one two three four i need to take all the subdirectories inside home?? with full path. i need only one level of... (4 Replies)
Discussion started by: Little
4 Replies

4. Shell Programming and Scripting

List directories and count files inside

I'm trying to make a script that will list all directories under a selection as well as the number of files in each. I cannot get it to work under a symbolic link. The file structure is: XXX_20131127_001 dir01 (sym link) 2404x912 file.0000.xxx to ... (10 Replies)
Discussion started by: scribling
10 Replies

5. Shell Programming and Scripting

How to list all the files, directories and sub-directories in the current path except one directory?

Can anyone come up with a unix command that lists all the files, directories and sub-directories in the current directory except a folder called log.? Thank you in advance. (7 Replies)
Discussion started by: Manjunath B
7 Replies

6. Shell Programming and Scripting

Compressing all directories inside a directory and remove the uncompressed version

hi pls give me a script to compress all directories inside a directory and remove the original uncompressed version... >> please also tell the single commmand to uncompress all the directories back...whemn needed (2 Replies)
Discussion started by: dll_fpga
2 Replies

7. UNIX for Dummies Questions & Answers

create a file inside a directory

create a file inside a directory in one command like current directory is root i want to create a directory inside root and a file inside that directory is there any command like touch /d/d.txt d directory does not exist (1 Reply)
Discussion started by: abhisheklodha13
1 Replies

8. UNIX for Dummies Questions & Answers

creating separate directories according to file extension and keeping file in different directory as

unix program to which a directory name will be passed as parameter. This directory will contain files with various extensions. This script will create directories with the names of the extention of the files and then put the files in the corresponding folder. All files which do not have any... (2 Replies)
Discussion started by: Deekay.p
2 Replies

9. Shell Programming and Scripting

i want to delete a file based on existing file in a directory

hi i am having four files in a directory.like 1)sampleRej 2)exampleRej 3)samplemain 4)examplemain my requirement is i have to search for the rejected files (sampleRej,exampleRej) in a directory.if these files in that directory then i have to delete the main files... (3 Replies)
Discussion started by: srivsn
3 Replies

10. UNIX for Dummies Questions & Answers

list file's by size order in sepecfied directory and sub directories

How do I list files of type "*.file" for example by size order recursively ? (2 Replies)
Discussion started by: ferretman
2 Replies
Login or Register to Ask a Question