Script to copy certain info from several directories


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to copy certain info from several directories
# 8  
Old 08-31-2012
I may have edited something in underneath you while you replied. Sorry, I'm sneaky like that. Smilie
# 9  
Old 08-31-2012
Quote:
Originally Posted by Corona688
Hmm, this may or may not be difficult. Do you have stat? It can print epoch times, which you feed into a numeric sort, and then get rid of with awk once they've served their purpose...

Code:
find . -name '*.root' | xargs stat -c "%Z %n" | sort -r -n | awk '{ print $2 }' > listfile

while read LINE
do
        echo mv "$LINE" /path/to/dest
done < listfile



Are you sure it should work?? It failed to pass all the directories info.

Thanks
Pooja

---------- Post updated at 05:11 PM ---------- Previous update was at 05:01 PM ----------

Needed an immediate help:
In the given code piece, I am running it with command
Code:
./ScriptName condorjob

the Issues is that the DataMC.sh which is simple script file failed to copy to the temp.sh or (TEMPSCRIPT). I have run this kind of script earlier as well and they workrd fine.
I am not sure what I am missing here.
Kindly comment.

thanks

Code:
#!/bin/bash
date

# Global Parameters
#PATHNAME=/eos/uscms/store/user/pooja04//analysis2012/525/data/0001/data/30Aug2012/   #533
PATHNAME=/eos/uscms/store/user/pooja04//analysis2012/525/data/doubleele/2012/datav1/
FileName=DataFileName
#TARGETPATH=
SCRIPT=DataMC.sh
TEMPSCRIPT=temp.sh
CARD=card_Data2011AB_Zee_40GeV
CondorJob=Job_condor

function pause(){
   read -p "$*"
}

usage () {
    echo "Script to submit offline batch mode jobs"
    echo "Possible arguments:"
    echo "  batchjob - to submit batch jobs"
    echo "  condorjob - to submit batch jobs"
    echo "  help - shows this help"
    echo "   ./Script4OfflineJobs.sh batchjob NoofJobs"
}

# this sets CLASSPATH                                                                                                                                                  
path () {
    source /uscms/home/pooja04/ToBegin.sh
    cd /uscms_data/d3/pooja04/CMSSW_5_3_3_patch2/src/ElectroWeakAnalysis/MultiBosons/test
    cmsenv
    cd /uscms/home/pooja04/script/AnalysisCode/Code42012/v2
}


#To Copy the Files Inforamtion from the Destination Folder
CopyFilesInfo() {
    ls -ltr "$PATHNAME" | grep root | awk '{print string path $9}' string="$CONSTANT" path="$PATHNAME"  > "$FileName"
#    ls -ltr $PATHNAME grep root | awk '{print "$PATHNAME"$9}' > "$FileName"
    echo "FileName are Copied"
}

#Divide the jobs in small files
DivideJobs() {
    echo "Division Of Jobs"
    #split -$4  $3 chunk
    split -10  $FileName chunk
    
    i=0
    for file in chunk*
      do
      ((i=i+1))
      new_file="data"$i".list"
      perl -0pe 's/\n$//' $file >  $new_file
      
      #to remove the processed file   
      rm -rf $file
      echo $i ' succesful'
    done
}

#For Submiting the batchjobs
SubmitBatchJob () {
    max=$2
    for (( i=0; i<=$max; ++i )) ; 
      do
      echo "SHELL SCRIPTING ================ $i ==============" 
      new_file="temp"$i".sh"
      cp $SCRIPT $new_file
      
      if [ $i == 0 ]; then
	  source $new_file
	  pause 'Press [Enter] key to continue...'
	  rm $new_file
      else
	  if [ $i > 0 ]; then
	      { rm $CARD; sed -e "s/data$i/data$((i+1))/g" > $CARD; } < $CARD
	      source $new_file
	      pause 'Press [Enter] key to continue...'
	      rm $new_file
	  else
	      echo "SUCCESFULLY DONE.."
	  fi
      fi
    done
}    


#For Submiting the batchjobs
SubmitCondorJob () {

    for (( i=0; i < 89 ; i++ ))
      do
      echo "CondorJob Submission For  ================ $i ==============" 
      new_file="temp"$i".sh"

      if [ $i == 0 ]; then
      cp $SCRIPT $TEMPSCRIPT
	  echo "$SCRIPT"
	  

	  condor_submit $CondorJob
	  rm $TEMPSCRIPT
	  echo "Going To Sleep.."
	  sleep 100
      else
	  if [ $i > 0 ]; then
	      { rm $CARD; sed -e "s/data$i/data$((i+1))/g" > $CARD; } < $CARD
	     cp $SCRIPT $TEMPSCRIPT 
	    
	      condor_submit $CondorJob
	      rm $TEMPSCRIPT
	
	      echo "Going To Sleep.."
	      sleep 100
	      echo "SUCCESFULLY DONE.."
	  fi
      fi
    done
}    

if [ $# -lt 1 ]; then
    usage
    exit 1
fi

if [ $1 = "help" ]; then
    usage
    exit 0

elif [ "$1" = "batchjob" ]; then
    path  
    CopyFilesInfo
    DivideJobs
    SubmitBatchJob
    exit 0

elif [ "$1" == "condorjob" ]; then
    path
    CopyFilesInfo 
    DivideJobs
    SubmitCondorJob
    exit 0
fi

# 10  
Old 08-31-2012
Don't know what goes wrong in your script, but I noticed some things...

# ls -ltr $PATHNAME grep root | awk '{print "$PATHNAME"$9}' > "$FileName"
- this is commented, but there is a missed "|" (pipe) before "grep root", if you should use the line again.

elif [ "$1" == "condorjob" ]; then
- should be single "=" I think.
# 11  
Old 08-31-2012
Quote:
Originally Posted by nrjrasaxena
Are you sure it should work?? It failed to pass all the directories info.
In what way did it not work? It worked on my machine.

Show me what you did and what happened word for word, letter for letter, keystroke for keystroke.
# 12  
Old 09-01-2012
The script I ran had the following content.
Code:
#!/bin/bash                                                                                                                                                            
find /eos/uscms/store/user/pooja04//analysis2012/525/data/doubleele/2012/ -name '*.root' | xargs stat -c "%Z %n" | sort -r -n | awk '{ print $2 }' > listfile

while read LINE
do
        echo mv "$LINE" /path/to/dest
done < listfile

It ran fine. But the final file "listfile" only have the information of one directory and some information of second directory, but not directories which are at location /eos/uscms/store/user/pooja04//analysis2012/525/data/doubleele/2012/.

The output file is big, do you want to look at it? I can post if you wish.
Let me know

Thanks
Pooja

---------- Post updated at 01:59 AM ---------- Previous update was at 01:57 AM ----------

Quote:
Originally Posted by 244an
Don't know what goes wrong in your script, but I noticed some things...

# ls -ltr $PATHNAME grep root | awk '{print "$PATHNAME"$9}' > "$FileName"
- this is commented, but there is a missed "|" (pipe) before "grep root", if you should use the line again.

elif [ "$1" == "condorjob" ]; then
- should be single "=" I think.
thanks for both the suggestions. But in the end, it is giving me hard time in that CP command. which should have ran fine. gurrrrr Smilie No idea what is happeing..

thnaks
# 13  
Old 09-01-2012
I still haven't studied your script in details, and don't understand what you mean with "hard time in that CP command", but there is one more thing I noticed:
Code:
...
SubmitBatchJob () {
  max=$2
...

it seems that "max" will get a NULL value because you call the function "SubmitBatchJob" with no parameters? The "$2" is parameter 2 for the function, not the main script.
This User Gave Thanks to 244an For This Post:
# 14  
Old 09-01-2012
Quote:
Originally Posted by nrjrasaxena
It ran fine. But the final file "listfile" only have the information of one directory and some information of second directory, but not directories which are at location /eos/uscms/store/user/pooja04//analysis2012/525/data/doubleele/2012/.
Find is recursive, it will print everything.

xargs can split across as many split calls as needed.

sort can handle input of arbitrary size too.

I see no reason anything should be truncating or missing output there, except.. Hm. Do any of the directory names contain spaces?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Find and Copy Directories ONLY

I am trying to copy only the date specific folders/directories using the following command. However, the following copy command is also copying files from the root folder (OriginalFolder). find /OriginalFolder/ -type -d \{ -mtime 1 -o -mtime 2 \ } -exec cp -R {} /CopyTo/'hostname'__CopyTo/ \;... (2 Replies)
Discussion started by: apacheLinux
2 Replies

2. Shell Programming and Scripting

Copy directories in make file

LD:=C:/WindRiver/diab/5.9.3.0/WIN32/bin/dld.exe CFILES:=$(wildcard *.c) OBJFILES:=$(subst .c,.o, $(CFILES)) OBJ_PATH:=$(PRJ_PATH)/out/ ADDOBJFILES := $(addprefix $(OBJ_PATH),$(OBJFILES)) FILES:=C:/EB/tresos/workspace/Test_Spi/output/src copyfiles: cp ... (3 Replies)
Discussion started by: ushacy
3 Replies

3. Shell Programming and Scripting

Shell script to copy particular file from directories recursively

I have directory path in which there are several sub directories. In all these sub dir there will be one env.cnf file. I want to copy this env.cnf file from each sub dir's and place them in destination path by creating same filename as sub dir_env.cnf. After copying env.cnf files from source... (4 Replies)
Discussion started by: Optimus81
4 Replies

4. Shell Programming and Scripting

Compare & Copy Directories : Bash Script Help

Beginner/Intermediate shell; comfortable in the command line. I have been looking for a solution to a backup problem. I need to compare Directory 1 to Directory 2 and copy all modified or new files/directories from Directory 1 to Directory 3. I need the directory and file structure to be... (4 Replies)
Discussion started by: Rod
4 Replies

5. Shell Programming and Scripting

How to extract the day of the year and use that info to copy a file remotely

Hello, Thank you in advance for helping a newbie who is having great trouble with this simple task. I'm allowed to copy one file remotely each night due to bandwidth restrictions. A new file gets generated once a day, and I need to copy the previous day's file. Here is what I'd like to do:... (1 Reply)
Discussion started by: tmozdzen
1 Replies

6. UNIX for Dummies Questions & Answers

how can i copy those files into other directories have the same name

how can i copy those files into other directories have the same name but different in the end i have files in directory called test: 10_10_asdadfsdfad.txt 10_10_11_asdawqefwkjasd.txt 10_10_11_12_asdafjjhoqwd.txt i want to put them in exist directory thart i have on my system i have... (1 Reply)
Discussion started by: t17
1 Replies

7. Shell Programming and Scripting

Need help to copy the lease info block

Hi, am having a lease file which contains lots os lease info. In that i have to copy the whole block(shown below) by identifying the mac and change the IP according to the i/p. I have used like sed s/${ip_addr}/$ch_ip/g $temp_file | grep -B5 "${mac}" >> $persistent_file sed... (2 Replies)
Discussion started by: SMNK
2 Replies

8. UNIX for Dummies Questions & Answers

Copy file into directories and sub-directories

Hello- I need to copy a file into multiple directories, and each directory's sub-directories (of which there are 5) Currently, the parent directory is set up like this: dir1 sub-dir1 sub-dir2 sub-dir3 sub-dir4 sub-dir5 dir2 sub-dir1 sub-dir2 sub-dir3 ... (1 Reply)
Discussion started by: penlok
1 Replies

9. UNIX for Dummies Questions & Answers

Copy selected Directories

I wonder if someone would help me a little here. I have a directory (folder on a mac) with about 100 subfolders and sub-subfolders and files there in. All sub directories have the same name structure, "AAA Name". Like this: ISP CompanyName ITS CompanyName KEL CompanyName KRA CompanyName... (2 Replies)
Discussion started by: sigurarm
2 Replies

10. UNIX for Dummies Questions & Answers

Copy and keep owner info

I need to copy all fiels and directories to another locathon and keep the owner info and rights. I tried cp -r - p * /newdir No good. (5 Replies)
Discussion started by: aarono
5 Replies
Login or Register to Ask a Question