Sponsored Content
Top Forums Shell Programming and Scripting A simpler way to do this (save a list of files based on part of their name) Post 302839517 by LMHmedchem on Thursday 1st of August 2013 09:17:20 PM
Old 08-01-2013
This is the full script I have that does what I want.

Code:
#!/bin/bash


# loop through all files and copy the top 5 EV and CV to continue w/ random weight files

FOLDS=(f0 f1 f2 f3 f4 f5 f6 f7 f8 f9)

for FOLD in "${FOLDS[@]}"
do

   # get directory list
   FILES='./'$FOLD'/'*'out.txt'

   # reinitalize
   FILENAME=""
   EV_MAE_VALUE=0
   CV_MAE_VALUE=0

   EV_MAE_0=1000.0
   EV_MAE_1=1000.0
   EV_MAE_2=1000.0
   EV_MAE_3=1000.0
   EV_MAE_4=1000.0

   EV_FILES=(NULL0 NULL1 NULL2 NULL3 NULL4)

   CV_MAE_0=1000.0
   CV_MAE_1=1000.0
   CV_MAE_2=1000.0
   CV_MAE_3=1000.0
   CV_MAE_4=1000.0

   CV_FILES=(NULL0 NULL1 NULL2 NULL3 NULL4)

   for INFILE in $FILES
   do

   #  remove directory from path
      FILENAME=`echo $INFILE | awk 'BEGIN {FS="/"} {print $3}'`
   #  find ev mae value
      EV_MAE_VALUE=`echo $FILENAME | awk 'BEGIN {FS="_"} {print $1}'`
   #  find ev mae value
      CV_MAE_VALUE=`echo $FILENAME | awk 'BEGIN {FS="_"} {print $3}'`

   # save the names of the top 5 EV files
      if (( $(bc <<< "$EV_MAE_VALUE < $EV_MAE_0") == 1 ))
      then
         #bump down current list items
         EV_FILES[4]=${EV_FILES[3]}; EV_MAE_4=$EV_MAE_3
         EV_FILES[3]=${EV_FILES[2]}; EV_MAE_3=$EV_MAE_2
         EV_FILES[2]=${EV_FILES[1]}; EV_MAE_2=$EV_MAE_1
         EV_FILES[1]=${EV_FILES[0]}; EV_MAE_1=$EV_MAE_0
         EV_FILES[0]=$FILENAME
         # assign EV_MAE_VALUE to top value
         EV_MAE_0=$EV_MAE_VALUE

      elif (( $(bc <<< "$EV_MAE_VALUE < $EV_MAE_1") == 1 ))
      then
         EV_FILES[4]=${EV_FILES[3]}; EV_MAE_4=$EV_MAE_3
         EV_FILES[3]=${EV_FILES[2]}; EV_MAE_3=$EV_MAE_2
         EV_FILES[2]=${EV_FILES[1]}; EV_MAE_2=$EV_MAE_1
         EV_FILES[1]=$FILENAME
         EV_MAE_1=$EV_MAE_VALUE

      elif (( $(bc <<< "$EV_MAE_VALUE < $EV_MAE_2") == 1 ))
      then
         EV_FILES[4]=${EV_FILES[3]}; EV_MAE_4=$EV_MAE_3
         EV_FILES[3]=${EV_FILES[2]}; EV_MAE_3=$EV_MAE_2
         EV_FILES[2]=$FILENAME
         EV_MAE_2=$EV_MAE_VALUE

      elif (( $(bc <<< "$EV_MAE_VALUE < $EV_MAE_3") == 1 ))
      then
         EV_FILES[4]=${EV_FILES[3]}; EV_MAE_4=$EV_MAE_3
         EV_FILES[3]=$FILENAME
         EV_MAE_3=$EV_MAE_VALUE

      elif (( $(bc <<< "$EV_MAE_VALUE < $EV_MAE_4") == 1 ))
      then
         EV_FILES[4]=$FILENAME
         EV_MAE_4=$EV_MAE_VALUE

      fi

   # save the names of the top 5 CV files
      if (( $(bc <<< "$CV_MAE_VALUE < $CV_MAE_0") == 1 ))
      then
         #bump down current list items
         CV_FILES[4]=${CV_FILES[3]}; CV_MAE_4=$CV_MAE_3
         CV_FILES[3]=${CV_FILES[2]}; CV_MAE_3=$CV_MAE_2
         CV_FILES[2]=${CV_FILES[1]}; CV_MAE_2=$CV_MAE_1
         CV_FILES[1]=${CV_FILES[0]}; CV_MAE_1=$CV_MAE_0
         CV_FILES[0]=$FILENAME
         # assign EV_MAE_VALUE to top value
         CV_MAE_0=$CV_MAE_VALUE

      elif (( $(bc <<< "$CV_MAE_VALUE < $CV_MAE_1") == 1 ))
      then
         CV_FILES[4]=${CV_FILES[3]}; CV_MAE_4=$CV_MAE_3
         CV_FILES[3]=${CV_FILES[2]}; CV_MAE_3=$CV_MAE_2
         CV_FILES[2]=${CV_FILES[1]}; CV_MAE_2=$CV_MAE_1
         CV_FILES[1]=$FILENAME
         CV_MAE_1=$CV_MAE_VALUE

      elif (( $(bc <<< "$CV_MAE_VALUE < $CV_MAE_2") == 1 ))
      then
         CV_FILES[4]=${CV_FILES[3]}; CV_MAE_4=$CV_MAE_3
         CV_FILES[3]=${CV_FILES[2]}; CV_MAE_3=$CV_MAE_2
         CV_FILES[2]=$FILENAME
         CV_MAE_2=$CV_MAE_VALUE

      elif (( $(bc <<< "$CV_MAE_VALUE < $CV_MAE_3") == 1 ))
      then
         CV_FILES[4]=${CV_FILES[3]}; CV_MAE_4=$CV_MAE_3
         CV_FILES[3]=$FILENAME
         CV_MAE_3=$CV_MAE_VALUE

      elif (( $(bc <<< "$CV_MAE_VALUE < $CV_MAE_4") == 1 ))
      then
         CV_FILES[4]=$FILENAME
         CV_MAE_4=$CV_MAE_VALUE

      fi

   done

   # copy list of filenames and corresponding ini weight sets to continue
   RAND_SET=""
   for I in "${EV_FILES[@]}"
   do
      # copy file to continue
      cp -p './'$FOLD'/'$I './'$FOLD'/'$FOLD'_continue/EV/'$I
      #  find random ini set number
      RAND_SET=`echo $I} | awk 'BEGIN {FS="_"} {print $5}'`
      # copy random ini weight file to continue
      cp -p './rnd_ini/'$FOLD'/ri_'$RAND_SET'_'*'.wts'  './'$FOLD'/'$FOLD'_continue/EV/'
   done

   for I in "${CV_FILES[@]}"
   do
      # copy file to continue
      cp -p './'$FOLD'/'$I './'$FOLD'/'$FOLD'_continue/CV/'$I
      #  find random ini set number
      RAND_SET=`echo $I} | awk 'BEGIN {FS="_"} {print $5}'`
      # copy random ini weight file to continue
      cp -p './rnd_ini/'$FOLD'/ri_'$RAND_SET'_'*'.wts'  './'$FOLD'/'$FOLD'_continue/CV/'
   done

   #move fold output files to stats folder
   mv './'$FOLD'/'*'out.txt' './'$FOLD'/'$FOLD'_stats/'

done

I thought it was overly long to post since my question was about the first part. This does the job that the first one I posted did, except that it does it twice. It loops through a set of sub folders f0-f9, finds the files with the top 5 EV MAE values (5 smallest field 1) and copies those files and a corresponding set of files to f*/f*_continue/EV/. Then it does the same thing for the top CV MAE values (5 smallest field 3) and copies to f*/f*_continue/CV/.

I have attached a new test dir with the script and supporting files. I have edited the script so that it is only working with f0, f1, f2 to help simplify things. The script will find the top EV MAE values by reading the first field, and then find the .wts file that goes with it. Both will be coppied to the continue folder.

For example, the top EV MAE file for f0 is,
53.96_E3000_50.19_E2200_35_ri_OA_f0_S1C_v17_52.26.1_4_ON_0.25lr.out.txt
so this will be copied to ./f0/f0_continue/EV/

The .wts file associated with this is 35 (field 5), so the script will also copy,
./rnd_ini/f0/ri_35*.wts
to ./f0/f0_continue/EV/

The top CV MAE file for f0 is,
54.90_E3000_48.65_E4300_23_ri_OA_f0_S1C_v17_52.26.1_4_ON_0.25lr.out.txt
so this will be copied to ./f0/f0_continue/CV/

The .wts file associated with this is 23 (field 5), so the script will also copy,
./rnd_ini/f0/ri_23*.wts
to ./f0/f0_continue/CV/

After a f* directory is processed, the script moves all the .out.txt files to ./f*/f*_stats/.

Since the version you posted accepts arguments, there would be no need to do both CV and EV in the same run. The script could be called twice with the proper arguments for each. It is also very nice that your method allows you to pick any number of files to collect. It will probably end up being about 20, but I'm not sure yet.

Thanks for all the help.

LMHmedchem
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

can I save list of files in memory and not in text file?

Hello all im using allot with the method of getting file list from misc place in unix and copy them into text file and then doing misc action on this list of files using foreach f (`cat file_list.txt`) do something with $f end can I replace this file_list.txt with some place in memory? ... (1 Reply)
Discussion started by: umen
1 Replies

2. UNIX for Dummies Questions & Answers

Report of duplicate files based on part of the filename

I have the files logged in the file system with names in the format of : filename_ordernumber_date_time eg: file_1_12012007_1101.txt file_2_12022007_1101.txt file_1_12032007_1101.txt I need to find out all the files that are logged multiple times with same order number. In the above eg, I... (1 Reply)
Discussion started by: sudheshnaiyer
1 Replies

3. Shell Programming and Scripting

strike last part from list of files

Hi, I have list of files as following: /home/abc/x/23344.php /home/axx/zz/ddddd/abc/7asda/2434.php /home/zzz/7x/y/114.php /home/assssc/x/yasyday/23664.php ( last part in each line is <somenumber.php> I need to somehow get this from the above: /home/abc/x/... (6 Replies)
Discussion started by: fed.linuxgossip
6 Replies

4. Shell Programming and Scripting

Compare two files based on integer part only

Please see how can I do this: File A (three columns): X1,Y1,1.01 X2,Y2,2.02 X3,Y3,4.03 File B (three columns): X1,Y1,1 X2,Y2,2 X3,Y3,4.0005 Now I have to compare file A and B based on the integer part of column 3. Means first 2 rows should be OK and the third row should not satisfy... (12 Replies)
Discussion started by: yale_work
12 Replies

5. Shell Programming and Scripting

find the line starting with a pattern and save a part in variable

Hi i have a file which has mutiple line in it. inside that i have a pattern similar to this /abc/def/hij i want to fine the pattern starting with "/" and get the first word in between the the symbols "/" i.e. "abc" in this case into a variable. thanks in advance (13 Replies)
Discussion started by: kichu
13 Replies

6. UNIX for Dummies Questions & Answers

List only files based on a pattern

Hi Gurus, I need to list only the files with out certain extension. For eg from the following list of files: I need to only list: Thanks Shash (7 Replies)
Discussion started by: shash
7 Replies

7. Shell Programming and Scripting

List duplicate files based on Name and size

Hello, I have a huge directory (with millions of files) and need to find out duplicates based on BOTH file name and File size. I know fdupes but it calculates MD5 which is very time-consuming and especially it takes forever as I have millions of files. Can anyone please suggest a script or... (7 Replies)
Discussion started by: prvnrk
7 Replies

8. Shell Programming and Scripting

Save value from output of Corestat and save in a list for each core

I am trying to modify the "corestat v1.1" code which is in Perl.The typical output of this code is below: Core Utilization CoreId %Usr %Sys %Total ------ ----- ----- ------ 5 4.91 0.01 4.92 6 0.06 ... (0 Replies)
Discussion started by: Zam_1234
0 Replies

9. UNIX for Dummies Questions & Answers

Rename files based on a list

Hi, I have a directory with a lot of files like this: a.bam b.bam c.bam I like to rename these files based on a list where the name of the files in the first column will be replasced by the names in the second column. Here is my list which is a tab-delimited text file: a x b y c ... (4 Replies)
Discussion started by: a_bahreini
4 Replies

10. Shell Programming and Scripting

Save an specific part of a expect_out in a variable

I have a expect file like this #!/opt/tools/unsupported/expect-5.39/bin/expect spawn ssh -l user ip expect_after eof {exit 0} set timeout 10 log_file /report.txt expect "Password:" { send "pasword\r" } expect "$ " { send "date\r" } expect "$ " { send "readlink /somelink\r" } set... (7 Replies)
Discussion started by: bebehnaz
7 Replies
All times are GMT -4. The time now is 01:04 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy