Sponsored Content
Top Forums Shell Programming and Scripting A simpler way to do this (save a list of files based on part of their name) Post 302839415 by LMHmedchem on Thursday 1st of August 2013 01:00:20 PM
Old 08-01-2013
A simpler way to do this (save a list of files based on part of their name)

Hello,

I have a script that checks every file with a specific extension in a specific directory. The file names contain some numerical output and I am recording the file names with the best n outcomes.

The script finds all files in the directory with the extension .out.txt and uses awk to parse the filename on underscore. In this case, I am reading the first field and looking for the smallest three values across the set of files. In other cases, I may be reading the third field. I understand that in this simple case, all I would have to do is take the first three files, but there will be other cases where that would not work.

This is the script at this point and there is sample input in the attached zip. The input file names look like,

48.93_E3200_55.94_E1900_34_ri_OA_f0_S1A_v17_52.26.1_4_ON_0.25lr.out.txt
49.15_E2700_51.98_E1200_32_ri_OA_f0_S1A_v17_52.26.1_4_ON_0.25lr.out.txt
49.16_E1600_52.54_E1600_44_ri_OA_f0_S1A_v17_52.26.1_4_ON_0.25lr.out.txt
50.36_E3400_55.09_E3000_35_ri_OA_f0_S1A_v17_52.26.1_4_ON_0.25lr.out.txt
50.62_E1700_51.92_E300_8_ri_OA_f0_S1A_v17_52.26.1_4_ON_0.25lr.out.txt

Code:
#!/bin/bash
# loop through all files and save the top 3 filenames

   # initalize
   FILENAME=""
   CURRENT_MAE_VALUE=0
   # these are initalized to an arbitrarily large value
   EV_MAE_0=1000.0
   EV_MAE_1=1000.0
   EV_MAE_2=1000.0

   EV_FILES=(NULL0 NULL1 NULL2)

   # set fold value
   FOLD=f0

   # get directory list
   FILES='./'$FOLD'/'*'out.txt'

   for INFILE in $FILES
   do

   #  remove directory from path
      FILENAME=`echo $INFILE | awk 'BEGIN {FS="/"} {print $3}'`
   #  find ev mae value
      CURRENT_MAE_VALUE=`echo $FILENAME | awk 'BEGIN {FS="_"} {print $1}'`

   # save the names of the top 3 EV files and EV values
      if (( $(bc <<< "$CURRENT_MAE_VALUE < $EV_MAE_0") == 1 ))
      then
         #bump down current list items
         EV_FILES[2]=${EV_FILES[1]}; EV_MAE_2=$EV_MAE_1
         EV_FILES[1]=${EV_FILES[0]}; EV_MAE_1=$EV_MAE_0
         EV_FILES[0]=$FILENAME
         # assign EV_MAE_VALUE to top value
         EV_MAE_0=$CURRENT_MAE_VALUE

      elif (( $(bc <<< "$CURRENT_MAE_VALUE < $EV_MAE_1") == 1 ))
      then
         #bump down current list items
         EV_FILES[2]=${EV_FILES[1]}; EV_MAE_2=$EV_MAE_1
         EV_FILES[1]=$FILENAME
         # assign EV_MAE_VALUE to second value
         EV_MAE_1=$CURRENT_MAE_VALUE

      elif (( $(bc <<< "$CURRENT_MAE_VALUE < $EV_MAE_2") == 1 ))
      then
         #bump down current list items
         EV_FILES[2]=$FILENAME
         # assign EV_MAE_VALUE to third value
         EV_MAE_2=$CURRENT_MAE_VALUE

      fi

   done

# print results
   echo "1st EV file"
   echo ${EV_FILES[0]}
   echo "EV MAE 0"  $EV_MAE_0
   echo""
   echo "2nd EV file"
   echo ${EV_FILES[1]}
   echo "EV MAE 1"  $EV_MAE_1
   echo""
   echo "3rd EV file"
   echo ${EV_FILES[2]}
   echo "EV MAE 2"  $EV_MAE_2
   echo""

My main question is about how to keep a running record of the file names of the best three values as I loop through the file names. This script does it by brute force and works alright, but I may need to save the top 20 or 50, and I don't look forward to coding that up with the method I used above.

Any suggestions?

LMHmedchem
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

can I save list of files in memory and not in text file?

Hello all im using allot with the method of getting file list from misc place in unix and copy them into text file and then doing misc action on this list of files using foreach f (`cat file_list.txt`) do something with $f end can I replace this file_list.txt with some place in memory? ... (1 Reply)
Discussion started by: umen
1 Replies

2. UNIX for Dummies Questions & Answers

Report of duplicate files based on part of the filename

I have the files logged in the file system with names in the format of : filename_ordernumber_date_time eg: file_1_12012007_1101.txt file_2_12022007_1101.txt file_1_12032007_1101.txt I need to find out all the files that are logged multiple times with same order number. In the above eg, I... (1 Reply)
Discussion started by: sudheshnaiyer
1 Replies

3. Shell Programming and Scripting

strike last part from list of files

Hi, I have list of files as following: /home/abc/x/23344.php /home/axx/zz/ddddd/abc/7asda/2434.php /home/zzz/7x/y/114.php /home/assssc/x/yasyday/23664.php ( last part in each line is <somenumber.php> I need to somehow get this from the above: /home/abc/x/... (6 Replies)
Discussion started by: fed.linuxgossip
6 Replies

4. Shell Programming and Scripting

Compare two files based on integer part only

Please see how can I do this: File A (three columns): X1,Y1,1.01 X2,Y2,2.02 X3,Y3,4.03 File B (three columns): X1,Y1,1 X2,Y2,2 X3,Y3,4.0005 Now I have to compare file A and B based on the integer part of column 3. Means first 2 rows should be OK and the third row should not satisfy... (12 Replies)
Discussion started by: yale_work
12 Replies

5. Shell Programming and Scripting

find the line starting with a pattern and save a part in variable

Hi i have a file which has mutiple line in it. inside that i have a pattern similar to this /abc/def/hij i want to fine the pattern starting with "/" and get the first word in between the the symbols "/" i.e. "abc" in this case into a variable. thanks in advance (13 Replies)
Discussion started by: kichu
13 Replies

6. UNIX for Dummies Questions & Answers

List only files based on a pattern

Hi Gurus, I need to list only the files with out certain extension. For eg from the following list of files: I need to only list: Thanks Shash (7 Replies)
Discussion started by: shash
7 Replies

7. Shell Programming and Scripting

List duplicate files based on Name and size

Hello, I have a huge directory (with millions of files) and need to find out duplicates based on BOTH file name and File size. I know fdupes but it calculates MD5 which is very time-consuming and especially it takes forever as I have millions of files. Can anyone please suggest a script or... (7 Replies)
Discussion started by: prvnrk
7 Replies

8. Shell Programming and Scripting

Save value from output of Corestat and save in a list for each core

I am trying to modify the "corestat v1.1" code which is in Perl.The typical output of this code is below: Core Utilization CoreId %Usr %Sys %Total ------ ----- ----- ------ 5 4.91 0.01 4.92 6 0.06 ... (0 Replies)
Discussion started by: Zam_1234
0 Replies

9. UNIX for Dummies Questions & Answers

Rename files based on a list

Hi, I have a directory with a lot of files like this: a.bam b.bam c.bam I like to rename these files based on a list where the name of the files in the first column will be replasced by the names in the second column. Here is my list which is a tab-delimited text file: a x b y c ... (4 Replies)
Discussion started by: a_bahreini
4 Replies

10. Shell Programming and Scripting

Save an specific part of a expect_out in a variable

I have a expect file like this #!/opt/tools/unsupported/expect-5.39/bin/expect spawn ssh -l user ip expect_after eof {exit 0} set timeout 10 log_file /report.txt expect "Password:" { send "pasword\r" } expect "$ " { send "date\r" } expect "$ " { send "readlink /somelink\r" } set... (7 Replies)
Discussion started by: bebehnaz
7 Replies
All times are GMT -4. The time now is 05:40 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy