Sponsored Content
Top Forums Shell Programming and Scripting A simpler way to do this (save a list of files based on part of their name) Post 302840507 by LMHmedchem on Monday 5th of August 2013 07:49:54 PM
Old 08-05-2013
This is why I often don't post what I want to do in its entirety. It seems that most of the time when I post a long script, no one seems to want to wade into it (which is quite understandable). I guess I still need to work on making posts that are long enough to convey what I am asking and get a workable solution, but short enough that they will actually be read.

I modified the code that you posted and have something that gives me what I need. This is the modified code,
Code:
#!/bin/bash

# argument $1 is the field to sort on based on file names as below

# 53.96_E3000_50.19_E2200_35_ri_OA_f0_S1C_v17_52.26.1_4_ON_0.25lr.out.txt
#     1     2     3     4  5

# argument $2 is the file count, meaning the number of files to find and copy
# argument $3 is the set type EV/CV

# for the top 10 EV outcomes call as ./01_copy_top_outcomes.sh 1 10 EV
# for the top 10 CV outcomes call as ./01_copy_top_outcomes.sh 3 10 CV

USAGE="./script_name  sort_field   file_count   set_type"

# field to sort on
KEY_FIELD=$1
# number of files to find
FILE_COUNT=$2
# processing set type EV/CV
SET_TYPE=$3

# make sure there are 3 arguments and the first two are numberss
if [ $# -ne 3 ] || [ "$1" != "${2%*[^0-9]*}" ] || [ "$2" != "${3%*[^0-9]*}" ]
then    echo "$USAGE"
        exit 1
fi

# loop on all folds
FOLDS=(f0 f1 f2 f3 f4 f5 f6 f7 f8 f9)

for FOLD in ${FOLDS[@]}
do

#  check if the directory exists, this should never throw.
   if [[ ! -d "$FOLD" ]]
   then  echo 'directory' $FOLD'/ does not exist, exit script'
         exit 1
   fi

   # change directory to current fold
   cd $FOLD
   echo "processing" $FOLD

   #re-initalize
   OUTPUT=""
   FILE_TEMP=""
   FILE_NAME=""
   RAND_SET=""

   # sort the list of filenames and output the top number "n" as specified in argument $3
   FILE_LIST=( $(df -h | ls *.out.txt | sort -t_ -k$KEY_FIELD,$KEY_FIELD'n' | awk -F_ -v f=$KEY_FIELD -v c=$FILE_COUNT 'NR > c {exit} {printf("%s", $0)}') )

   # loop up to file count to parse output and copy files that were found by sort
   for (( LOOP_CT=1; LOOP_CT<=$FILE_COUNT; LOOP_CT++ ))
   do

      # parse output string on .out.txt to locate individual files
      FILE_TEMP=`echo $FILE_LIST | awk -v N=$LOOP_CT 'BEGIN {FS=".out.txt"} {print $N}'`
      # restore file extension
      FILE_NAME=$FILE_TEMP'.out.txt'

      echo $FILE_NAME

      # copy file and corresponding ini weight set to continue
      # copy file to continue
      cp -p './'$FILE_NAME './'$FOLD'_continue/'$SET_TYPE'/'$FILE_NAME

      #  find random ini set number
      RAND_SET=`echo $FILE_NAME | awk 'BEGIN {FS="_"} {print $5}'`
      # copy random ini weight file to continue
      cp -p '../rnd_ini/'$FOLD'/ri_'$RAND_SET'_'*'.wts'  './'$FOLD'_continue/'$SET_TYPE'/'

   done

   # return to start directory
   cd ../

done

The entire list of files that is found ends up in the variable FILE_LIST, so that gets parsed into individual file names. Those files are copied to the proper location and an associated file is also located an copied. This loops through all sub folders f0-f9, so that is no longer an argument.

This seems reasonable and works, but I don't know awk well enough to see if there are any hidden problems. There is probably an easier way to copy the files I need, but I don't know how to copy in awk, so I needed to get the file names in bash variables that I know how to manipulate to some extent.

Do you see anything dreadfully wrong here? This does give me the ablity to specify the sort field and the number of files I want, which is a big improvement over what I first posted.

LMHmedchem
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

can I save list of files in memory and not in text file?

Hello all im using allot with the method of getting file list from misc place in unix and copy them into text file and then doing misc action on this list of files using foreach f (`cat file_list.txt`) do something with $f end can I replace this file_list.txt with some place in memory? ... (1 Reply)
Discussion started by: umen
1 Replies

2. UNIX for Dummies Questions & Answers

Report of duplicate files based on part of the filename

I have the files logged in the file system with names in the format of : filename_ordernumber_date_time eg: file_1_12012007_1101.txt file_2_12022007_1101.txt file_1_12032007_1101.txt I need to find out all the files that are logged multiple times with same order number. In the above eg, I... (1 Reply)
Discussion started by: sudheshnaiyer
1 Replies

3. Shell Programming and Scripting

strike last part from list of files

Hi, I have list of files as following: /home/abc/x/23344.php /home/axx/zz/ddddd/abc/7asda/2434.php /home/zzz/7x/y/114.php /home/assssc/x/yasyday/23664.php ( last part in each line is <somenumber.php> I need to somehow get this from the above: /home/abc/x/... (6 Replies)
Discussion started by: fed.linuxgossip
6 Replies

4. Shell Programming and Scripting

Compare two files based on integer part only

Please see how can I do this: File A (three columns): X1,Y1,1.01 X2,Y2,2.02 X3,Y3,4.03 File B (three columns): X1,Y1,1 X2,Y2,2 X3,Y3,4.0005 Now I have to compare file A and B based on the integer part of column 3. Means first 2 rows should be OK and the third row should not satisfy... (12 Replies)
Discussion started by: yale_work
12 Replies

5. Shell Programming and Scripting

find the line starting with a pattern and save a part in variable

Hi i have a file which has mutiple line in it. inside that i have a pattern similar to this /abc/def/hij i want to fine the pattern starting with "/" and get the first word in between the the symbols "/" i.e. "abc" in this case into a variable. thanks in advance (13 Replies)
Discussion started by: kichu
13 Replies

6. UNIX for Dummies Questions & Answers

List only files based on a pattern

Hi Gurus, I need to list only the files with out certain extension. For eg from the following list of files: I need to only list: Thanks Shash (7 Replies)
Discussion started by: shash
7 Replies

7. Shell Programming and Scripting

List duplicate files based on Name and size

Hello, I have a huge directory (with millions of files) and need to find out duplicates based on BOTH file name and File size. I know fdupes but it calculates MD5 which is very time-consuming and especially it takes forever as I have millions of files. Can anyone please suggest a script or... (7 Replies)
Discussion started by: prvnrk
7 Replies

8. Shell Programming and Scripting

Save value from output of Corestat and save in a list for each core

I am trying to modify the "corestat v1.1" code which is in Perl.The typical output of this code is below: Core Utilization CoreId %Usr %Sys %Total ------ ----- ----- ------ 5 4.91 0.01 4.92 6 0.06 ... (0 Replies)
Discussion started by: Zam_1234
0 Replies

9. UNIX for Dummies Questions & Answers

Rename files based on a list

Hi, I have a directory with a lot of files like this: a.bam b.bam c.bam I like to rename these files based on a list where the name of the files in the first column will be replasced by the names in the second column. Here is my list which is a tab-delimited text file: a x b y c ... (4 Replies)
Discussion started by: a_bahreini
4 Replies

10. Shell Programming and Scripting

Save an specific part of a expect_out in a variable

I have a expect file like this #!/opt/tools/unsupported/expect-5.39/bin/expect spawn ssh -l user ip expect_after eof {exit 0} set timeout 10 log_file /report.txt expect "Password:" { send "pasword\r" } expect "$ " { send "date\r" } expect "$ " { send "readlink /somelink\r" } set... (7 Replies)
Discussion started by: bebehnaz
7 Replies
FOLD(1) 							   User Commands							   FOLD(1)

NAME
fold - wrap each input line to fit in specified width SYNOPSIS
fold [OPTION]... [FILE]... DESCRIPTION
Wrap input lines in each FILE (standard input by default), writing to standard output. Mandatory arguments to long options are mandatory for short options too. -b, --bytes count bytes rather than columns -c, --characters count characters rather than columns -s, --spaces break at spaces -w, --width=WIDTH use WIDTH columns instead of 80 --help display this help and exit --version output version information and exit GNU coreutils online help: <http://www.gnu.org/software/coreutils/> Report fold translation bugs to <http://translationproject.org/team/> AUTHOR
Written by David MacKenzie. COPYRIGHT
Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. SEE ALSO
The full documentation for fold is maintained as a Texinfo manual. If the info and fold programs are properly installed at your site, the command info coreutils 'fold invocation' should give you access to the complete manual. GNU coreutils 8.22 June 2014 FOLD(1)
All times are GMT -4. The time now is 06:04 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy