Process a specific number of files ina list Post: 302887306

Sponsored Content

Top Forums Shell Programming and Scripting Process a specific number of files ina list Post 302887306 by LMHmedchem on Thursday 6th of February 2014 06:02:30 PM

02-06-2014

Registered User

Process a specific number of files ina list

Hello,

I have a list of files that was created with,

Code:

FILES='./'$FOLD'/'$FOLD'_continue/'$OPTIMIZE_ON'/'*'out.txt'

I am doing a loop on this list

Code:

for INPUT in $FILES
do
...
done

but I may not want to process everything. Is there a simple way to just process the first 5,10,n, etc in this list.

I process each file name with awk to extract two numbers from each file name,

Code:

#remove path from filename
FILENAME=`echo $INPUT | awk 'BEGIN {FS="/"} {print $5}'`
echo $FILENAME

# store random ini set
RAND_SET=`echo $FILENAME | awk 'BEGIN {FS="_"} {print $5}'`
echo "RAND_SET"  $RAND_SET

# find MAE min epoch values
MAE_EPOCH=`echo $FILENAME | awk 'BEGIN {FS="_"} {print $2}'`

#remove leading E
MAE_EPOCH=${MAE_EPOCH#"E"}
#create value at -100 epochs
let "MAE_EPOCH= $MAE_EPOCH"
echo "MAE_EPOCH"  $MAE_EPOCH

I end up with the numbers $MAE_EPOCH and $RAND_SET for each file. What I would really like to do is to scan all the files, extract the MAE_EPOCH, and then process the best some number, like the top 5 (based on the lowest values for MAE_EPOCH). I need to know the value of RAND_SET associated with each MAE_EPOCH value.

File names look like,

Code:

108.72_E1300_101.62_E3000_39_ri_OA_f0_S4A_v2_42.41.1_ON_0.25lr.out.txt

The text in blue is what I am capturing. I would need both the MAE_EPOCH and the corresponding RAND_SET value for each file I am going to process. I guess I would loop on the file set and then store the data for the files I need to process, but I'm not so sure how to do that kind of thing in bash.

Help would be greatly appreciated,

LMHmedchem

Last edited by Scrutinizer; 02-06-2014 at 07:14 PM.. Reason: Additional code tags

LMHmedchem

View Public Profile for LMHmedchem

Find all posts by LMHmedchem

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk command to find the count of files ina directory

hi Gurus, can anyone provide a awk command to get teh count of number of file sin a specific directory. appreciate any kind of information.. thanks

2. Shell Programming and Scripting

Retreive content between specific lines ina file

Hi I have a text file which has two sets of lines repeating for "n" number of times.Some data is printed between the two lines.I want to retrieve all the data thats there in between those two set of lines.I have the string value of those two set of lines. To be much more clearer ...

3. UNIX for Dummies Questions & Answers

How to reduce multiple files into a specific number of files

Can anyone please let me know how do I reduce files into a specific number of files by cat'ing files? For example: 15 files must be reduced to 1 or 5 or 9 (possible values 1 to 14)

4. Shell Programming and Scripting

Creating large number of files of specific size

Hi I am new to shell scripting.I want to create a batch file which creates a desired number of files with a specific size say 1MB each to consume space.How can i go about it using for loop /any other loop condition using shell script? Thanks

5. UNIX for Dummies Questions & Answers

Unix command to count the number of files with specific characters in name

Hey all, I'm looking for a command that will search a directory (and all subdirectories) and give me a file count for the number of files that contain specific characters within its filename. e.g. I want to find the number of files that contain "-a.jpg" in their name. All the searching I've...

6. Shell Programming and Scripting

highly specific search and replace for a large number of files

hey guys, I have a directory with about 600 files. I need to find a specific word inside a command and replace only that instance of the word in many files. For example, lets say I have a command called 'foo' in many files. One of the input arguments of the 'foo' call is 'bar'. The word 'bar'...

7. Shell Programming and Scripting

Need script to count specific word and iterate over number of files.

Hi Experts, I want to know the count of specific word in a file. I have almost 600+ files. So I want to loop thru each file and get the count of the specific word. Please help me on achieving this... Many thanks

8. UNIX for Dummies Questions & Answers

grep specific number from a list of numbers

Hello. I have 9060 files labelled File1 to File9060. They are in numerical order. When I grep a file eg. File90 it will show me all files that contain the pattern "File90", eg File901 or File9001. I can only get specific files for File1000 or higher. How can I resolve this problem? Is there a...

9. UNIX for Dummies Questions & Answers

[Solved] How to find particular files ina directory?

Hi, I am trying to write a script to find some files in a directory Example: if i have files like 2014-02-01_aaaa.txt 2014-02-01_bbbb.txt 2014-02-01_cccc.txt 2014-02-01_dddd.txt and some other files how can i just check to see if there four files exits or not i tried some thing like this...

10. Shell Programming and Scripting

List files with number to select based on number

Hi experts, I am using KSH and I am need to display file with number in front of file names and user can select it by entering the number. I am trying to use following command to display list with numbers. but I do not know how to capture number and identify what file it is to be used for...

LEARN ABOUT DEBIAN

bup-margin

bup-margin(1)						      General Commands Manual						     bup-margin(1)

NAME

       bup-margin - figure out your deduplication safety margin

SYNOPSIS

       bup margin [options...]

DESCRIPTION

       bup margin  iterates  through  all  objects  in	your  bup repository, calculating the largest number of prefix bits shared between any two
       entries.  This number, n, identifies the longest subset of SHA-1 you could use and still encounter a collision between your object ids.

       For example, one system that was tested had a collection of 11 million objects (70 GB), and bup margin returned 45.  That  means  a  46-bit
       hash  would be sufficient to avoid all collisions among that set of objects; each object in that repository could be uniquely identified by
       its first 46 bits.

       The number of bits needed seems to increase by about 1 or 2 for every doubling of the number of objects.  Since SHA-1 hashes have 160 bits,
       that  leaves 115 bits of margin.  Of course, because SHA-1 hashes are essentially random, it's theoretically possible to use many more bits
       with far fewer objects.

       If you're paranoid about the possibility of SHA-1 collisions, you can monitor your repository by running bup margin occasionally to see	if
       you're getting dangerously close to 160 bits.

OPTIONS

       --predict
	      Guess  the offset into each index file where a particular object will appear, and report the maximum deviation of the correct answer
	      from the guess.  This is potentially useful for tuning an interpolation search algorithm.

       --ignore-midx
	      don't use .midx files, use only .idx files.  This is only really useful when used with --predict.

EXAMPLE

	      $ bup margin
	      Reading indexes: 100.00% (1612581/1612581), done.
	      40
	      40 matching prefix bits
	      1.94 bits per doubling
	      120 bits (61.86 doublings) remaining
	      4.19338e+18 times larger is possible

	      Everyone on earth could have 625878182 data sets
	      like yours, all in one repository, and we would
	      expect 1 object collision.

	      $ bup margin --predict
	      PackIdxList: using 1 index.
	      Reading indexes: 100.00% (1612581/1612581), done.
	      915 of 1612581 (0.057%)

SEE ALSO

       bup-midx(1), bup-save(1)

BUP

       Part of the bup(1) suite.

AUTHORS

       Avery Pennarun <apenwarr@gmail.com>.

Bup unknown-															     bup-margin(1)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk command to find the count of files ina directory

Discussion started by: sish78

2. Shell Programming and Scripting

Retreive content between specific lines ina file

Discussion started by: chennaitomcruis

3. UNIX for Dummies Questions & Answers

How to reduce multiple files into a specific number of files

Discussion started by: aryanbelank

4. Shell Programming and Scripting

Creating large number of files of specific size

Discussion started by: swatideswal

5. UNIX for Dummies Questions & Answers

Unix command to count the number of files with specific characters in name

Discussion started by: murphysm

6. Shell Programming and Scripting

highly specific search and replace for a large number of files

Discussion started by: ksubrama

7. Shell Programming and Scripting

Need script to count specific word and iterate over number of files.

Discussion started by: elamurugu

8. UNIX for Dummies Questions & Answers

grep specific number from a list of numbers

Discussion started by: godzilla07

9. UNIX for Dummies Questions & Answers

[Solved] How to find particular files ina directory?

Discussion started by: vikatakavi

10. Shell Programming and Scripting

List files with number to select based on number

Discussion started by: mysocks

LEARN ABOUT DEBIAN

bup-margin