Split list of files into an array and pass to function Post: 302931535

Sponsored Content

Top Forums Shell Programming and Scripting Split list of files into an array and pass to function Post 302931535 by LMHmedchem on Tuesday 13th of January 2015 03:18:28 PM

01-13-2015

Registered User

This is what I have set up instead of passing the array.

calling code

Code:

# the number of availalbe cores
if [ "$CORES" == "quad" ]; then

   # create 4 arrays, each with ~25% of filenames
   NUM_FILE_LISTS='4'
   PROCESSED='0'

   # get size of each subset
    BASE_LIST_SIZE=$(((NUM_INPUT_FILES) / NUM_FILE_LISTS))
   LEFTOVER=$((NUM_INPUT_FILES % NUM_FILE_LISTS))

   # set up start elements and number of elements for all lists
   # list 0
   START_ELEMENT_0='0'
   NUMBER_OF_ELEMENTS_0=$((BASE_LIST_SIZE + (LEFTOVER > 0)))
   # keep track of number of files processed
   let "PROCESSED=$PROCESSED+$NUMBER_OF_ELEMENTS_0"
      
   # list 1   
   START_ELEMENT_1=$PROCESSED
   #let "START_ELEMENT_1=$START_ELEMENT_0+$NUMBER_OF_ELEMENTS_0"
   NUMBER_OF_ELEMENTS_1=$((BASE_LIST_SIZE + (LEFTOVER > 1)))
   let "PROCESSED=$PROCESSED+$NUMBER_OF_ELEMENTS_1"
 
   # list 2  
   START_ELEMENT_2=$PROCESSED
   NUMBER_OF_ELEMENTS_2=$((BASE_LIST_SIZE + (LEFTOVER > 2)))
   # keep track of number of files processed
   let "PROCESSED=$PROCESSED+$NUMBER_OF_ELEMENTS_2"
 
   # list 3  
   START_ELEMENT_3=$PROCESSED
   # assign the rest to this list
   let "NUMBER_OF_ELEMENTS_3=$NUM_INPUT_FILES-$PROCESSED"
   # keep track of number of files processed
   let "PROCESSED=$PROCESSED+$NUMBER_OF_ELEMENTS_3"

      # call functions to process stats
      run_stats_program  $SET  $FOLD  $START_ELEMENT_0  $NUMBER_OF_ELEMENTS_0 &
      # to prevent terminal overrun
      sleep 2
      run_stats_program  $SET  $FOLD  $START_ELEMENT_1  $NUMBER_OF_ELEMENTS_1 &
      sleep 2
      run_stats_program  $SET  $FOLD  $START_ELEMENT_2  $NUMBER_OF_ELEMENTS_2 &
      sleep 2
      run_stats_program  $SET  $FOLD  $START_ELEMENT_3  $NUMBER_OF_ELEMENTS_3 &
      sleep 2
      # wait untill subshells have returned
      wait

 fi

called function

Code:

function run_stats_program {

   # function args
   SET_F=$1
   FOLD_F=$2
   START_ELEMENT_F=$3
   NUMBER_OF_ELEMENTS_F=$4
   
   # get list of stats input files in fold directory
   STATS_INPUT_FILENAMES_F=($(ls  './'$SET_F'/'$FOLD_F'/'*'in.txt'))
 
   # create file list as subest of STATS_INPUT_FILENAMES_F
   FILE_LIST=("${STATS_INPUT_FILENAMES_F[@]:$START_ELEMENT_F:$NUMBER_OF_ELEMENTS_F}")

   for INPUT_FILE in "${FILE_LIST[@]}"
   do
      echo $INPUT_FILE
   done
}

All this does at this point is print the filenames. In the end, this will process the 4 file lists in 4 subshells. Processing involved calling a c++ widget to process each file. This setup allows 4 instances of the c++ app to run simultaneously and use availalble CPU resources. There will be a similar code block for hex core.

I get this this is written in long form at the moment. It would be nice for the code to be a bit more compact and elegant, but I don't see a clear way to put the function calls in a loop or something like that.

LMHmedchem

Last edited by LMHmedchem; 01-13-2015 at 05:52 PM..

LMHmedchem

View Public Profile for LMHmedchem

Find all posts by LMHmedchem

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Can we pass array with call by value in function

I want to pass an array in my function, And my function will be changing the elements of the array in the fuction, but it should not affect the values in my array variable of main function

2. Shell Programming and Scripting

Array split function & hashes

Hi, If this is the array that is being returned to me: How would I get the values for each of the 3 records? This works for 1 Record: foreach $item (@results) { ($id, $id2, $name, $date, $email) = split(/\|/, $item, 5); print "$name<br>"; }

3. Shell Programming and Scripting

How to pass an array from SHELL to C function

Hi, I have an output generated from a shell script like; 0x41,0xF2,0x59,0xDD,0x86,0xD3,0xEF,0x61,0xF2 How can I pass this value to the C function, as below; int main(int argc, char *argv) { unsigned char hellopdu={above value}; } Regards Elthox

4. Shell Programming and Scripting

Find and split the list of files with suffiz of seg**

Hi,. I am writing a script to get the new files and split them. Requirement Find the new files under the path "/wload/scmp/app/data/OAS" (There are 5 sub folders). Gunzip the files which are having .gz suffix. Put the list of files in the filename in the format...

5. Shell Programming and Scripting

Split the file and access that files through array and loop

Hi All, the below is my requirement.. i need to split the file based on line and put that files in a array and need to access that files through loop finally i should send the files through mail.. how can we achieve this ..I am new to shell script please guide me.. I am using KSH.. ...

6. Shell Programming and Scripting

How to pass an array to a function in shell script.?

hi, I have a array say SAP_ARRAY="s1.txt" SAP_ARRAY="s2.txt" how can i pass this full array to a function. here is the sample code i am using.. CHECK_NO_FILES() { FARRAY=$1 echo "FARRAY = $FARRAY" echo "FARRAY = $FARRAY" ...............

7. Shell Programming and Scripting

Question about sorting -- how to pass an array to a function

Hi, guys I just wanted to sort the elements of an array ascendingly. I know the following code does work well: array=(13 435 8 23 100) for i in {0..4} do j=$((i+1)) while ] do if } -le ${array} ]] then : else min=${array} ${array}=${array} ${array}=$min fi...

8. Shell Programming and Scripting

Pass array to a function and display the array

Hi All I have multiple arrays like below. set -A val1 1 2 4 5 set -A val2 a b c d . . . Now i would like to pass the individual arrays one by one to a function and display/ do some action. Note : I am using ksh Can you please advise any solution... Thanks in advance.

9. Shell Programming and Scripting

Pass an array to awk to sequentially look for a list of items in a file

Hello, I need to collect some statistical results from a series of files that are being generated by other software. The files are tab delimited. There are 4 different sets of statistics in each file where there is a line indicating what the statistic set is, followed by 5 lines of values. It...

10. Shell Programming and Scripting

How to pass and read an array in ksh shell script function.?

I'm able to read & print an array in varaible called "filelist" I need to pass this array variable to a function called verify() and then read and loop through the passed array inside the function. Unfortunately it does not print the entire array from inside the funstion's loop. #/bin/ksh...

LEARN ABOUT DEBIAN

petal::i18n

Petal::I18N(3pm)					User Contributed Perl Documentation					  Petal::I18N(3pm)

NAME

       Petal::I18N - Attempt at implementing ZPT I18N for Petal

SYNOPSIS

       in your Perl code:

	 use Petal;
	 use Petal::TranslationService::Gettext;

	 my $translation_service = new Petal::TranslationService::Gettext (
	     locale_dir  => '/path/to/my/app/locale',
	     target_lang => gimme_target_lang(),
	 );

	 my $template = new Petal (
	     file => 'example.html',
	     translation_service => $translation_service
	 );

	 # we want to internationalize to the h4x0rz 31337 l4nGu4g3z. w00t!
	 my $translation_service = Petal::TranslationService::h4x0r->new();
	 my $template = new Petal (
	     file => 'silly_example.xhtml',
	     translation_service => $ts,
	 );

	 print $template->process ();

I18N Howto
   Preparing your templates:
       Say your un-internationalized template looks like this:

	 <html xmlns:tal="http://purl.org/petal/1.0/">
	   <body>
	     <img src="/images/my_logo.png"
		  alt="the logo of our organisation" />

	     <p>Hello,
		<span petal:content="user_name">Joe</span>.</p>

	     <p>How are you today?</p>
	   </body>
	 </html>

       You need to markup your template according to the ZPT I18N specification, which can be found at
       http://dev.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/ZPTInternationalizationSupport

	 <html xmlns:tal="http://purl.org/petal/1.0/"
	       xmlns:i18n="http://xml.zope.org/namespaces/i18n"
	       i18n:domain="my_app">
	   <body>
	     <img src="/images/my_logo.png"
		  alt="the logo of our organisation"
		  i18n:attributes="alt" />
	     <p i18n:translate="">Hello, <span petal:content="user_name">Joe</span>.</p>
	     <p i18n:translate="">How are you today?</p>
	   </body>
	 </html>

   Extracting I18N strings:
       Once your templates are marked up properly, you will need to use a tool to extract the I18N strings into .pot (po template) files. To my
       knowledge you can use i18ndude (standalone python executable), i18nextract.py (part of Zope 3), or I18NFool.

       I use i18ndude to find strings which are not marked up properly with i18n:translate attributes and I18NFool for extracting strings and
       managing .po files.

       Assuming you're using i18nfool:

	 mkdir -p /path/to/my/app/locale
	 cd /path/to/my/app/locale
	 i18nfool-extract /path/to/my/template/example.html
	 mkdir en
	 mkdir fr
	 mkdir es
	 i18nfool-update

       Then you translate the .po files into their respective target languages. When that's done, you type:

	 cd /path/to/my/app/locale
	 i18nfool-build

       And it builds all the .mo files.

   Making your application use a Gettext translation service:
       Previously you might have had:

	 use Petal;
	 # lotsa code here
	 my $template = Petal->new ('example.html');

       This needs to become:

	 use Petal;
	 use Petal::TranslationService::Gettext;
	 # lotsa code here
	 my $template = Petal->new ('example.html');
	 $template->{translation_service} = Petal::TranslationService::Gettext->new (
	     locale_dir  => '/path/to/my/app/locale',
	     target_lang => gimme_language_code(),
	 );

       Where gimme_language_code() returns a language code depending on LC_LANG, content-negotiation, config-file, or whatever mechanism you are
       using to decide which language is desired.

   And then?
       And then that's it! Your application should be easily internationalizable.  There are a few traps / gotchas thought, which are described
       below.

BUGS, TRAPS, GOTCHAS and other niceties
   Translation Phase
       The translation step takes place ONLY ONCE THE TEMPLATE HAS BEEN PROCESSED.

       So if you have:

	 <p i18n:translate="">Hello,
	   <span i18n:name="user_login" tal:replace="self/user_login">Joe</span>
	 </p>

       It most likely will not work because the tal:replace would remove the <span> tag and also the i18n:name in the process.

       This means that instead of receiving something such as:

	 Hello, ${user_login}

       The translation service would receive:

	 Hello, Fred Flintstone

       Or

	 Hello, Joe SixPack

       etc.

       To fix this issue, use tal:content instead of tal:replace and leave the span and its i18n:name attribute.

   Character sets
       I haven't worried too much about them (yet) so if you run into trouble join the Petal mailing list and we'll try to fix any issues
       together.

   Limitations
       At the moment, Petal::I18N supports the following constructs:

       xmlns:i18n="http://xml.zope.org/namespaces/i18n"
       i18n:translate
       i18n:domain
       i18n:name
       i18n:attribute

       It does *NOT* (well, not yet) support i18n:source, i18n:target or i18n:data.

perl v5.12.4							    2011-08-25							  Petal::I18N(3pm)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Can we pass array with call by value in function

Discussion started by: ranjithpr

2. Shell Programming and Scripting

Array split function & hashes

Discussion started by: novera

3. Shell Programming and Scripting

How to pass an array from SHELL to C function

Discussion started by: elthox

4. Shell Programming and Scripting

Find and split the list of files with suffiz of seg**

Discussion started by: Satish Shettar