Sponsored Content
Top Forums Shell Programming and Scripting Split list of files into an array and pass to function Post 302931535 by LMHmedchem on Tuesday 13th of January 2015 03:18:28 PM
Old 01-13-2015
This is what I have set up instead of passing the array.

calling code
Code:
# the number of availalbe cores
if [ "$CORES" == "quad" ]; then

   # create 4 arrays, each with ~25% of filenames
   NUM_FILE_LISTS='4'
   PROCESSED='0'

   # get size of each subset
    BASE_LIST_SIZE=$(((NUM_INPUT_FILES) / NUM_FILE_LISTS))
   LEFTOVER=$((NUM_INPUT_FILES % NUM_FILE_LISTS))

   # set up start elements and number of elements for all lists
   # list 0
   START_ELEMENT_0='0'
   NUMBER_OF_ELEMENTS_0=$((BASE_LIST_SIZE + (LEFTOVER > 0)))
   # keep track of number of files processed
   let "PROCESSED=$PROCESSED+$NUMBER_OF_ELEMENTS_0"
      
   # list 1   
   START_ELEMENT_1=$PROCESSED
   #let "START_ELEMENT_1=$START_ELEMENT_0+$NUMBER_OF_ELEMENTS_0"
   NUMBER_OF_ELEMENTS_1=$((BASE_LIST_SIZE + (LEFTOVER > 1)))
   let "PROCESSED=$PROCESSED+$NUMBER_OF_ELEMENTS_1"
 
   # list 2  
   START_ELEMENT_2=$PROCESSED
   NUMBER_OF_ELEMENTS_2=$((BASE_LIST_SIZE + (LEFTOVER > 2)))
   # keep track of number of files processed
   let "PROCESSED=$PROCESSED+$NUMBER_OF_ELEMENTS_2"
 
   # list 3  
   START_ELEMENT_3=$PROCESSED
   # assign the rest to this list
   let "NUMBER_OF_ELEMENTS_3=$NUM_INPUT_FILES-$PROCESSED"
   # keep track of number of files processed
   let "PROCESSED=$PROCESSED+$NUMBER_OF_ELEMENTS_3"

      # call functions to process stats
      run_stats_program  $SET  $FOLD  $START_ELEMENT_0  $NUMBER_OF_ELEMENTS_0 &
      # to prevent terminal overrun
      sleep 2
      run_stats_program  $SET  $FOLD  $START_ELEMENT_1  $NUMBER_OF_ELEMENTS_1 &
      sleep 2
      run_stats_program  $SET  $FOLD  $START_ELEMENT_2  $NUMBER_OF_ELEMENTS_2 &
      sleep 2
      run_stats_program  $SET  $FOLD  $START_ELEMENT_3  $NUMBER_OF_ELEMENTS_3 &
      sleep 2
      # wait untill subshells have returned
      wait

 fi

called function
Code:
function run_stats_program {

   # function args
   SET_F=$1
   FOLD_F=$2
   START_ELEMENT_F=$3
   NUMBER_OF_ELEMENTS_F=$4
   
   # get list of stats input files in fold directory
   STATS_INPUT_FILENAMES_F=($(ls  './'$SET_F'/'$FOLD_F'/'*'in.txt'))
 
   # create file list as subest of STATS_INPUT_FILENAMES_F
   FILE_LIST=("${STATS_INPUT_FILENAMES_F[@]:$START_ELEMENT_F:$NUMBER_OF_ELEMENTS_F}")

   for INPUT_FILE in "${FILE_LIST[@]}"
   do
      echo $INPUT_FILE
   done
}

All this does at this point is print the filenames. In the end, this will process the 4 file lists in 4 subshells. Processing involved calling a c++ widget to process each file. This setup allows 4 instances of the c++ app to run simultaneously and use availalble CPU resources. There will be a similar code block for hex core.

I get this this is written in long form at the moment. It would be nice for the code to be a bit more compact and elegant, but I don't see a clear way to put the function calls in a loop or something like that.

LMHmedchem

Last edited by LMHmedchem; 01-13-2015 at 05:52 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Can we pass array with call by value in function

I want to pass an array in my function, And my function will be changing the elements of the array in the fuction, but it should not affect the values in my array variable of main function (1 Reply)
Discussion started by: ranjithpr
1 Replies

2. Shell Programming and Scripting

Array split function & hashes

Hi, If this is the array that is being returned to me: How would I get the values for each of the 3 records? This works for 1 Record: foreach $item (@results) { ($id, $id2, $name, $date, $email) = split(/\|/, $item, 5); print "$name<br>"; } (2 Replies)
Discussion started by: novera
2 Replies

3. Shell Programming and Scripting

How to pass an array from SHELL to C function

Hi, I have an output generated from a shell script like; 0x41,0xF2,0x59,0xDD,0x86,0xD3,0xEF,0x61,0xF2 How can I pass this value to the C function, as below; int main(int argc, char *argv) { unsigned char hellopdu={above value}; } Regards Elthox (1 Reply)
Discussion started by: elthox
1 Replies

4. Shell Programming and Scripting

Find and split the list of files with suffiz of seg**

Hi,. I am writing a script to get the new files and split them. Requirement Find the new files under the path "/wload/scmp/app/data/OAS" (There are 5 sub folders). Gunzip the files which are having .gz suffix. Put the list of files in the filename in the format... (0 Replies)
Discussion started by: Satish Shettar
0 Replies

5. Shell Programming and Scripting

Split the file and access that files through array and loop

Hi All, the below is my requirement.. i need to split the file based on line and put that files in a array and need to access that files through loop finally i should send the files through mail.. how can we achieve this ..I am new to shell script please guide me.. I am using KSH.. ... (11 Replies)
Discussion started by: kalidoss
11 Replies

6. Shell Programming and Scripting

How to pass an array to a function in shell script.?

hi, I have a array say SAP_ARRAY="s1.txt" SAP_ARRAY="s2.txt" how can i pass this full array to a function. here is the sample code i am using.. CHECK_NO_FILES() { FARRAY=$1 echo "FARRAY = $FARRAY" echo "FARRAY = $FARRAY" ............... (5 Replies)
Discussion started by: Little
5 Replies

7. Shell Programming and Scripting

Question about sorting -- how to pass an array to a function

Hi, guys I just wanted to sort the elements of an array ascendingly. I know the following code does work well: array=(13 435 8 23 100) for i in {0..4} do j=$((i+1)) while ] do if } -le ${array} ]] then : else min=${array} ${array}=${array} ${array}=$min fi... (5 Replies)
Discussion started by: franksunnn
5 Replies

8. Shell Programming and Scripting

Pass array to a function and display the array

Hi All I have multiple arrays like below. set -A val1 1 2 4 5 set -A val2 a b c d . . . Now i would like to pass the individual arrays one by one to a function and display/ do some action. Note : I am using ksh Can you please advise any solution... Thanks in advance. (7 Replies)
Discussion started by: Girish19
7 Replies

9. Shell Programming and Scripting

Pass an array to awk to sequentially look for a list of items in a file

Hello, I need to collect some statistical results from a series of files that are being generated by other software. The files are tab delimited. There are 4 different sets of statistics in each file where there is a line indicating what the statistic set is, followed by 5 lines of values. It... (8 Replies)
Discussion started by: LMHmedchem
8 Replies

10. Shell Programming and Scripting

How to pass and read an array in ksh shell script function.?

I'm able to read & print an array in varaible called "filelist" I need to pass this array variable to a function called verify() and then read and loop through the passed array inside the function. Unfortunately it does not print the entire array from inside the funstion's loop. #/bin/ksh... (5 Replies)
Discussion started by: mohtashims
5 Replies
agent(3pm)						User Contributed Perl Documentation						agent(3pm)

NAME
NetSNMP::agent - Perl extension for the net-snmp agent. SYNOPSIS
use NetSNMP::agent; my $agent = new NetSNMP::agent('Name' => 'my_agent_name'); DESCRIPTION
This module implements an API set to make a SNMP agent act as a snmp agent, a snmp subagent (using the AgentX subagent protocol) and/or embedded perl-APIs directly within the traditional net-snmp agent demon. Also see the tutorial about the genaral Net-SNMP C API, which this module implements in a perl-way, and a perl specific tutorial at: http://www.net-snmp.org/tutorial-5/toolkit/ EXAMPLES
Sub-agent example use NetSNMP::agent (':all'); use NetSNMP::ASN qw(ASN_OCTET_STR); my $value = "hello world"; sub myhandler { my ($handler, $registration_info, $request_info, $requests) = @_; my $request; for($request = $requests; $request; $request = $request->next()) { my $oid = $request->getOID(); if ($request_info->getMode() == MODE_GET) { # ... generally, you would calculate value from oid if ($oid == new NetSNMP::OID(".1.3.6.1.4.1.8072.9999.9999.7375.1.0")) { $request->setValue(ASN_OCTET_STR, $value); } } elsif ($request_info->getMode() == MODE_GETNEXT) { # ... generally, you would calculate value from oid if ($oid < new NetSNMP::OID(".1.3.6.1.4.1.8072.9999.9999.7375.1.0")) { $request->setOID(".1.3.6.1.4.1.8072.9999.9999.7375.1.0"); $request->setValue(ASN_OCTET_STR, $value); } } elsif ($request_info->getMode() == MODE_SET_RESERVE1) { if ($oid != new NetSNMP::OID(".1.3.6.1.4.1.8072.9999.9999.7375.1.0")) { # do error checking here $request->setError($request_info, SNMP_ERR_NOSUCHNAME); } } elsif ($request_info->getMode() == MODE_SET_ACTION) { # ... (or use the value) $value = $request->getValue(); } } } my $agent = new NetSNMP::agent( # makes the agent read a my_agent_name.conf file 'Name' => "my_agent_name", 'AgentX' => 1 ); $agent->register("my_agent_name", ".1.3.6.1.4.1.8072.9999.9999.7375", &myhandler); my $running = 1; while($running) { $agent->agent_check_and_process(1); } $agent->shutdown(); Embedded agent example # place this in a .pl file, and then in your snmpd.conf file put: # perl do '/path/to/file.pl'; use NetSNMP::agent; my $agent; sub myhandler { my ($handler, $registration_info, $request_info, $requests) = @_; # ... } $agent = new NetSNMP::agent( 'Name' => 'my_agent_name' ); $agent->register("my_agent_name", ".1.3.6.1.4.1.8072.9999.9999.7375", &myhandler); $agent->main_loop(); CONSTRUCTOR
new ( OPTIONS ) This is the constructor for a new NetSNMP::agent object. Possible options are: Name - Name of the agent (optional, defaults to "perl") (The snmp library will read a NAME.conf snmp configuration file based on this argument.) AgentX - Make us a sub-agent (0 = false, 1 = true) (The Net-SNMP master agent must be running first) Ports - Ports this agent will listen on (EG: "udp:161,tcp:161") Example: $agent = new NetSNMP::agent( 'Name' => 'my_agent_name', 'AgentX' => 1 ); METHODS
register (NAME, OID, &handler_routine ) Registers the callback handler with given OID. $agent->register(); A return code of 0 indicates no error. agent_check_and_process ( BLOCKING ) Run one iteration of the main loop. BLOCKING - Blocking or non-blocking call. 1 = true, 0 = false. $agent->agent_check_and_process(1); main_loop () Runs the agent in a loop. Does not return. shutdown () Nicely shuts down the agent or sub-agent. $agent->shutdown(); HANDLER CALLBACKS
handler ( HANDLER, REGISTRATION_INFO, REQUEST_INFO, REQUESTS ) The handler is called with the following parameters: HANDLER - FIXME REGISTRATION_INFO - what are the correct meanings of these? REQUEST_INFO - REQUESTS - Example handler: sub myhandler { my ($handler, $reg_info, $request_info, $requests) = @_; # ... } The handler subroutine will be called when a SNMP request received by the agent for anything below the registered OID. The handler is passed 4 arguments: $handler, $registration_info, $request_info, $requests. These match the arguments passed to the C version of the same API. Note that they are not entirely complete objects but are functional "enough" at this point in time. $request_info object functions getMode () Returns the mode of the request. See the MODES section for list of valid modes. $mode = $request->getMode(); getRootOID () Returns a NetSNMP::OID object that describes the registration point that the handler is getting called for (in case you register one handler function with multiple OIDs, which should be rare anyway) $root_oid = $request->getRootOID(); $request object functions next () Returns the next request in the list or undef if there is no next request. $request = $request->next(); getOID () Returns the oid of the request (a NetSNMP::OID class). $oid = $request->getOID(); setOID (new NetSNMP::OID("someoid")) Sets the OID of the request to a passed oid value. This should generally only be done during handling of GETNEXT requests. $request->setOID(new NetSNMP::OID("someoid")); getValue () Returns the value of the request. Used for example when setting values. $value = $request->getValue(); FIXME: how to get the type of the value? Is it even available? [Wes: no, not yet.] setValue ( TYPE, DATA ) Sets the data to be returned to the daemon. Returns 1 on success, 0 on error. TYPE - Type of the data. See NetSNMP::ASN for valid types. DATA - The data to return. $ret = $request->setValue(ASN_OCTET_STR, "test"); setError ( REQUEST_INFO, ERROR_CODE ) Sets the given error code for the request. See the ERROR CODES section for list of valid codes. $request->setError($request_info, SNMP_ERR_NOTWRITABLE); getProcessed () The processed flag indicates that a request does not need to be dealt with because someone else (a higher handler) has dealt with it already. $processed = $request->getProcessed(); setProcessed ( PROCESSED ) Sets the processed flag flag in the request. You generally should not have to set this yourself. PROCESSED - 0 = false, 1 = true $request->setProcessed(1); getDelegated () If you can handle a request in the background or at a future time (EG, you're waiting on a file handle, or network traffic, or ...), the delegated flag can be set in the request. When the request is processed in the future the flag should be set back to 0 so the agent will know that it can wrap up the original request and send it back to the manager. This has not been tested within perl, but it hopefully should work. $delegated = $request->getDelegated(); setDelegated ( DELEGATED ) Sets the delegated flag. DELEGATED - 0 = false, 1 = true $request->setDelegated(1); getRepeat () The repeat flag indicates that a getbulk operation is being handled and this indicates how many answers need to be returned. Generally, if you didn't register to directly handle getbulk support yourself, you won't need to deal with this value. $repeat = $request->getRepeat(); setRepeat ( REPEAT ) Sets the repeat count (decrement after answering requests if you handle getbulk requests yourself) REPEAT - repeat count FIXME $request->setRepeat(5); MODES
MODE_GET MODE_GETBULK MODE_GETNEXT MODE_SET_ACTION MODE_SET_BEGIN MODE_SET_COMMIT MODE_SET_FREE MODE_SET_RESERVE1 MODE_SET_RESERVE2 MODE_SET_UNDO ERROR CODES
SNMP_ERR_NOERROR SNMP_ERR_TOOBIG SNMP_ERR_NOSUCHNAME SNMP_ERR_BADVALUE SNMP_ERR_READONLY SNMP_ERR_GENERR SNMP_ERR_NOACCESS SNMP_ERR_WRONGTYPE SNMP_ERR_WRONGLENGTH SNMP_ERR_WRONGENCODING SNMP_ERR_WRONGVALUE SNMP_ERR_NOCREATION SNMP_ERR_INCONSISTENTVALUE SNMP_ERR_RESOURCEUNAVAILABLE SNMP_ERR_COMMITFAILED SNMP_ERR_UNDOFAILED SNMP_ERR_AUTHORIZATIONERROR SNMP_ERR_NOTWRITABLE AUTHOR
Please mail the net-snmp-users@lists.sourceforge.net mailing list for help, questions or comments about this module. Module written by: Wes Hardaker <hardaker@users.sourceforge.net> Documentation written by: Toni Willberg <toniw@iki.fi> Wes Hardaker <hardaker@users.sourceforge.net> SEE ALSO
NetSNMP::OID(3), NetSNMP::ASN(3), perl(1). perl v5.14.2 2009-10-29 agent(3pm)
All times are GMT -4. The time now is 07:15 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy