Need help with some iterative file processing Post: 302239103

Sponsored Content

Top Forums Shell Programming and Scripting Need help with some iterative file processing Post 302239103 by inditopgun on Tuesday 23rd of September 2008 12:08:43 AM

09-23-2008

Registered User

Need help with some iterative file processing

Gurus - Please help with this urgent situation.

I have the following problem to solve using a shell script:

1. There are about 13 files named CONTAINER_1.lst, CONTAINER_2.lst, CONTAINER_3.lst .....CONTAINER_13.lst
2. Each of these files contain about 8 lines (in most cases) containing container instance names like CONTAINER_X_InstanceY where X will correspond to the container in question (ranging from 1 thru 13) and Y is some random variable value
3. For each of the 13 containers, there is a file named, CONTAINER_X_commands.sh where X corresponds to the container number. Each of the CONTAINER_X_commands.sh file contains about 3 to 5 commands with a fixed embedded string called "HELLOWORLD"
4. For each container name, I would like to first read all the CONTAINER_X_InstanceY variables (one at a time), then repeat each of command in the CONTAINER_X_commands.sh file, N number of times where N is the number of lines in the CONTAINER_X.lst (equal to the number of container instances) Once the yanking/creating multiple copies of commands is done, I would like to replace each occurrence of the "HELLOWORD" string with CONTAINER_X_InstanceY container instance name variable

Here is the code I have written so far :
for container_list in `ls -1 CONTAINER*.lst | sort -u`
do
container_name=`echo $container_list | sed 's;$.*$\..*;\1;'`
cmd_suffix=_commands.sh
command_file=$container_name$cmd_suffix
for container_instance_name in `cat $container_list`
do
for command in `cat $command_file`
do
echo $command | sed 's/HELLOWORLD/$container_instance_name/g;' >> /tmp/all_commands_in_singlefile.lst
done
done
done

There are 3 problems in my output :
1. It's printing each word in the commands.sh file on separate line. I would like to keep each command intact on it's own line without breaking each command word on separate line (I think this is an issue with my innermost for loop)
2. the sed is actually replacing HELLOWORLD with the literal string "$container_instnace_name" instead of using the actual variable value
3. I am also suspecting that the output file is containing lesser number of rows of commands, than I would actually anticipate. But this is yet to be substantiated because of newlines introduced in my output.

Any help will be greatly appreciated!
indi

inditopgun

View Public Profile for inditopgun

Find all posts by inditopgun

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Have a shell script check for a file to exist before processing another file

I have a shell script that runs all the time looking for a certain type of file and then it processes the file through a series of other scripts. The script is watching a directory that has files uploaded to it via SFTP. It already checks the size of the file to make sure that it is not still...

2. Shell Programming and Scripting

Iterative operation

grep -o '\{1,3\}\.\{1,3\}\.\{1,3\}\.\{1,3\}' then how do i iterate the file names??

3. Shell Programming and Scripting

Iterative statement to cut values from a line

Hi I am new to shell scripting and trying to get values from a text file, I have a text file with values seperated with "|". like aga|120220090525|120220090525|120220090525|120220090530 bab|120220090530|120220090530|120220090535|120220090535|120220090535...

4. Shell Programming and Scripting

how to change the current file processing to some other random file in awk ?

Hello, say suppose i am processing an file emp.dat the field of which are deptno empno empname etc now say suppose i want to change the file to emp.lst then how can i do it? Here i what i attempted but in vain BEGIN{ system("sort emp.dat > emp.lst") FILENAME="emp.lst" } { print...

5. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log...

6. Shell Programming and Scripting

iterative parsing

I have always struggled when parsing a file vertically vs. by line horizontally. Can't seem to get my head around the concept. Here again I need to convert vertical output to horizontal output. original output root@acuransx:bpplsched 2000-STAND3 -v -M acuransx -l <2>bpplsched: INITIATING:...

7. Shell Programming and Scripting

Recursive file processing from a path and printing output in a file

Hi All, The script below read the path and searches for the directories/subdirectories and for the files. If files are found in the sub directories then read the content of the all files and put the content in csv(comma delimted) format and the call the write to xml function to write the std...

8. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M...

9. UNIX for Dummies Questions & Answers

awk - Rename output file, after processing, same as input file

I have one input file ABC.txt and one output DEF.txt. After the ABC is processed and created output, I want to rename ABC.txt to ABC.orig and DEF to ABC.txt. Currently when I am doing this, it does not process the input file as it cannot read and write to the same file. How can I achieve this? ...

LEARN ABOUT CENTOS

ibdmchk

IBDMCHK(1)						       IB DATA MODEL PACKAGE							IBDMCHK(1)

NAME

       IBDM - Network Checker and Attributes Analyzer

SYNOPSYS

       ibdmchk has two operation modes: design and verification.

       DESIGN:

       ibdmchk [-v][-h][-u][-r <roots file>] -t <topology file> -n <SM Node>
	 -p <SM Port> [-e] [-l <lmc>]

       VERIFICATION:

       ibdmchk [-v][-h][-r <roots file>] [-s <subnet file>] [-f <fdb file>]
	 [-m <mcfdbs file> [-l <lmc>] [-u] [-M] [-a]
	 [-c <path sl file>] [-d <sl2vl tables file>]

DESIGN MODE DESCRIPTION

       The Design mode is intended to be used before the cluster is built. It provides basic checks of the specified network as described by a
       topology file. After simulating the SM LID assignment and routing algorithms it provides reports of the (H)CA to (H)CA paths depth his-
       togram and credit deadlock potential in the resulting routing scheme.

ARGUMENTS
:
       -t|--topo <topo file>
	      The topology file specifying the network. See man ibdm-topology-file

       -n|--node <SM Node>
	      The name of the Subnet Manager node (syntax: <Topo-File-System>/U1)

       -p|--port <SM Port>
	      The port number by which the SM nodes is attached to the fabric.

OPTIONS

       -v|--verbose
	      Verbsoe mode

       -h|--help
	      Provides this help message

       -l|--lmc <lmc>
	      LMC value > 0 means assigning 2^lmc lids to each port.

       -e|--enh
	      Use enhanced routing algorithm when LMC > 0 and report the resulting paths correlation (using same system/node) histogram

       -r|--roots <roots file> A file with all the roots node names (one on each line).

VERIFICATION MODE DESCRIPTION

       After the cluster is built and OpenSM is run (using flag -D 0x43) it reports the subnet and FDB tables into the files osm-subnet.lst,
       osm.fdbs and osm.fdbs in /var/log/ (or subnet.lst, osm.fdbs and osm.mcfdbs into /var/cache/ibutils in older OpenSM versions). ibdiagnet is
       also producing the same files in its output directory.  Based on these files the utility checks all CA to CA connectivity. Further analysis
       for credit deadlock potential is performed and reported.  In case of an LMC > 0 it reports histograms for how many systems and nodes are
       common between the different paths for the same port pairs.

ARGUMENTS

       -l|--lmc <lmc>
	      The LMC value used while running OpenSM. Mandatory if not the default 0.

OPTIONS

       -v|--verbose
	      Verbsoe mode

       -h|--help
	      Provides this help message

       -s|--subnet <file>
	      OpenSM subnet.lst file (default is /var/log/osm-subnet.lst or /var/cache/ibutils/subnet.lst)

       -f|--fdb <file>
	      OpenSM dump of Ucast LFDB. Use -D 0x41 to generate it (default is /var/log/osm.fdbs or /var/cache/ibutils/osm.fdbs).

       -m|--mcfdb <file>
	      OpenSM dump of Multicast LFDB. Use -D 0x41 to generate it (default is /var/log/osm.mcfdbs or /var/cache/ibutils/osm.mcfdbs).

       -c|--psl <file>
	      Source Guid to DLID -> Path SL mapping.  Each line holds: src_node_guid dlid sl.	Used by the credit loop check to accuartly model
	      SL/VL based routing.

       -d|--slvl <file>
	      SL2VL mapping of all switces.  Each line holds: swguid iport oport 0x(sl0)(sl1) 0x(sl2)(sl3)...  Used by the credit loop check to
	      accuartly model SL/VL based routing.

       -r|--roots <roots file>
	      A file holding all root nodes guids (one per line).

       -u|--updn
	      Use Up/Down based credit loop check algorithm instead of the generic one.

       -M|--MFT
	      Include multicast routing in credit loops analysis.

       -a|--all
	      Verify not only CA-CA paths but also SW-SW, CA-SW and SW-CA.  Both check for FDB path connectivity and also for credit loops.

AUTHOR

       Eitan Zahavi, Mellanox Technologies LTD, eitan@mellanox.co.il

IBDM 1.0							    2009-03-17								IBDMCHK(1)