Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Bash script with python slicing on multiple data files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash script with python slicing on multiple data files
# 1  
Old 05-15-2013
Bash script with python slicing on multiple data files
I have 2 files generated in linux that has common output and were produced across multiple hosts with the same setup/configs. These files do some simple reporting on resource allocation and user sessions. So, essentially, say, 10 hosts, with the same (2) system reporting in the files, so a combination of 20 files total.

Neither of the files contain any reference to the host name, but do contain resource info and correlative user info per machine.

I need to meet some basic reporting objectives that include:

- taking cumulative reporting data on the resource and ranking the machines by availability

- taking cumulative reporting data on the users and ranking on usage

I created a simple python script that runs through all the files and produces output that summarizes per machine, but I've not got the parsed data converted into a real data value, so its not sortable

I also think that the following needs to happen:

- merge data files in linux w/ piping

- append the machine name/# to the 1st column

- read that w/ python slicing (since the machine name will always be the first "X" # of characters in the line

- use a bash script, wrapped, then execute the python script after bash has merged all the data files

so something like

/bin/bash
merge
push to python
I might need to change my python script so that the function input isnt the machine name, but more like "big_data_file" etc

that make sense?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

Python script to run multiple command and append data in output csv file

Experts, I am writing a script and able to write only small piece of code and not able to collect logic to complete this task. In input file have to look for name like like this (BGL_HSR_901_1AG_A_CR9KTR10) before sh iss neors. Record this (BGL_HSR_901_1AG_A_CR9KTR10) in csv file Now have to... (0 Replies)
Discussion started by: as7951
0 Replies

2. UNIX for Beginners Questions & Answers

Using bash script : How to Import data from a dsv file into multiple tables in mysql

HI I have a dsv file that looks like: <<BOF>> record_number|id_number|first name|last name|msisdn|network|points|card number|gender 312|9101011234011|Test Junior|Smith|071 123 4321|MTN|73|1241551413214444|M 313|9012023213011|Bob|Smith|27743334321|Vodacom|3|1231233232323244|M... (4 Replies)
Discussion started by: tera
4 Replies

3. Shell Programming and Scripting

In PErl script: need to read the data one file and generate multiple files based on the data

We have the data looks like below in a log file. I want to generat files based on the string between two hash(#) symbol like below Source: #ext1#test1.tale2 drop #ext1#test11.tale21 drop #ext1#test123.tale21 drop #ext2#test1.tale21 drop #ext2#test12.tale21 drop #ext3#test11.tale21 drop... (5 Replies)
Discussion started by: Sanjeev G
5 Replies

4. Shell Programming and Scripting

Loop through multiple files in bash script

Hi Everybody, I'm a newbie to shell scripting, and I'd appreciate some help. I have a bunch of .txt files that have some unwanted content. I want to remove lines 1-3 and 1028-1098. #!/bin/bash for '*.txt' in <path to folder> do sed '1,3 d' "$f"; sed '1028,1098 d' "$f"; done I... (2 Replies)
Discussion started by: BabyNuke
2 Replies

5. Programming

Python script for extracting data using two files

Hello, I have two files. File 1 is a list of interested IDs Ex1 Ex2 Ex3File 2 is the original file with over 8000 columns and 20 millions rows and is a compressed file .gz Ex1 xx xx xx xx .... Ex2 xx xx xx xx .... Ex2 xx xx xx xx ....Now I need to extract the information for all the IDs of... (4 Replies)
Discussion started by: nans
4 Replies

6. Shell Programming and Scripting

Bash script to copy timestamps of multiple files

Hi, I have a bunch of media files in a directory that have been converted (from MTS to MOV format), so my directory contains something like this: clip1.mts clip1.mov clip2.mts clip2.mov The problem is that the .mov files that have been created have the timestamps of the conversion task,... (2 Replies)
Discussion started by: Krakus
2 Replies

7. Shell Programming and Scripting

bash script to compile multiple .c files with some options

I'm trying to write a bash script and call it "compile" such that running it allows me to compile multiple files with the options "-help," "-backup," and "-clean". I've got the code for the options written, i just can't figure out how to read the input string and then translate that into option... (5 Replies)
Discussion started by: travis.batzer
5 Replies

8. Shell Programming and Scripting

How to write bash script to explode multiple zip files

I have a directory full of zip files. How would I write a bash script to enumerate all the zip files, remove the ".zip" from the file name, create a directory by that name and unzip each zip file into its corresponding directory? Thanks! Siegfried (3 Replies)
Discussion started by: siegfried
3 Replies

9. Shell Programming and Scripting

moving multiple folders/files in subversion using bash script

Hi, I'm new here an dlearning a lot from this forum. i didnt find any solution for this in the forum. I have already checked in folders in subversion named HTT01,... HTT21.. and have files in each folder like below: HTT01/HTT01_00000.hex HTT01/HTT01_00000_fb_result.hex... (2 Replies)
Discussion started by: ravishan21
2 Replies

10. Shell Programming and Scripting

Script to delete all data from multiple files

its urgent!!!!!!111 i need a script which can delete data from multiple files. plz if anybody knows the script plz write a mail to me : (Email addresses are not allowed) (5 Replies)
Discussion started by: uni_ajay_r
5 Replies
Login or Register to Ask a Question

LEARN ABOUT OSX

pythonw

PYTHON(1)						    BSD General Commands Manual 						 PYTHON(1)

NAME

     python, pythonw -- an interpreted, interactive, object-oriented programming language

SYNOPSIS

     python ...
     pythonw ...

DESCRIPTION

     To support multiple versions, the programs named python and pythonw now just select the real version of Python to run, depending on various
     settings.	(As of Python 2.5, python and pythonw are interchangeable; both execute Python in the context of an application bundle, which
     means they have access to the Graphical User Interface; thus both can, when properly programmed, display windows, dialogs, etc.)  The current
     supported versions are 2.6 and 2.7, with the default being 2.6.  Use

	   % man python2.6
	   % man python2.7
	   % man pythonw2.6
	   % man pythonw2.7

     to see the man page for a specific version.  Without a version specified,

	   % man pydoc

     and the like, will show the man page for the (unmodified) default version of Python (2.6).  To see the man page for a specific version, use,
     for example,

	   % man pydoc2.7

CHANGING THE DEFAULT PYTHON

     Using

	   % defaults write com.apple.versioner.python Version 2.7

     will make version 2.7 the user default when running the both the python and pythonw commands (versioner is the internal name of the version-
     selection software used).

     To set a system-wide default, replace 'com.apple.versioner.python' with '/Library/Preferences/com.apple.versioner.python' (admin privileges
     will be required).

     The environment variable VERSIONER_PYTHON_VERSION can also be used to set the python and pythonw version:

	   % export VERSIONER_PYTHON_VERSION=2.7 # Bourne-like shells
		or
	   % setenv VERSIONER_PYTHON_VERSION 2.7 # C-like shells
	   % python ...

     This environment variable takes precedence over the preference file settings.

64-BIT SUPPORT
     Versions 2.6 and 2.7 support 64-bit execution (which is on by default).

     Like the version of Python, the python command can select between 32 and 64-bit execution (when both are available).  Use:

	   % defaults write com.apple.versioner.python Prefer-32-Bit -bool yes

     to make 32-bit execution the user default (using '/Library/Preferences/com.apple.versioner.python' will set the system-wide default).  The
     environment variable VERSIONER_PYTHON_PREFER_32_BIT can also be used (has precedence over the preference file):

	   % export VERSIONER_PYTHON_PREFER_32_BIT=yes # Bourne-like shells
		or
	   % setenv VERSIONER_PYTHON_PREFER_32_BIT yes # C-like shells

     Again, the preference setting and environmental variable applies to both python and pythonw.

USING A SPECIFIC VERSION

     Rather than using the python command, one can use a specific version directly.  For example, running python2.7 from the command line will run
     the 2.7 version of Python, independent of what the default version of Python is.

     One can use a specific version of Python on the #! line of a script, but that may have portability and future compatibility issues.

     Note that the preference files and environment variable that apply to the python command, do not apply when running a specific version of
     Python.  In particular, running python2.6 will always default to 64-bit execution (unless one uses the arch(1) command to specifically select
     a 32-bit architecture).

SEE ALSO

     python2.6(1), python2.7(1), pythonw2.6(1), pythonw2.7(1), arch(1)

BSD
								   Aug 10, 2008 							       BSD