Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Bash script with python slicing on multiple data files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash script with python slicing on multiple data files
# 1  
Old 05-15-2013
Bash script with python slicing on multiple data files
I have 2 files generated in linux that has common output and were produced across multiple hosts with the same setup/configs. These files do some simple reporting on resource allocation and user sessions. So, essentially, say, 10 hosts, with the same (2) system reporting in the files, so a combination of 20 files total.

Neither of the files contain any reference to the host name, but do contain resource info and correlative user info per machine.

I need to meet some basic reporting objectives that include:

- taking cumulative reporting data on the resource and ranking the machines by availability

- taking cumulative reporting data on the users and ranking on usage

I created a simple python script that runs through all the files and produces output that summarizes per machine, but I've not got the parsed data converted into a real data value, so its not sortable

I also think that the following needs to happen:

- merge data files in linux w/ piping

- append the machine name/# to the 1st column

- read that w/ python slicing (since the machine name will always be the first "X" # of characters in the line

- use a bash script, wrapped, then execute the python script after bash has merged all the data files

so something like

/bin/bash
merge
push to python
I might need to change my python script so that the function input isnt the machine name, but more like "big_data_file" etc

that make sense?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

Python script to run multiple command and append data in output csv file

Experts, I am writing a script and able to write only small piece of code and not able to collect logic to complete this task. In input file have to look for name like like this (BGL_HSR_901_1AG_A_CR9KTR10) before sh iss neors. Record this (BGL_HSR_901_1AG_A_CR9KTR10) in csv file Now have to... (0 Replies)
Discussion started by: as7951
0 Replies

2. UNIX for Beginners Questions & Answers

Using bash script : How to Import data from a dsv file into multiple tables in mysql

HI I have a dsv file that looks like: <<BOF>> record_number|id_number|first name|last name|msisdn|network|points|card number|gender 312|9101011234011|Test Junior|Smith|071 123 4321|MTN|73|1241551413214444|M 313|9012023213011|Bob|Smith|27743334321|Vodacom|3|1231233232323244|M... (4 Replies)
Discussion started by: tera
4 Replies

3. Shell Programming and Scripting

In PErl script: need to read the data one file and generate multiple files based on the data

We have the data looks like below in a log file. I want to generat files based on the string between two hash(#) symbol like below Source: #ext1#test1.tale2 drop #ext1#test11.tale21 drop #ext1#test123.tale21 drop #ext2#test1.tale21 drop #ext2#test12.tale21 drop #ext3#test11.tale21 drop... (5 Replies)
Discussion started by: Sanjeev G
5 Replies

4. Shell Programming and Scripting

Loop through multiple files in bash script

Hi Everybody, I'm a newbie to shell scripting, and I'd appreciate some help. I have a bunch of .txt files that have some unwanted content. I want to remove lines 1-3 and 1028-1098. #!/bin/bash for '*.txt' in <path to folder> do sed '1,3 d' "$f"; sed '1028,1098 d' "$f"; done I... (2 Replies)
Discussion started by: BabyNuke
2 Replies

5. Programming

Python script for extracting data using two files

Hello, I have two files. File 1 is a list of interested IDs Ex1 Ex2 Ex3File 2 is the original file with over 8000 columns and 20 millions rows and is a compressed file .gz Ex1 xx xx xx xx .... Ex2 xx xx xx xx .... Ex2 xx xx xx xx ....Now I need to extract the information for all the IDs of... (4 Replies)
Discussion started by: nans
4 Replies

6. Shell Programming and Scripting

Bash script to copy timestamps of multiple files

Hi, I have a bunch of media files in a directory that have been converted (from MTS to MOV format), so my directory contains something like this: clip1.mts clip1.mov clip2.mts clip2.mov The problem is that the .mov files that have been created have the timestamps of the conversion task,... (2 Replies)
Discussion started by: Krakus
2 Replies

7. Shell Programming and Scripting

bash script to compile multiple .c files with some options

I'm trying to write a bash script and call it "compile" such that running it allows me to compile multiple files with the options "-help," "-backup," and "-clean". I've got the code for the options written, i just can't figure out how to read the input string and then translate that into option... (5 Replies)
Discussion started by: travis.batzer
5 Replies

8. Shell Programming and Scripting

How to write bash script to explode multiple zip files

I have a directory full of zip files. How would I write a bash script to enumerate all the zip files, remove the ".zip" from the file name, create a directory by that name and unzip each zip file into its corresponding directory? Thanks! Siegfried (3 Replies)
Discussion started by: siegfried
3 Replies

9. Shell Programming and Scripting

moving multiple folders/files in subversion using bash script

Hi, I'm new here an dlearning a lot from this forum. i didnt find any solution for this in the forum. I have already checked in folders in subversion named HTT01,... HTT21.. and have files in each folder like below: HTT01/HTT01_00000.hex HTT01/HTT01_00000_fb_result.hex... (2 Replies)
Discussion started by: ravishan21
2 Replies

10. Shell Programming and Scripting

Script to delete all data from multiple files

its urgent!!!!!!111 i need a script which can delete data from multiple files. plz if anybody knows the script plz write a mail to me : (Email addresses are not allowed) (5 Replies)
Discussion started by: uni_ajay_r
5 Replies
Login or Register to Ask a Question

LEARN ABOUT DEBIAN

dh_pycentral

DH_PYCENTRAL(1) 						     Debhelper							   DH_PYCENTRAL(1)

NAME

       dh_pycentral - use the python-central framework to handle Python modules and extensions

SYNOPSIS

       dh_pycentral [debhelper options] [-n] [-Xitem] [-V version] [module dirs ...]

DESCRIPTION

       dh_pycentral is a debhelper program that will scan your package, detect public Python modules and move them in /usr/share/pycentral so that
       python-central can byte-compile those for all supported Python versions.  Extensions are kept into the original installation location.

       Moving the files to the pycentral location and adding symbolic links to /usr/lib/pythonX.Y/*-packages can be done by setting the
       environment varibale DH_PYCENTRAL to a string containing the string include-links.

       Moving the files to the pycentral location can be disabled by setting the environment varibale DH_PYCENTRAL to a string containing the
       string nomove.

       The functionality to shorten the time of unavailabilty of files during unpack and configure has been removed (symlinking files in the
       preinst and not removing the symlinked files on upgrade) in version 0.6.9.

       You must have filled the XS-Python-Version header to indicate the set of python versions that are going to be supported. dh_pycentral
       expects the XB-Python-Version for each binary package it is supposed to work on.

       dh_pycentral will also generate substitution variables: the ${python:Provides} variable will contain versioned provides of the package (if
       the package's name starts with "python-"). A python-foo package could provide "python2.3-foo" and "python2.4-foo" at the same time. Python
       extensions have to provide those whereas it's only option for pure python modules.

       The ${python:Versions} variable should be used to provide the required XB-Python-Version field listing the python versions supported by the
       package.

OPTIONS

       module dirs
	   If your package installs python modules in non-standard directories, you can make dh_pycentral check those directories by passing their
	   names on the command line. By default, it will check /usr/lib/$PACKAGE, /usr/share/$PACKAGE, /usr/lib/games/$PACKAGE,
	   /usr/share/games/$PACKAGE, /usr/lib/python?.?/site-packages and /usr/lib/python?.?/dist-packages.

	   Note: only /usr/lib/python?.?/site-packages and the extra names on the command line are searched for binary (.so) modules.

       -V version
	   If the .py files your package ships are meant to be used by a specific pythonX.Y version, you can use this option to specify the
	   desired version, such as 2.3. Do not use if you ship modules in /usr/lib/site-python.

	   With the new policy, this option is mostly deprecated. Use the XS-Python-Field to indicate that you're using a specific python version.

       -n, --noscripts
	   Do not modify postinst/postrm scripts.

       -Xitem, --exclude=item
	   Exclude files that contain "item" anywhere in their filename from being taken into account to generate the python dependency. You may
	   use this option multiple times to build up a list of things to exclude.

CONFORMS TO

       Python policy, version 0.4.1 (2006-06-20)

SEE ALSO

       debhelper(7)

       This program is a part of python-central but is made to work with debhelper.

AUTHORS

       Raphael Hertzog <hertzog@debian.org>

       Also includes bits of the old dh_python written by Josselin Mouette <joss@debian.org> who used many ideas from Brendan O'Dea
       <bod@debian.org>.

								    2011-04-14							   DH_PYCENTRAL(1)