Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Bash script with python slicing on multiple data files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash script with python slicing on multiple data files
# 1  
Old 05-15-2013
Bash script with python slicing on multiple data files
I have 2 files generated in linux that has common output and were produced across multiple hosts with the same setup/configs. These files do some simple reporting on resource allocation and user sessions. So, essentially, say, 10 hosts, with the same (2) system reporting in the files, so a combination of 20 files total.

Neither of the files contain any reference to the host name, but do contain resource info and correlative user info per machine.

I need to meet some basic reporting objectives that include:

- taking cumulative reporting data on the resource and ranking the machines by availability

- taking cumulative reporting data on the users and ranking on usage

I created a simple python script that runs through all the files and produces output that summarizes per machine, but I've not got the parsed data converted into a real data value, so its not sortable

I also think that the following needs to happen:

- merge data files in linux w/ piping

- append the machine name/# to the 1st column

- read that w/ python slicing (since the machine name will always be the first "X" # of characters in the line

- use a bash script, wrapped, then execute the python script after bash has merged all the data files

so something like

/bin/bash
merge
push to python
I might need to change my python script so that the function input isnt the machine name, but more like "big_data_file" etc

that make sense?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

Python script to run multiple command and append data in output csv file

Experts, I am writing a script and able to write only small piece of code and not able to collect logic to complete this task. In input file have to look for name like like this (BGL_HSR_901_1AG_A_CR9KTR10) before sh iss neors. Record this (BGL_HSR_901_1AG_A_CR9KTR10) in csv file Now have to... (0 Replies)
Discussion started by: as7951
0 Replies

2. UNIX for Beginners Questions & Answers

Using bash script : How to Import data from a dsv file into multiple tables in mysql

HI I have a dsv file that looks like: <<BOF>> record_number|id_number|first name|last name|msisdn|network|points|card number|gender 312|9101011234011|Test Junior|Smith|071 123 4321|MTN|73|1241551413214444|M 313|9012023213011|Bob|Smith|27743334321|Vodacom|3|1231233232323244|M... (4 Replies)
Discussion started by: tera
4 Replies

3. Shell Programming and Scripting

In PErl script: need to read the data one file and generate multiple files based on the data

We have the data looks like below in a log file. I want to generat files based on the string between two hash(#) symbol like below Source: #ext1#test1.tale2 drop #ext1#test11.tale21 drop #ext1#test123.tale21 drop #ext2#test1.tale21 drop #ext2#test12.tale21 drop #ext3#test11.tale21 drop... (5 Replies)
Discussion started by: Sanjeev G
5 Replies

4. Shell Programming and Scripting

Loop through multiple files in bash script

Hi Everybody, I'm a newbie to shell scripting, and I'd appreciate some help. I have a bunch of .txt files that have some unwanted content. I want to remove lines 1-3 and 1028-1098. #!/bin/bash for '*.txt' in <path to folder> do sed '1,3 d' "$f"; sed '1028,1098 d' "$f"; done I... (2 Replies)
Discussion started by: BabyNuke
2 Replies

5. Programming

Python script for extracting data using two files

Hello, I have two files. File 1 is a list of interested IDs Ex1 Ex2 Ex3File 2 is the original file with over 8000 columns and 20 millions rows and is a compressed file .gz Ex1 xx xx xx xx .... Ex2 xx xx xx xx .... Ex2 xx xx xx xx ....Now I need to extract the information for all the IDs of... (4 Replies)
Discussion started by: nans
4 Replies

6. Shell Programming and Scripting

Bash script to copy timestamps of multiple files

Hi, I have a bunch of media files in a directory that have been converted (from MTS to MOV format), so my directory contains something like this: clip1.mts clip1.mov clip2.mts clip2.mov The problem is that the .mov files that have been created have the timestamps of the conversion task,... (2 Replies)
Discussion started by: Krakus
2 Replies

7. Shell Programming and Scripting

bash script to compile multiple .c files with some options

I'm trying to write a bash script and call it "compile" such that running it allows me to compile multiple files with the options "-help," "-backup," and "-clean". I've got the code for the options written, i just can't figure out how to read the input string and then translate that into option... (5 Replies)
Discussion started by: travis.batzer
5 Replies

8. Shell Programming and Scripting

How to write bash script to explode multiple zip files

I have a directory full of zip files. How would I write a bash script to enumerate all the zip files, remove the ".zip" from the file name, create a directory by that name and unzip each zip file into its corresponding directory? Thanks! Siegfried (3 Replies)
Discussion started by: siegfried
3 Replies

9. Shell Programming and Scripting

moving multiple folders/files in subversion using bash script

Hi, I'm new here an dlearning a lot from this forum. i didnt find any solution for this in the forum. I have already checked in folders in subversion named HTT01,... HTT21.. and have files in each folder like below: HTT01/HTT01_00000.hex HTT01/HTT01_00000_fb_result.hex... (2 Replies)
Discussion started by: ravishan21
2 Replies

10. Shell Programming and Scripting

Script to delete all data from multiple files

its urgent!!!!!!111 i need a script which can delete data from multiple files. plz if anybody knows the script plz write a mail to me : (Email addresses are not allowed) (5 Replies)
Discussion started by: uni_ajay_r
5 Replies
Login or Register to Ask a Question

LEARN ABOUT DEBIAN

update-python-modules

UPDATE-PYTHON-MODULES(8)				      System Manager's Manual					  UPDATE-PYTHON-MODULES(8)

NAME

       update-python-modules - byte-compile python modules

SYNOPSIS

       update-python-modules [options] [package.public [...]]

DESCRIPTION

       The update-python-modules command is part of the python-support bundle.

       update-python-modules  is  responsible for the byte-compilation of python modules. When used in byte-compilation mode, it will byte-compile
       modules references in the files given on the command line. When used in	installation  mode,  it  will  byte-compile  them  once  for  each
       installed python version, and will make them available for all of them.

       At  each  invocation,  with  or without arguments, update-python-modules will check for installed python versions, and will build or remove
       modules according to new or removed versions since the last time it was run.

ARGUMENTS

       Arguments must be files or directories located in the /usr/share/python-support directory. Depending on their extension, they  are  treated
       differently.

       package.public
	      A file listing public modules to install for each version. These modules must lie in /usr/share/pyshared or /usr/lib/pyshared.

       package.private
	      A file listing private modules to byte-compile for one Python version.

       package/
	      A legacy directory hierarchy of public modules. The C extensions and portions that change with the Python version are to be found in
	      /usr/lib/python-support/pythonX.Y/package/.

       package.dirs
	      A legacy file listing directories where to find private modules.

OPTIONS

       -h, --help
	      Show summary of options.

       -v, --verbose
	      Detail all actions while they are performed.

       -b, --bytecompile
	      Only accept to work on private modules. When given this option, update-python-modules will fail if passed references to public  mod-
	      ules.

       -i, --install
	      Only  accept  to work on public modules. When given this option, update-python-modules will fail if passed references to public mod-
	      ules.

       -c, --clean
	      Clean modules instead of installing them.

       -a, --rebuild-all
	      Rebuild all private modules. This is necessary when the default python version was changed, for example.

       -f, --force-rebuild-all
	      Clean all modules (public and private), and rebuild them.

FILES

       /usr/lib/pymodules/pythonX.Y/
	      These directories contain the byte-compiled modules for version pythonX.Y.

SEE ALSO

       /usr/share/doc/python-support/README.gz

AUTHOR

       Josselin Mouette <joss@debian.org>.

								    18 Feb 2009 					  UPDATE-PYTHON-MODULES(8)