05-15-2013
Bash script with python slicing on multiple data files
I have 2 files generated in linux that has common output and were produced across multiple hosts with the same setup/configs. These files do some simple reporting on resource allocation and user sessions. So, essentially, say, 10 hosts, with the same (2) system reporting in the files, so a combination of 20 files total.
Neither of the files contain any reference to the host name, but do contain resource info and correlative user info per machine.
I need to meet some basic reporting objectives that include:
- taking cumulative reporting data on the resource and ranking the machines by availability
- taking cumulative reporting data on the users and ranking on usage
I created a simple python script that runs through all the files and produces output that summarizes per machine, but I've not got the parsed data converted into a real data value, so its not sortable
I also think that the following needs to happen:
- merge data files in linux w/ piping
- append the machine name/# to the 1st column
- read that w/ python slicing (since the machine name will always be the first "X" # of characters in the line
- use a bash script, wrapped, then execute the python script after bash has merged all the data files
so something like
/bin/bash
merge
push to python
I might need to change my python script so that the function input isnt the machine name, but more like "big_data_file" etc
that make sense?
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
its urgent!!!!!!111
i need a script which can delete data from multiple files.
plz if anybody knows the script plz write a mail to me :
(Email addresses are not allowed) (5 Replies)
Discussion started by: uni_ajay_r
5 Replies
2. Shell Programming and Scripting
Hi, I'm new here an dlearning a lot from this forum. i didnt find any solution for this in the forum.
I have already checked in folders in subversion named
HTT01,... HTT21.. and have files in each folder like below:
HTT01/HTT01_00000.hex
HTT01/HTT01_00000_fb_result.hex... (2 Replies)
Discussion started by: ravishan21
2 Replies
3. Shell Programming and Scripting
I have a directory full of zip files.
How would I write a bash script to enumerate all the zip files, remove the ".zip" from the file name, create a directory by that name and unzip each zip file into its corresponding directory?
Thanks!
Siegfried (3 Replies)
Discussion started by: siegfried
3 Replies
4. Shell Programming and Scripting
I'm trying to write a bash script and call it "compile" such that running it allows me to compile multiple files with the options "-help," "-backup," and "-clean". I've got the code for the options written, i just can't figure out how to read the input string and then translate that into option... (5 Replies)
Discussion started by: travis.batzer
5 Replies
5. Shell Programming and Scripting
Hi, I have a bunch of media files in a directory that have been converted (from MTS to MOV format), so my directory contains something like this:
clip1.mts
clip1.mov
clip2.mts
clip2.mov
The problem is that the .mov files that have been created have the timestamps of the conversion task,... (2 Replies)
Discussion started by: Krakus
2 Replies
6. Programming
Hello,
I have two files.
File 1 is a list of interested IDs
Ex1
Ex2
Ex3File 2 is the original file with over 8000 columns and 20 millions rows and is a compressed file .gz
Ex1 xx xx xx xx ....
Ex2 xx xx xx xx ....
Ex2 xx xx xx xx ....Now I need to extract the information for all the IDs of... (4 Replies)
Discussion started by: nans
4 Replies
7. Shell Programming and Scripting
Hi Everybody,
I'm a newbie to shell scripting, and I'd appreciate some help. I have a bunch of .txt files that have some unwanted content. I want to remove lines 1-3 and 1028-1098.
#!/bin/bash
for '*.txt' in <path to folder>
do
sed '1,3 d' "$f";
sed '1028,1098 d' "$f";
done
I... (2 Replies)
Discussion started by: BabyNuke
2 Replies
8. Shell Programming and Scripting
We have the data looks like below in a log file.
I want to generat files based on the string between two hash(#) symbol like below
Source:
#ext1#test1.tale2 drop
#ext1#test11.tale21 drop
#ext1#test123.tale21 drop
#ext2#test1.tale21 drop
#ext2#test12.tale21 drop
#ext3#test11.tale21 drop... (5 Replies)
Discussion started by: Sanjeev G
5 Replies
9. UNIX for Beginners Questions & Answers
HI
I have a dsv file that looks like:
<<BOF>>
record_number|id_number|first name|last name|msisdn|network|points|card number|gender
312|9101011234011|Test Junior|Smith|071 123 4321|MTN|73|1241551413214444|M
313|9012023213011|Bob|Smith|27743334321|Vodacom|3|1231233232323244|M... (4 Replies)
Discussion started by: tera
4 Replies
10. Programming
Experts,
I am writing a script and able to write only small piece of code and not able to collect logic to complete this task.
In input file have to look for name like like this (BGL_HSR_901_1AG_A_CR9KTR10) before sh iss neors. Record this (BGL_HSR_901_1AG_A_CR9KTR10) in csv file
Now have to... (0 Replies)
Discussion started by: as7951
0 Replies
LEARN ABOUT DEBIAN
update-python-modules
UPDATE-PYTHON-MODULES(8) System Manager's Manual UPDATE-PYTHON-MODULES(8)
NAME
update-python-modules - byte-compile python modules
SYNOPSIS
update-python-modules [options] [package.public [...]]
DESCRIPTION
The update-python-modules command is part of the python-support bundle.
update-python-modules is responsible for the byte-compilation of python modules. When used in byte-compilation mode, it will byte-compile
modules references in the files given on the command line. When used in installation mode, it will byte-compile them once for each
installed python version, and will make them available for all of them.
At each invocation, with or without arguments, update-python-modules will check for installed python versions, and will build or remove
modules according to new or removed versions since the last time it was run.
ARGUMENTS
Arguments must be files or directories located in the /usr/share/python-support directory. Depending on their extension, they are treated
differently.
package.public
A file listing public modules to install for each version. These modules must lie in /usr/share/pyshared or /usr/lib/pyshared.
package.private
A file listing private modules to byte-compile for one Python version.
package/
A legacy directory hierarchy of public modules. The C extensions and portions that change with the Python version are to be found in
/usr/lib/python-support/pythonX.Y/package/.
package.dirs
A legacy file listing directories where to find private modules.
OPTIONS
-h, --help
Show summary of options.
-v, --verbose
Detail all actions while they are performed.
-b, --bytecompile
Only accept to work on private modules. When given this option, update-python-modules will fail if passed references to public mod-
ules.
-i, --install
Only accept to work on public modules. When given this option, update-python-modules will fail if passed references to public mod-
ules.
-c, --clean
Clean modules instead of installing them.
-a, --rebuild-all
Rebuild all private modules. This is necessary when the default python version was changed, for example.
-f, --force-rebuild-all
Clean all modules (public and private), and rebuild them.
FILES
/usr/lib/pymodules/pythonX.Y/
These directories contain the byte-compiled modules for version pythonX.Y.
SEE ALSO
/usr/share/doc/python-support/README.gz
AUTHOR
Josselin Mouette <joss@debian.org>.
18 Feb 2009 UPDATE-PYTHON-MODULES(8)