Storage usage over time


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Storage usage over time
# 1  
Old 09-12-2017
Storage usage over time

Hi every one, im kind new in scripting.
i need to write a script that will collect my storage usage (df -k) like every hour and save the max amout and min amout into csv file so over time lest say one month we have the min/max values. Is it possible with awk command ?
Any ideas are welcome.Smilie
# 2  
Old 09-12-2017
Welcome venus699,

I have a few to questions pose in response first:-
  • What have you tried so far?
  • What output/errors do you get?
  • What OS and version are you using?
  • What are your preferred tools? (C, shell, perl, awk, etc.)
  • What logical process have you considered? (to help steer us to follow what you are trying to achieve)
Most importantly, What have you tried so far?

There are probably many ways to achieve most tasks, so giving us an idea of your style and thoughts will help us guide you to an answer most suitable to you so you can adjust it to suit your needs in future.


We're all here to learn and getting the relevant information will help us all.



Kind regards,
Robin
# 3  
Old 09-13-2017
actualy just tryed some awk comands. and did some research around it, but canīt find anything that compare 2 files. lest say:
  • Record the first file df -kP > file1.csv
  • compare file1 to max.csv and save the respective values that are greater than the previous we had on max.csv

I'm open to ideas, I'm trying to use awk maybe python but I have low skills.
I'm using opensuse.

---------- Post updated 09-13-17 at 08:26 AM ---------- Previous update was 09-12-17 at 09:52 AM ----------

This is what I have archive atm, still not done yet having some troubles with duplicate fields for some reason. if any one up to test it would be cool.

Code:
fileactual="results/fileactual.csv"
fileMaxMin="results/maxminconsumption.csv"
filemax="results/filemax.csv"
filemin="results/filemin.csv"

filecomparemax="results/comparemax.csv"
filecomparemax2="results/compare2max.csv"

filecomparemin="results/comparemin.csv"
filecomparemin2="results/compar2min.csv"


#--------print all the df data
df -kP | awk '{print $0}' > $fileactual

if [ ! -f $filemax ]; then
  df -kP | awk '{print $0}' > $filemax
fi

if [ ! -f $filemin ]; then
  df -kP | awk '{print $0}' > $filemin
fi



#--------past the 2 files so later we can compare
paste -d" " $fileactual $filemax > $filecomparemax
paste -d" " $fileactual $filemin > $filecomparemin

#--------remove first line from DF command.
awk '{if (NR!=1) {print}}' $filecomparemax > $filecomparemax2

awk '{if (NR!=1) {print}}' $filecomparemin > $filecomparemin2
#-------selecing the biger value on column 3 = used space and print the row
awk '$3 <= $9 {print $1,$2,$3,$4,$5,$6}; $3 >= $9 {print $7,$8,$9,$10,$11,$12}' $filecomparemax2 > $filemax
#-------selecing the lower value on column 3 = used space and print the row
awk '$3 >= $9 {print $1,$2,$3,$4,$5,$6}; $3 <= $9 {print $7,$8,$9,$10,$11,$12}' $filecomparemin2 > $filemin

echo 'THIS IS THE MAX VALUES

------------------------------' > $fileMaxMin
awk 'NR==1' $fileactual >> $fileMaxMin

cat $filemax >> $fileMaxMin

echo '-----------------------------

THIS IS THE LOWER VALUES

------------------------------' >> $fileMaxMin
awk 'NR==1' $fileactual  >> $fileMaxMin

cat $filemin >> $fileMaxMin

rm $filecomparemax
rm $filecomparemax2
rm $filecomparemin
rm $filecomparemin2
rm $fileactual


probably not the best approach looking for new ideas.

Last edited by rbatte1; 09-15-2017 at 12:35 PM.. Reason: Added ICODE tags and corrected spelling
# 4  
Old 09-15-2017
i made it work, does the job i needed.
fell free to use it, dont forget when opening the file maxminconsumption.csv to import it into excel,i use -> , to discting each columm.

Cheers Smilie


Code:
#|/bin/bash
# ----------------------------------------------- #
#          storageconsumption.sh                  #
#     Storage max / min values                  #
#                   Version 1.0                        #
#                   by:                                    #
# ----------------------------------------------- #

fileactual="fileactual.csv"
fileMaxMin="maxminconsumption.csv"
filemax="filemax.csv"
filemin="filemin.csv"
filecomparemax="comparemax.csv"
filecomparemin="comparemin.csv"



#--------print all the df data to the actualfile
df -hP | awk '{print $0}' > $fileactual


#-------Check if file exists, if it donst will save fresh data from DF command
if [ ! -f $filemax ]; then
  df -hP | awk '{print $0}' > $filemax
fi

if [ ! -f $filemin ]; then
  df -hP | awk '{print $0}' > $filemin
fi


#--------past the 2 files in a new one so in the next command "awk" we can compare the columms
paste -d" " $fileactual $filemax > $filecomparemax
paste -d" " $fileactual $filemin > $filecomparemin


#-------selecing the biger value on column 3 = used space and print the total row. colummns 1 to 6 is from the actual file, 7 to 12 is from the filemax / filemin
awk '$3 > $9 {print $1","$2","$3","$4","$5","$6}; $3 < $9 {print $7","$8","$9","$10","$11","$12} ; $3==$9 {print $7","$8","$9","$10","$11","$12}' $filecomparemax > $filemax
#-------selecing the lower value on column 3 = used space and print the row
awk '$3 < $9 {print $1","$2","$3","$4","$5","$6}; $3 > $9 {print $7","$8","$9","$10","$11","$12} ; $3==$9 {print $7","$8","$9","$10","$11","$12}' $filecomparemin > $filemin

#-------print the header of our final file.
echo 'THIS IS THE MAX VALUES

-----------------,-----------------,-----------------,-----------------,-----------------,----------------- ' > $fileMaxMin

#-------print our MAX values
cat $filemax >> $fileMaxMin

echo '-----------------,-----------------,-----------------,-----------------,-----------------,----------------- 

THIS IS THE LOWER VALUES

-----------------,-----------------,-----------------,-----------------,-----------------,----------------- ' >> $fileMaxMin

#-------print our MIN values
cat $filemin >> $fileMaxMin

#------- delete desnecessary files.
rm $filecomparemax
rm $filecomparemin
rm $fileactual

This User Gave Thanks to venus699 For This Post:
# 5  
Old 09-20-2017
i did another one, to collect data over time, the space of folder in filepath.
fell free to use it Smilie

also if any one know how to improve the code to be more efficient just share, im here to learn Smilie
Code:
filepath="/home/"
foldername="*"
data="data1.csv"
actualdata2="data2.csv"
actualdata3="data3.csv"
actualheader="data4.csv"
#---------------new file if $data dont exist yet
if [ ! -e $data ]; then
	echo " " >> $data
	for file in $filepath$foldername
		do
		du -sh $file | awk  '{print $2}' >>  $data
	done

fi
#---------------input from wc -l so later we can check if we need to update our file
countfilepath=`ls $filepath | wc -l`
countfileline=`awk '{if (NR!=1) {print}}' data1.csv | wc -l`

#------------- file existe do this.
if [ -e $data ]; then
#--------------- if number of files is not equal to number of lines in file
    if [ $countfilepath -eq $countfileline ]; then
		echo $(date +%m'-'%d) >> $actualdata2
		for file in $filepath$foldername
       		do
	        du -sh $file | awk  '{print $1}' >>  $actualdata2
		done
	    paste -d" " $data $actualdata2 > $actualdata3
	    mv $actualdata3 $data
            rm $actualdata2 

    else
	    mv $data $(date +%Y%m%d)'_'$(date +%H%M%S)'_'$data	
	    echo " " >> $data
	    	
		for file in $filepath$foldername
                    do
                    du -sh $file | awk  '{print $2}' >>  $data
	        done
	    echo $(date +%m'-'%d) >> $actualdata2	
		for file in $filepath$foldername
                    do 
                    du -sh $file | awk  '{print $1}' >>  $actualdata2
                done
            paste -d" " $data $actualdata2 > $actualdata3
            mv $actualdata3 $data
	    rm $actualdata2  
    fi
fi

Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Linux

Ps command on cpu usage and time

Hi All, Am very new to Linux and unix ...need below help . need to list of process consuming more than 40% cpu and which are older than 10 days of a particular user .... Thanks V (4 Replies)
Discussion started by: venky456
4 Replies

2. UNIX for Dummies Questions & Answers

Memory usage per user,percent usage,sytem time in ksh

Let's say i have 20 users logged on Server. How can I know how much memory percent used each of them is using with system time in each user? (2 Replies)
Discussion started by: roy1912
2 Replies

3. UNIX for Advanced & Expert Users

Find memory usage along with time

Hi Guys, I have a script. It calls an executable inside (programmed in C). I will have to find the execution time of that script and amount of memory consumed by that process as well. #!/bin/sh echo "Script starting" echo "executable staring" executable parm1 parm2 parm3 echo... (4 Replies)
Discussion started by: PikK45
4 Replies

4. Shell Programming and Scripting

Users and their storage usage

Hey, I'm trying to make a script that can list users and their storage usage in a file. Im trying to get it to where the format is: User xxkbs User2 xkb and so on. So far I figured out how to get the users, but I cannot figure out how to get the storage usage per user. This is what... (2 Replies)
Discussion started by: HakerDemon
2 Replies

5. Programming

usb real time video storage

Hi, Am working on an embedded device with a camera and I would want to send the video data it streams to a usb device connected to it. Am using the c language and of course Linux (Fedora). Any help and suggestions are most welcomed. Thanks in advance.:) (1 Reply)
Discussion started by: THSstd
1 Replies

6. HP-UX

how can I find cpu usage memory usage swap usage and logical volume usage

how can I find cpu usage memory usage swap usage and I want to know CPU usage above X% and contiue Y times and memory usage above X % and contiue Y times my final destination is monitor process logical volume usage above X % and number of Logical voluage above can I not to... (3 Replies)
Discussion started by: alert0919
3 Replies

7. AIX

AIX 4.3 storage usage report

hi, i'm new to this company and i'm using old AIX 4.3 with DAS and SSA storage. What is the best way to query a report for disk capacity, usage, free, etc. I know in AIX 5 there's a command for "du -g /" or "du -m /" to report the total usage but I'm not using AIX5. In AIX 4.3 I can use "df... (5 Replies)
Discussion started by: venerayan
5 Replies

8. Shell Programming and Scripting

time command usage in sh script

how do i get the following type of information when we have a time command in sh script. 0.01user 0.00system 1:32.98elapsed 0%CPU (0avgtext+0avgdata 5360maxresident)k 0inputs+0outputs (342major+0minor)pagefaults 0swaps I always see the following for the time command real 0m0.000s user ... (7 Replies)
Discussion started by: kris_search
7 Replies
Login or Register to Ask a Question