Login or Register to Ask a Question and Join Our Community


Programming

Program or bash script to see total progress of copy

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Programming Program or bash script to see total progress of copy
# 8  
Old 09-16-2017
Hi,

Another method is to compare the static size of the directory source (in whatever terms you desire) with the changing size of the destination.

See for example post 2 in thread https://www.unix.com/unix-advanced-ex...gress-bar.html

You'd probably need to modify the cp command to indicate you wish to copy a directory, probably like adding -r

The stat command would produce similar results to du, but in blocks (%b), and you may want to substitute du.

I don't see an ETA being calculated, but, as with the suggestion from Corona688 for rsync, it would avoid touching the data many times.

The utility tree v1.7.0 produces measurements quite quickly, so you also could use it. Here's a sample of timing for a largish directory with many items, disk being an SSD in this case (and producing 25K lines of output, to be discarded, keeping stderr):
Code:
1892 directories, 23706 files

real    0m0.812s
user    0m0.132s
sys     0m0.336s

from:
Code:
time tree src/

Best wishes ... cheers, drl

---------- Post updated at 21:36 ---------- Previous update was at 06:34 ----------

Hi.

See also many comments on rsync and progress at linux - Showing total progress in rsync: is it possible? - Server Fault

Best wishes ... cheers, drl
This User Gave Thanks to drl For This Post:
rbatte1
# 9  
Old 09-20-2017
just thought of another idea -

il get the size of the source path -

Code:
du -s /source_path/

then i will start the copy -

Code:
cp -r /source_path/ /destination_path/

while im copying i will monitor the progress -

Code:
watch -n 0.5 du -s /destination_path/

but i want to do this all in a bash script but my issue is it wont watch the destination path while the copy is going on, how do i do both at the same time

rob
# 10  
Old 09-20-2017
Put the cp into background. But, don't expect that method to be too exact. You'll need to know every single file to be included or excluded, and other disk activity may interfere.
# 11  
Old 09-20-2017
You seem to want it sizewise, not timewise. Try this, although due to I/O buffering the "progress bar" may not be in correct sync:
Code:
awk '
BEGIN           {printf "\r%101s", "|"
                }

NR == FNR       {C[$2] = $1
                 SUM  += $1
                 next
                }

                {gsub (/\047/, _, $1)
                 TOTSIZ += C[$1]
                 PCT     = TOTSIZ/SUM*100
                 TMP     = sprintf ("%*s", PCT, " ")
                 gsub (/ /, "-", TMP)
                 printf "\r%s", TMP
                }

END             {printf RS
                }
' <(du -b /source_path/*) <(cp -v /source_path/* /destination_path/ 2>&1)

# 12  
Old 09-20-2017
Hi.

There is a script at linux - Showing total progress in rsync: is it possible? - Server Fault by author nito near the end of the thread.

I copied it into a file I call watch-running-process

Here is a driver script:
Code:
#!/usr/bin/env bash

# @(#) s1       Demonstrate rate, ETA of running process, assumes /proc IO, like cp, rsync, etc.

LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C

SOURCE=/not-backed-up
DESTIN=/tmp
volume=$( du -s -BM $SOURCE | sed 's/M.*$//' )

cp -r $SOURCE $DESTIN &
my_pid=$!

./watch-running-process "$my_pid"  "$volume"

exit 0

The parameters are the PID you are watching, the volume of the source in MB, and an optional delay for the loop (default 5 seconds). Because you supply the PID, you can run anything you want, just as long as it does io that is captured in /proc (so not Solaris, macOS, BSD, etc.). I have not looked in detail at the numbers in /proc, but the script seems to work. If you are interested / curious, see man proc, look at the entry for io, read_bytes, etc.

Here are display snapshots near the beginning, middle, and end of the process for cp as done prior to the call to the monitoring script:
Code:
Monitoring PID: 29096

Read :      539.04 MiB in 5.03 s
Write:      651.68 MiB in 5.03 s

Read Rate : 107.16 MiB/s ( Avg: 58.05, Max: 126.64 )
Write Rate: 129.55 MiB/s ( Avg: 193.95, Max: 544.66 )

Done      : 7.63 GiB / 14.11 GiB (54.07 %)
ETA       : 00:00:34.21 (34.21s)
Elapsed   :       00:41

-----

Monitoring PID: 29096

Read :      336.91 MiB in 5.04 s
Write:      337.91 MiB in 5.04 s

Read Rate : 66.84 MiB/s ( Avg: 50.63, Max: 126.64 )
Write Rate: 67.04 MiB/s ( Avg: 107.78, Max: 544.66 )

Done      : 13.24 GiB / 14.11 GiB (93.83 %)
ETA       : 00:00:08.26 (8.26s)
Elapsed   :       02:08

-----

Monitoring PID: 29096

Read :      105.94 MiB in 5.03 s
Write:      105.86 MiB in 5.03 s

Read Rate : 21.06 MiB/s ( Avg: 50.43, Max: 126.64 )
Write Rate: 21.04 MiB/s ( Avg: 101.45, Max: 544.66 )

Done      : 13.96 GiB / 14.11 GiB (98.93 %)
ETA       : 00:00:01.51 (1.51s)
Elapsed   :       02:23
----- Finished -----

There is not a real progress bar, but it includes an ETA, along with data rates. Given that the numbers are all available in the script, a scaled progress bar probably could be done.

Best wishes ... cheers, drl
# 13  
Old 09-21-2017
i have seen this -

bash - How to add a progress bar to a shell script? - Stack Overflow

if you read the answer by mitch he has made a script that does this with echo commands

but when i try it and run it on my linux box i just get 33, 66, 100% with the hashes, how can i get this to count up from 1-100

also how can i implement this with my copy command "cp -r /source /dest"

many thanks,

rob
# 14  
Old 09-21-2017
cp does not have this feature. Full stop. Have you considered trying a program which does have this feature?

As per my post a page ago:

Quote:
Originally Posted by Corona688
How about rsync? It supports progress and same-system copy:

Code:
rsync -aI --progress source/ destination

-a for recursive, and -I to ignore timestamps and copy everything it finds.

Note the trailing / on the source is important! Otherwise you'll end up with destination/source/filename instead of destination/filename
I should mention, rsync is also extremely common and probably installed on your system already.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Bash script to get total size off of remainder calculated

I am working on a script to get the final total size and so far have the following and wondering if this can be improved. # Compare the desired size of each lvm to the standard size. If it is desired is larger than calculate the difference and keep that value as the amount to add to that LVM. ... (5 Replies)
Discussion started by: user3528
5 Replies

2. UNIX for Dummies Questions & Answers

Running a C/C++ program and/or bash script from a server

I wish to be able to give to a client the opportunity to : 0) Turn one of my ubuntu computers into a webserver 1) See a webpage after visiting a url where an external user/client can set a couple of variables (e.g. Number1= ?, Number2=?) 2) By pressing "run" the program runs on my machine 3)... (1 Reply)
Discussion started by: frad
1 Replies

3. Programming

Putting bash script in C program

suppose i have a bash script: #!/bin/bash echo "hello" echo "how are you" echo "today" how can i put the entire script above into a basic c program? i do not want to translate the bash code to a c code. i want C to run the bash code. is this possible? i found this on the... (15 Replies)
Discussion started by: SkySmart
15 Replies

4. Shell Programming and Scripting

Bash script to start program and answer prompts?

I'm trying to write a script the simplifies the execution of a program: After starting the program (sh ~/.mfix/model/make_mfix) I am prompted four times for options: Do you need SMP version? (y/n) Do you need DMP version? (y/n) Do you need debug version? (y/n) Force re-compilation of... (2 Replies)
Discussion started by: lanew
2 Replies

5. UNIX for Dummies Questions & Answers

Bash script to execute a program to rename files

I just can't figure it out , so please just give me a pice of advise how to: The existing Linux program foo2bar takes as its only argument the name of a single foo file and converts it to an appropriately-named bar file. Provide a script that when executed will run foo2bar against all foo... (4 Replies)
Discussion started by: raymen
4 Replies

6. Shell Programming and Scripting

bash script to check if a program is running

I'm a bit new to bash programming and I was assigned the job of writing a script that will check to see if a program server is running and to restart the program if it is not up. The script is supposed to check the program every hour (which I have looked up and I believe I know how to do) and send... (3 Replies)
Discussion started by: mcknz
3 Replies

7. Shell Programming and Scripting

Problem with bash shell script program

Hi, This is my program. #!/bin/bash today=`date +"%b-%d-%Y"` SERVICE="pbxconnect.php" if ; then echo "pbx program is running" else nohup php pbxconnect.php > logpbx-$today.txt & fi On executing using "sh myprogram.sh" , i get the following error. myprogram.sh: line 4: ' My... (7 Replies)
Discussion started by: gskumar1234
7 Replies

8. Shell Programming and Scripting

bash script to copy files

hey everyone, new here i have arch setup and i am using smbnetfs to mount some windows shares in /mnt/smbnet what i want to do is copy files from my home dir to a dir in /mnt/smbnet but i also need it to remove files if i have deleted them from my home dir seems that cp would be the... (8 Replies)
Discussion started by: dodgefan67
8 Replies

9. Homework & Coursework Questions

Run Program from Bash CGI-Script

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: This is a problem I am having with my 2 semester senior project. I have a LAMP server running Ubuntu 9.10 with... (8 Replies)
Discussion started by: JMooney5115
8 Replies

10. Shell Programming and Scripting

Making a progress gauge in a bash script

Hello once again: One thing that seems to be a nice feature is a progress gauge... so I can see how long an operation will take for a task to complete if it is requiring a lot of processing or the file is enormous. I have seen references to gauge operations, but I don't know anything about it or... (1 Reply)
Discussion started by: ccox85
1 Replies
Login or Register to Ask a Question