Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Ext3 to NTFS - transfering data Post 302469901 by DGPickett on Monday 8th of November 2010 03:07:39 PM
Old 11-08-2010
Quote:
Originally Posted by Corona688
That would take even longer. It'd still have to read and write all the data, plus, it would have to compress and extract it too!
Well, it is pipeline parallel. However, if the network is a bottleneck, then something like
Code:
find . . . | cpio -oaH crc | gzip -9 | rsh that_host 'gunzip|cpio -idmH crc'

can speed things up. More than one at once is good, too, to keep thoses queues full all down the line. A lot of UNIX is single-buffered. This uses my simple line demus to spread the load to as many as 8 streams (since it is a dumb rotation, a big file may block one stream for a while, so adding a few extras beyond net saturation is good):
Code:
find . . . | dumux 8 "cpio -oaH crc | gzip -9 | rsh that_host 'gunzip|cpio -idmH crc' "
 
$ cat mysrc/dumux.c
 
#include <stdio.h>
#include <stdlib.h>
#include <strings.h>
static void usage(){
        fputs(
"\n"
"Usage: dumux <ct> <cmd> [ -l <line_ct> ]\n"
"\n"
"Runs <ct> copies of <cmd> and sends <line_ct> (default 1) lines to each\n"
"in rotation.\n"
"\n",
                stderr );
        exit( 1 );
 }
int main( int argc, char ** argv ){
        FILE **fp = NULL ;
        int i, x, c, l, lct = 1 ;
        if ( argc < 3
          || 2 > ( x = atoi( argv[1] ))){
                usage();
         }
        if ( argc > 3
          && ( argc != 5
            || strcmp( argv[3], "-l" )
            || 1 > ( lct = atoi( argv[4] )))){
                usage();
         }
        if ( !( fp = (FILE **)calloc( x, sizeof (FILE *)))){
                perror( "malloc" );
                exit( 2 );
                }
        for ( i = 0 ; i < x ; i++ ){
                if ( !( fp[i] = popen( argv[2], "w" ))){
                        perror( "popen( $2 )" );
                        exit( 3 );
                        }
                }
        i = l = 0 ;
        while ( EOF != ( c = getchar())){
                if ( EOF == putc( c, fp[i] )){
                        perror( "putc( popen( $2 ))" );
                        exit( 4 );
                        }
                if ( c == '\n'
                  && ++l == lct ){
                        l = 0 ;
                        if ( ++i == x ){
                                i = 0 ;
                         }
                 }
         }
        if ( ferror( stdin )){
                perror( "stdin" );
                exit( 5 );
                }
        for ( i = 0 ; i < x ; i++ ){
                if ( 0 > fclose( fp[i] )){
                        perror( "fclose( popen( $2 ))" );
                        }
                }
        exit( 0 );
}

 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Transfering a window to another screen

Dear Experts I am using a workstatin with two screens and its OS is solaris 8. I wish to trnsfer some application windows from one screen to another one without need to closing the window and open it from other screen. Please advise Bests Reza (3 Replies)
Discussion started by: Reza Nazarian
3 Replies

2. UNIX for Dummies Questions & Answers

Transfering files

Hi all. 1. How can i copy files from one unix system to another. should i use ftp? so How? 2. How can i create an archive whose extention is tar.gz? and how can i decompress them later? 3. WHat is RPM ? what does it stands for? Thanks (5 Replies)
Discussion started by: vbs
5 Replies

3. UNIX for Dummies Questions & Answers

Transfering files from one server to another.

My oracle database is generating archive logs. I want to copy those archive logs over to backup server on a regular basis. I know how to create an ftp job and I can put it in my crontab. My problem is that I don't know how to send the files just once instead of sends all the files in the... (4 Replies)
Discussion started by: Alan Bird
4 Replies

4. Filesystems, Disks and Memory

Transfering size between partitions

Below is a list of the file system on my Sun system. How can I transfer more disk space from the "/space" partition to the "/" partition with out rebuilding? / /dev/dsk/c0t0d0 6191949 5736718 393312 94% /proc /proc 0 0 0 0% /dev/fd fd 0 0 0 0% /etc/mntta ... (10 Replies)
Discussion started by: meyersp
10 Replies

5. BSD

Mounting ext3 & NTFS on PC-BSD!!

I'm having problem mounting ext3 & ntfs partitions on my PC-BSD OS. Can anyone please help me out here. What are the changes required to be done in fstab?? Are there any patches to be installed?? (1 Reply)
Discussion started by: void_man()
1 Replies

6. UNIX for Dummies Questions & Answers

transfering a directory to the toolman

I have a question about transferring a directory to the toolman. I have a directory called assn3 that contains two txt files and one empty directory and I want to transfer the assn3 to my my tooman account. Every time I try to transfer the file it says that the assn3 is not a regular file! what... (3 Replies)
Discussion started by: aama100
3 Replies

7. Red Hat

CentOS and XP dualboot + ext3/ntfs mount

I have WinXP Pro SP2 on first disk /dev/hda, which my BIOS currently has as first boot device. I have CentOS-5.2-x86_64 on second disk /dev/hdb, which I can boot into if I set my BIOS to boot from this HDD. It is using LVM with vg00 and a single LV for root (/) filesystem (ext3). How do I get... (19 Replies)
Discussion started by: apra143
19 Replies

8. UNIX for Dummies Questions & Answers

Data files on AIX to NTFS Drive

How can I copy a data disk on an AIX system to a Windows readable format (0 Replies)
Discussion started by: Phill
0 Replies

9. UNIX for Advanced & Expert Users

Errors received while restoring my data from ext3 filesystem

Hi All, I have dual boot on my laptop - Win8 And linuxMint. From linuxmint i had copied all my data found on the NTFS partitions (on my laptop) to an external hard drive (formatted with ext3). i used rsync for this. Now after my hard disk crashed, am restoring the data back from ext3... (2 Replies)
Discussion started by: coolatt
2 Replies
POPEN(3)						   BSD Library Functions Manual 						  POPEN(3)

NAME
pclose, popen -- process I/O LIBRARY
Standard C Library (libc, -lc) SYNOPSIS
#include <stdio.h> FILE * popen(const char *command, const char *mode); int pclose(FILE *stream); DESCRIPTION
The popen() function ``opens'' a process by creating a bidirectional pipe, forking, and invoking the shell. Any streams opened by previous popen() calls in the parent process are closed in the new child process. Historically, popen() was implemented with a unidirectional pipe; hence, many implementations of popen() only allow the mode argument to specify reading or writing, not both. Because popen() is now imple- mented using a bidirectional pipe, the mode argument may request a bidirectional data flow. The mode argument is a pointer to a null-termi- nated string which must be 'r' for reading, 'w' for writing, or 'r+' for reading and writing. The command argument is a pointer to a null-terminated string containing a shell command line. This command is passed to /bin/sh, using the -c flag; interpretation, if any, is performed by the shell. The return value from popen() is a normal standard I/O stream in all respects, save that it must be closed with pclose() rather than fclose(). Writing to such a stream writes to the standard input of the command; the command's standard output is the same as that of the process that called popen(), unless this is altered by the command itself. Conversely, reading from a ``popened'' stream reads the command's standard output, and the command's standard input is the same as that of the process that called popen(). Note that output popen() streams are fully buffered, by default. The pclose() function waits for the associated process to terminate; it returns the exit status of the command, as returned by wait4(2). RETURN VALUES
The popen() function returns NULL if the fork(2) or pipe(2) calls fail, or if it cannot allocate memory. The pclose() function returns -1 if stream is not associated with a ``popened'' command, if stream already ``pclosed'', or if wait4(2) returns an error. ERRORS
The popen() function does not reliably set errno. SEE ALSO
sh(1), fork(2), pipe(2), wait4(2), fclose(3), fflush(3), fopen(3), stdio(3), system(3) BUGS
Since the standard input of a command opened for reading shares its seek offset with the process that called popen(), if the original process has done a buffered read, the command's input position may not be as expected. Similarly, the output from a command opened for writing may become intermingled with that of the original process. The latter can be avoided by calling fflush(3) before popen(). Failure to execute the shell is indistinguishable from the shell's failure to execute command, or an immediate exit of the command. The only hint is an exit status of 127. The popen() function always calls sh(1), never calls csh(1). HISTORY
A popen() and a pclose() function appeared in Version 7 AT&T UNIX. Bidirectional functionality was added in FreeBSD 2.2.6. BSD
May 3, 1995 BSD
All times are GMT -4. The time now is 07:24 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy