Multi thread shell programming

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Multi thread shell programming
# 1  
Old 08-30-2011
Multi thread shell programming

I have a unix directory where a million of small text files getting accumulated every week.

As of now there is a shell batch program in place which merges all the files in this directory into a single file and ftp to other system.

Previously the volume of the files would be around 1 lakh and we used to manage with single program.

Now this batch program taking so much time and running for days to merge the files.

any better approach to merge the files faster?

I would think... split the files based on some criteria and multi thread the program.

My files all the times start with a number.

MergeFilesNameStartwith0 & MergeFilesNameStartwith1 & MergeFilesNameStartwith2 .... MergeFilesNameStartwith9

Is this good idea? any suggetions appreciated

since the directory is having a million of files it could not list even 20k at a time

so using below expression to get the each file to merge as of now.

for file in ls | head -1500 
cat $file >> $concatFile

yours suggetions are welcome

Moderator's Comments:
Mod Comment Please use code tags, thanks. See PM.

Last edited by zaxxon; 08-30-2011 at 05:12 AM.. Reason: code tags
# 2  
Old 08-30-2011
Look at GNU parallel. Their man page has good examples.
# 3  
Old 08-30-2011
Try cpio and check if it is faster

ls server.xml*  |head -500 |cpio -ocv >$concatFile

Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Programming

Multi process programming in C

So I am trying to learn C and am coding some scripts on my own. For a start I have decided to port the shell script developed by wisecracker into C.( Here is the link to that script A simple reminder script for beginners to shell scripting. | Unix Linux Forums | OS X (Apple) ) This is what I... (7 Replies)
Discussion started by: chacko193
7 Replies

2. UNIX for Dummies Questions & Answers

Shell script to read lines in a text file and filter user data Shell Programming and Scripting

sxsaaas (3 Replies)
Discussion started by: VikrantD
3 Replies

3. Shell Programming and Scripting

Multi thread awk command for faster performance

Hi, I have a script below for extracting xml from a file. for i in *.txt do echo $i awk '/<.*/ , /.*<\/.*>/' "$i" | tr -d '\n' echo -ne '\n' done . I read about using multi threading to speed up the script. I do not know much about it but read it on this forum. Is it a... (21 Replies)
Discussion started by: chetan.c
21 Replies

4. Programming

Multi thread data sharing problem in uclinux

hello, I have wrote a multi thread application to run under uclinux. the problem is that threads does not share data. using the ps command it shows a single process for each thread. I test the application under Ubuntu 8.04 and Open Suse 10.3 with 2.6 kernel and there were no problems and also... (8 Replies)
Discussion started by: mrhosseini
8 Replies

5. UNIX for Dummies Questions & Answers

error in thread programming

Can you please let me know how to resolve this error: cc pc1.c /tmp/cc2os7n1.o: In function `main':pc1.c:(.text+0x89): undefined reference to `pthread_create' collect2: ld returned 1 exit status (3 Replies)
Discussion started by: p_aishwarya
3 Replies

6. Programming

About thread programming !

pthread_create(&tid, &Attr, A, NULL); pthread_create(&tid, &Attr, B, NULL); pthread_create(&tid, &Attr, C, NULL); A,B,C are the name of the functiont that the thread excute Hi all ! I want to ask so with these commands ,how the program will excute . What the way it will do . Please... (0 Replies)
Discussion started by: iwbasts
0 Replies

7. Programming

Posix Thread Programming

Hello, i have 2 questions: 1. Can I get the current memory usage of a thread? 2. Can I use a member-function as (void*)(*)(void*) method to create a new thread with "pthread_create(...)"?? I would be happy about any suggestion. Regards, Rolf (2 Replies)
Discussion started by: rkasel
2 Replies

8. AIX

Thread programming

Are there any differences about pthread programming between AIX 5.2 and True 64 UNIX 5.1B ? Anyone can give some special tips on AIX? (0 Replies)
Discussion started by: Frank2004
0 Replies

9. Programming

Multi threading using posix thread library

hi all, can anyone tell me some good site for the mutithreading tutorials, its application, and some code examples. -sushil (2 Replies)
Discussion started by: shushilmore
2 Replies
Login or Register to Ask a Question