Multi thread awk command for faster performance


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Multi thread awk command for faster performance
# 1  
Old 04-28-2012
Multi thread awk command for faster performance

Hi,

I have a script below for extracting xml from a file.
Code:
for i in *.txt
do
echo $i
awk '/<.*/ , /.*<\/.*>/' "$i" | tr -d '\n'
echo -ne '\n' 
done

.

I read about using multi threading to speed up the script.
I do not know much about it but read it on this forum.

Is it a possible option here? Otherwise please guide me to make this script perform faster as i have XML data of aorund 100mb to extract from files.

Thanks,
Chetan.C
# 2  
Old 04-28-2012
You can split the files that you need to process into a number of batches and run your code concurrently on each of them.
# 3  
Old 04-29-2012
Hi Bartus,

Is it like if there are 100 files then get 50 files and call function concurrently.

Like this?
Code:
 
function first {
#to_process=$1
cd $1
for i in *.txt
do
echo $i
awk '/<.*/ , /.*<\/.*>/' "$i" | tr -d '\n'
echo -ne '\n' 
done
}
first /home/Folder1 &
first /home/Folder2 &
wait

Please let me know if im wrong.Also is multithreading an option here?

Thanks,
Chetan
# 4  
Old 04-29-2012
Yes, this is what I meant.
# 5  
Old 04-29-2012
Thanks Bartus.

Can you tell me how I can loop this for dynamic number of folders?

Like the number of folders may chnage and the function has to be called for the folders present only.

Thanks,
Chetan.C
# 6  
Old 04-29-2012
Try:
Code:
for dir in `find /home -type d -name "Folder*"`; do
  first $dir &
done

This User Gave Thanks to bartus11 For This Post:
# 7  
Old 04-29-2012
Hi.
Quote:
A thread is a lightweight process. The implementation of threads and processes differs from one operating system to another, but in most cases, a thread is contained inside a process. Multiple threads can exist within the same process and share resources such as memory, while different processes do not share these resources. -- excerpt from http://en.wikipedia.org/wiki/Thread_(computing)
Here is a sample use of GNU parallel that counts file contents with wc:
Code:
#!/usr/bin/env bash

# @(#) s1	Demonstrate multiple processes simultaneously, with "GNU parallel".

pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C parallel

pl " Structure of directories:"
tree d1 d2

pl " Results of parallel processes:"
ls d1/* d2/* |
grep txt |
parallel --ungroup 'echo -n job {#}, process $$", wc = "; wc {}' |
align

exit 0

producing:
Code:
% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0.8 (lenny) 
bash GNU bash 3.2.39
parallel GNU parallel 20111122

-----
 Structure of directories:
d1
|-- a.txt
|-- b.txt
|-- binary-1.exe
`-- c.txt
d2
|-- frog-town.jpg
|-- x.txt
`-- y.txt

0 directories, 7 files

-----
 Results of parallel processes:
job 1, process 27495, wc =  4  16   70 d1/a.txt
job 2, process 27515, wc = 16  16  123 d1/b.txt
job 3, process 27535, wc = 26 265 1464 d1/c.txt
job 4, process 27555, wc =  4  16   70 d2/x.txt
job 5, process 27575, wc = 16  16  123 d2/y.txt

Each one of the tasks was run as a separate process. The calling sequence for parallel is complex, so some experimentation might be useful. I have not tried it, but I think parallel claims to be able to utilize different computers for tasks.

The code for the (perl) parallel script is at GNU Parallel - GNU Project - Free Software Foundation

Best wishes ... cheers, drl
This User Gave Thanks to drl For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: 127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies

2. Shell Programming and Scripting

How to make awk command faster?

I have the below command which is referring a large file and it is taking 3 hours to run. Can something be done to make this command faster. awk -F ',' '{OFS=","}{ if ($13 == "9999") print $1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12 }' ${NLAP_TEMP}/hist1.out|sort -T ${NLAP_TEMP} |uniq>... (13 Replies)
Discussion started by: Peu Mukherjee
13 Replies

3. Shell Programming and Scripting

How to substract selective values in multi row, multi column file (using awk or sed?)

Hi, I have a problem where I need to make this input: nameRow1a,text1a,text2a,floatValue1a,FloatValue2a,...,floatValue140a nameRow1b,text1b,text2b,floatValue1b,FloatValue2b,...,floatValue140b look like this output: nameRow1a,text1b,text2a,(floatValue1a - floatValue1b),(floatValue2a -... (4 Replies)
Discussion started by: nricardo
4 Replies

4. Shell Programming and Scripting

Making a faster alternative to a slow awk command

Hi, I have a large number of input files with two columns of numbers. For example: 83 1453 99 3255 99 8482 99 7372 83 175 I only wish to retain lines where the numbers fullfil two requirements. E.g: =83 1000<=<=2000 To do this I use the following... (10 Replies)
Discussion started by: s052866
10 Replies

5. Shell Programming and Scripting

Faster way to use this awk command

awk "/May 23, 2012 /,0" /var/tmp/datafile the above command pulls out information in the datafile. the information it pulls is from the date specified to the end of the file. now, how can i make this faster if the datafile is huge? even if it wasn't huge, i feel there's a better/faster way to... (8 Replies)
Discussion started by: SkySmart
8 Replies

6. Shell Programming and Scripting

Multi thread shell programming

I have a unix directory where a million of small text files getting accumulated every week. As of now there is a shell batch program in place which merges all the files in this directory into a single file and ftp to other system. Previously the volume of the files would be around 1 lakh... (2 Replies)
Discussion started by: vk39221
2 Replies

7. Programming

Multi thread data sharing problem in uclinux

hello, I have wrote a multi thread application to run under uclinux. the problem is that threads does not share data. using the ps command it shows a single process for each thread. I test the application under Ubuntu 8.04 and Open Suse 10.3 with 2.6 kernel and there were no problems and also... (8 Replies)
Discussion started by: mrhosseini
8 Replies

8. UNIX for Dummies Questions & Answers

Which command will be faster? y?

i)wc -c/etc/passwd|awk'{print $1}' ii)ls -al/etc/passwd|awk'{print $5}' (4 Replies)
Discussion started by: karthi_g
4 Replies

9. Programming

Multi threading using posix thread library

hi all, can anyone tell me some good site for the mutithreading tutorials, its application, and some code examples. -sushil (2 Replies)
Discussion started by: shushilmore
2 Replies
Login or Register to Ask a Question