Visit Our UNIX and Linux User Community


Shell Programming in unix


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell Programming in unix
# 1  
Old 02-14-2008
Shell Programming in unix

Hi,

i want to read a full file.

If i want to split the file and by reading parralel each, i can save the time.

Can any body give me the suggesion??

ia m using this function to read a file and using that i have to grep in another file. since the file 1 is huge it is taking lot of time.

iam planning to split that and run parraley.

suggesions pls



read_file1()
{
cat file1.txt | while read line
do
grep $line file2.txt >> outputfile.dat
done
}
nivas
# 2  
Old 02-14-2008
Quote:
Originally Posted by nivas
Hi,
i want to read a full file. If i want to split the file and by reading parralel each, i can save the time. Can any body give me the suggesion?? ia m using this function to read a file and using that i have to grep in another file. ince the file 1 is huge it is taking lot of time. iam planning to split that and run parraley. suggesions pls
Am I reading this correct as:

You have a very large file and it takes a lot of time to process it. So you are going to split up the file and have each piece processed separately.

I am not sure how exactly where parallel processing comes into play in your example or how this is faster that reading the entire file (unless your program loads the entire file into for some reason)

Have you looked at the 'split' command? It will split a file into separate files based on size as well as lines. So you could break the 'large' file into separate 1000 line files and then process them accordingly.
# 3  
Old 02-14-2008
iam using the split command and i want to know how many files it has splitted.

ex:

big file 1 lac lines.

split 5000 bigfilename.txt

then i want to read thhis splitted file one by one and run the grep command as abouve.
nivas
# 4  
Old 02-14-2008
non-parallel processing ...

Code:
split --lines 5000 bigfilename.txt tmp_
for PART in $( ls tmp_* ); do
    grep something $PART
done


Previous Thread | Next Thread
Test Your Knowledge in Computers #582
Difficulty: Medium
A buffered input or output stream is one in which data that is read or written is actually stored in memory in a buffer until such a time as the buffer is flushed and the accumulated data is passed to or from the actual file.
True or False?

10 More Discussions You Might Find Interesting

1. Homework & Coursework Questions

Help for programming a UNIX Shell in C++

1. The problem statement, all variables and given/known data: Hello! :) I currently got the task of programming a UNIX Shell for practice. The functionality is as follows: 1. Entering commands with the keyboard. Enter stops the input and creates a process which should start any program 2.... (0 Replies)
Discussion started by: DarkDan
0 Replies

2. Shell Programming and Scripting

Help for programming a UNIX Shell in C++

Hello! :) I currently got the task of programming a UNIX Shell for practice. The functionality is as follows: 1. Entering commands with the keyboard. Enter stops the input and creates a process which should start any program 2. the shell waits for termination of each command before... (1 Reply)
Discussion started by: DarkDan
1 Replies

3. Shell Programming and Scripting

Best way to learn UNIX and shell Programming

Guys, What do you think is the best way to learn UNIX and shell scripting? ** I keep on searching tutorials online, where I loose most of my time :( Let me know the way you learnt the UNIX concepts, your replies might help me learn more. Thanks a ton:b: (2 Replies)
Discussion started by: dnam9917
2 Replies

4. Shell Programming and Scripting

UNIX shell programming

Hi guys i have two different line input M5.7&a : M5 minimum density is 20%, maximum density is 80%, DENSITY PERM=M5.8&a(180) ......... 6 violations found. M6.7&a : M6 minimum density is 20%, maximum density is 80%, DENSITY A=M6.8&a(180) ......... 6 violations found. need... (2 Replies)
Discussion started by: ashokkrishna063
2 Replies

5. UNIX for Dummies Questions & Answers

How does unix system administration, unix programming, unix network programming differ?

How does unix system administration, unix programming, unix network programming differ? Please help. (0 Replies)
Discussion started by: thulasidharan2k
0 Replies

6. Shell Programming and Scripting

Matching strings in unix shell programming

#!bin/sh `clear` for i in $(seq -w 15 37); do #echo $i wget --dns-timeout=0.0005 http://napdweb${i}.eao.abn-iad.ea.com:8000/webcore/test/test.jsp -o 1 A=`cat 1` C=$(expr "$A" :... (7 Replies)
Discussion started by: veerumahanthi41
7 Replies

7. Shell Programming and Scripting

Very New To Unix Shell Programming:Plz Help

Hi Gurus I am very new to Unix Shell Prog. I have a file in format Q1 Dirname-FileName Score Remarks i.e. containing columns separated by space. I want to read Column 1 and 2 and then join them to make a string that would be a path to a file. I will use this string to fetch the files and... (4 Replies)
Discussion started by: kimskams80
4 Replies

8. Shell Programming and Scripting

UNIX Shell Scripting / Programming

Hi, I am looking for a PDF or an e-book which can show in details how to do Shell Scripting or Programming. Can anybody provide me with a link to such a tutorial? I have downloaded some tutorials but they show only basics and not give any in-depth study material. I am using Red Hat Linux... (2 Replies)
Discussion started by: indiansoil
2 Replies

9. Shell Programming and Scripting

shell programming in unix

Hi, Iam using split command to split the files. Splitted files will be named as xaa xab xac xad etc. Directories will be test1,test2,test3..... Now i want to select the splitted files one by one and have to place in test directories. can anybody give the soulution??? (2 Replies)
Discussion started by: nivas
2 Replies

10. Shell Programming and Scripting

Check lists for Unix Shell Programming

Hi all, Can anyone provide me any checklists or a list of steps I should follow before executing my scripts. Could also tell me if there are any other standards to be followed while shell programming like naming conventions for variables etc. Your help would be much appreciated. Regards,... (2 Replies)
Discussion started by: srikanth_ksv
2 Replies

Featured Tech Videos