Sponsored Content
Top Forums UNIX for Advanced & Expert Users Recursive directory search using ls instead of find Post 302537497 by Corona688 on Friday 8th of July 2011 10:56:46 AM
Old 07-08-2011
Quote:
Originally Posted by bearvarine
One of the phenomena I have noticed over my years of being involved in Unix/Linux is that people tend to over-use shell scripting.

The problem with shell scripting is, simply, performance.
One of the phenomena I have noticed over my years of being involved with UNIX/Linux is that people tend to blame poor shell scripts on the language.

The program above isn't slow because it's shell. It's slow because of things like these:
Code:
ls -lS | egrep '^d' | sed "s/[ \t][ \t]*/ /g" | cut -d" " -f9 | head -n $x | tail -n 1

Six programs and five pipes, to do something you could've done in two or less! How find could be slower I can't imagine -- perhaps he didn't realize find is recursive?
Quote:
Its one thing to accomplish small to medium tasks with a shell script. But once you begin doing serious processing work, involving tight loops of text processing, you will very quickly run into trouble. The reason is because most things done in a shell script are done by small programs - cut, sed, awk, head, tail, etc. When you combine dozens of these in a loop that will be running heavily - the computer has to launch THOUSANDS of tiny programs to accomplish the overall task.
If you'd been programming shell for years, you ought to know:

1) awk can make a decent a replacement for all the tools you listed above -- in combinations, even -- being capable of quite complex programs in its own right. Putting it in the same class as head, tail, etc is a bit of a misnomer and jamming it in the middle of a long pipe chain is generally misuse: awk can often replace the entire chain, sometimes the entire script.

2) It's often not necessary to run thousands of tiny processes to accomplish single tasks when people have chosen to do so. Efficient use of piping or external programs are powerful features, but too often they're abused, causing terrible performance.

Quote:
I have seen large powerful Unix systems brought to their knees by simple DB loader scripts done in ksh for this very reason.
Funny thing -- I've done that with Perl. I've also done it in assembly language. It's possible to write terrible code in any language. Smilie
Quote:
The solution is to use a more appropriate software tool to solve the problem. If you really want to do it in shell scripting style, why not try it in perl or python?
because those don't resemble shell languages? Someone who writes a shell script precisely the same way they'd write a perl or python one isn't utilizing the shell's important features.
Quote:
I know this to be a fact, because I've had a simple perl file and text search program in my toolbox since 1994.
Did you know many modern shells have regular expressions, can do substrings and simple text replacement, can pipe text between entire code blocks, can read line by line or token by token and split lines on tokens, can open/close/seek in files, etc, etc, etc -- all as shell builtin features?

All too often, people don't, and use thousands of tiny external programs instead.

The trick is to do large amounts of work with each process you make, never use them for anything trivial.

Last edited by Corona688; 07-08-2011 at 12:21 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

find file with date and recursive search for a text

Hey Guyz I have a requirement something like this.. a part of file name, date of modification of that file and a text is entered as input. like Date : 080206 (MMDDYY format.) filename : hotel_rates text : Jim now the file hotel_rates.ZZZ.123 (creation date is Aug 02 2006) should be... (10 Replies)
Discussion started by: rosh0623
10 Replies

2. UNIX for Dummies Questions & Answers

Unix find command to print directory and search string

Hi i need to print pathname in which the string present using 'find' command sample output like this Pathname String to be searched ---------- -------------------- /usr/test/myfile get /opt/test/somefile get Thanks in... (4 Replies)
Discussion started by: princein
4 Replies

3. Shell Programming and Scripting

non recursive search in the current directory only

Hi, Am trying for a script which should delete more than 15 days older files in my current directory.Am using the below piece of code: "find /tmp -type f -name "pattern" -mtime +15 -exec /usr/bin/ls -altr {} \;" "find /tmp -type f -name "pattern" -mtime +15 -exec /usr/bin/rm -f {} \;" ... (9 Replies)
Discussion started by: puppala
9 Replies

4. Shell Programming and Scripting

search directory-find files-append at end of line

Hi, I have a command "get_data" with some parameters in few *.text files of a directory. I want to first find those files that contain this command and then append the following parameter to the end of the command. example of an entry in the file :- get_data -x -m50 /etc/web/getid this... (1 Reply)
Discussion started by: PrasannaKS
1 Replies

5. UNIX for Dummies Questions & Answers

Restricting a Find search to the current directory only

Hi All, I am trying to delete file (with a mtime older than 2 days) from the current directory ONLY using: find . -daystart -maxdepth 1 -mtime 2 -exec rm {} \; but this doesn't seem to work it is still find files in subdirectories which I don't want to delete. Please can anyone offer... (2 Replies)
Discussion started by: daveu7
2 Replies

6. Shell Programming and Scripting

Find command to search files in a directory excluding subdirectories

Hi Forum, I am using the below command to find files older than x days in a directory excluding subdirectories. From the previous forums I got to know that prune command helps us not to descend in subdirectories. Though I am using it here, not getting the desired result. cd $dir... (8 Replies)
Discussion started by: jhilmil
8 Replies

7. Shell Programming and Scripting

How to restrict Find only search the current directory?

hello, all I have googled internet, read the man page of Find, searched this forum, but still could not figure out how. My current directory is: little@wenwen:~$ pwd /home/little little@wenwen:~$ I want to use find command to list the files in my current directory, how should i write... (3 Replies)
Discussion started by: littlewenwen
3 Replies

8. UNIX for Dummies Questions & Answers

Help needed - find command for recursive search

Hi All I have a requirement to find the file that are most latest to be modified in each directory. Can somebody help with the command please? E.g of the problem. The directory A is having sub directory which are having subdirectory an so on. I need a command which will find the... (2 Replies)
Discussion started by: sudeep.id
2 Replies

9. UNIX for Dummies Questions & Answers

How to search in specific directory using find?

Hi, Is there any way to use find command and search only specific subdirectories in a directory. for example /home/d1 /home/d2 /home/d3 i want to search in the following directories /home /home/d1 /home/d2 i do not want the find command to search the /home/d3 directory. (6 Replies)
Discussion started by: Little
6 Replies

10. Shell Programming and Scripting

Recursive folder search faster than find?

I'm trying to find folders created by a propritary data aquisition software with the .aps ending--yes, I have never encountered folder with a suffix before (some files also end in .aps) and sort them by date. I need the whole path ls -dt "$dataDir"*".aps"does exactly what I want except for the... (2 Replies)
Discussion started by: Michael Stora
2 Replies
All times are GMT -4. The time now is 04:19 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy