UNIX for Advanced & Expert Users

Not so much "fine" as "negligible". Correct. It's a bad habit in general -- test data tends to be small so the problem isn't apparent, only when you make it do real work will you run into trouble.

@Corona688
Yes, 36 cores (9x4). CPU power not an issue. Regularly running over 30,000 concurrent processes.

Bottleneck on reading large files is invariably the disc system, closely followed by the software. This is where reading moderate size files with "cat" scores over the read function in some unix utilities. I recognise that "cut" is actually one of the better ones.

For the advanced user with large data files I am not averse to using "dd" or "cpio" (or both) to read from the disc in an optimum manner.

On a single core system running ancient unix it was very important to minimise the number of concurrent processes. This is really not the case nowadays unless you happen to be running unix on a home system.

Back to the O/P.
The conventional answer is that running more processes is less efficient. On a modern large system with multiple processors (i.e. the norm) it can be more efficient to run a pipeline of multiple efficient processes than to run a single inefficient process.
The "Useless use of cat" brigade have clearly never used a modern computer where apparent inefficiencies are in fact covered by proper utilisation of the software and hardware as a team.
By applying lateral thought we can deduce that hardware design evolution is actually targetted towards making inefficient processes efficient. We can take advantage of that by tactical use of the previously-inefficent processes.

Nuff said.

That hardly compares to the 'cat' being run here, untuned and untunable. You're doubling the amount of work done for a <1% improvement in speed -- and with 30,000 concurrent processes, that's time something else probably could've used. In my early shell-scripting days I wrote scripts that I'm sure would need your 36 processors to function with any efficiency Having CPU power to waste hardly makes it a good idea to do so.

can you suggest one solid link or book that covers these basics...?

My stats do not show waste of CPU power. They show a CPU power saving by using "cat" because "cut" is less efficient at reading files. However in a single stream environment loading multiple processes in a long pipeline would have been a performance disaster.

In my early days or unix I have dealt with system crashes caused by for example: too many concurrent processes; too many forks; disc buffer overload; mysterious kernel crash etc. . It's hard to even generate these situations on modern systems after an initial large-scale kernel build.


There is a very good O'Reilly book "System Performance Tuning" but do bear in mind that it does not cover very large unix systems adequately.

What makes you arrive at the fact that..."cut" is less efficient at reading files than cat.

So reading the file, writing it to a pipe, and reading from the pipe, utilizing two separate CPU's simultaneously is more efficient than reading it once and using it once? If your CPU benchmarks show that this uses less CPU, frankly, they're wrong. Less total real time maybe, but nothing in that reduces the amount of CPU cut uses -- adding more commands can only add more CPU utilization.

The only performance benefit I can see is the pipe effectively acts as a read-ahead buffer, albeit a highly expensive one. With the power expended for that 1% performance improvement, how much more actual work could have been done instead by running two instances of cut on different data sets?

---------- Post updated at 11:08 AM ---------- Previous update was at 10:58 AM ----------

cut has to read line by line. cat can just read and write huge blocks.