Sponsored Content
Top Forums Shell Programming and Scripting Incredibly inefficient cat | grep script Post 302913107 by Chubler_XL on Wednesday 13th of August 2014 06:29:56 PM
Old 08-13-2014
Quote:
Originally Posted by Corona688
I wrote this perl script to do 'split' like things without the temp file mess
Off topic a little (apologies to the OP), but I love this util Corona688!

I was wondering if it could be done as a bash script. I came up with the script below.

While testing I did discover that if lines split evenly it does run the command 1 more time than needed with empty input (e.g. -l 10 for a 100 line file).

Code:
#!/bin/bash
lines=1000
ccount=0

while getopts l: opt 2> /dev/null
do
    case $opt in
        l) lines=$OPTARG ;;
        *) echo "Illegal option -$opt" >&2
           exit 1
        ;;
    esac
done
shift $((OPTIND-1))

if [ $# -le 0 ]
then
    cat >&2 <<EOF
linesplit:  Designed to read from standard input
            and split into multiple streams, running one command per loop
            and writing into its STDIN.  @FNAME@ is available as an
            sequence number if needed.

syntax:  linesplit [-l lines] command arguments @FNAME@ ...
EOF
    exit 1
fi

while [[ ${PIPESTATUS[0]} != 33 ]]
do
    ((ccount++))
    printf -v fname "%08d" $ccount
    for((n=0; n<lines; n++))
    do
        read line && echo "$line" || exit 33
    done | eval ${@//@FNAME@/$fname}
done

printf "Wrote %'d full chunks of %'d lines\n" $((ccount-1)) $lines >&2

eg:

Code:
$ printf "%s\n" $(seq 1 100) | ./linesplit.pl -l 10 wc -l
10
10
10
10
10
10
10
10
10
10
0
Wrote 10 chunks of 10 lines

$ printf "%s\n" $(seq 1 100) | ./linesplit.pl -l 3000 wc -l
100
Wrote 0 chunks of 3000 lines


Last edited by Chubler_XL; 08-13-2014 at 07:45 PM..
This User Gave Thanks to Chubler_XL For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

cat and grep not working

I am trying to cat a file and then grep that file for a number. I can do it fine on other files but this particular file will not do anything. I tried running it on an older file from the same device but it is just not working. The file is nothing more than a flat file on a unix box. Here is just a... (3 Replies)
Discussion started by: jphess
3 Replies

2. Shell Programming and Scripting

cat in the command line doesn't match cat in the script

Hello, So I sorted my file as I was supposed to: sort -n -r -k 2 -k 1 file1 | uniq > file2 and when I wrote > cat file2 in the command line, I got what I was expecting, but in the script itself ... sort -n -r -k 2 -k 1 averages | uniq > temp cat file2 It wrote a whole... (21 Replies)
Discussion started by: shira
21 Replies

3. Shell Programming and Scripting

Perl sum really inefficient!!

Hi all, I have a file like the following: ID, 2,Andrew,0,1,2,3,4,2,5,6,7,7,9,3,4,5,34,3,2,1,5,6,78,89,8,7,6...................... 4,James,0,6,7,0,5,6,4,7,8,9,6,46,6,3,2,5,6,87,0,341,0,5,2,5,6.................... END, (there are more entires on each line but to keep it simple I've left... (10 Replies)
Discussion started by: Donkey25
10 Replies

4. Shell Programming and Scripting

Problem with IF - CAT - GREP in simple shell script

Hi all, Here is my requirement I have to search 'ORA' word in out.log file,if it is present then i need to send that file (out.log) content to some mail id.If 'ORA' word is not in that file then i need to send 'load succesful' message to some mail id. The below the shell script is not... (5 Replies)
Discussion started by: mak_boop
5 Replies

5. Shell Programming and Scripting

cat /etc/passwd and grep -v on /etc/shells

Hi All, I'd like to do this cat /etc/passwd and grep -v on the /etc/shells list I'd like to find all shell that doesn't exist on the /etc/passwd. Is there an easy way without doing a egrep -v "/bin/sh|/bin/bash................"? How do I use a file /etc/shells as my list for... (4 Replies)
Discussion started by: itik
4 Replies

6. Shell Programming and Scripting

cat -n and grep

I am not sure if using cat -n is the most efficient way to split a file into multiple files, one file per line in the source file. I thought using cat -n would make it easy to process the file because it produces an output that numbers each line that I could then grep for with the regex "^ *$i".... (3 Replies)
Discussion started by: kapu
3 Replies

7. Shell Programming and Scripting

grep or cat using sed

Is there a way using grep or cat a file to create a new file based on whether the first 9 positions of each record is less than 399999999? This is a fixed file format. (3 Replies)
Discussion started by: ski
3 Replies

8. UNIX for Dummies Questions & Answers

Grep and cat combined

Hello, i need to search one word (snp1) from many files and copy the content of the columns of this word in new file. example: file 1: SNP BP CHR P snp1 1 3 0.01 snp2 2 2 0.05 . . file 2: SNP BP CHR P snp1 1 3 0.06 snp2 2 2 0.3 output... (6 Replies)
Discussion started by: biopsy
6 Replies

9. Shell Programming and Scripting

Replace cat and grep with <

Hello someone told me to use OS=`awk '{print int($3)}' < /etc/redhat-release` instead of OS=cat /etc/redhat-release | `awk '{print int($3)}'` any idea for the reason ? (5 Replies)
Discussion started by: nimafire
5 Replies

10. UNIX for Dummies Questions & Answers

Grep or cat The Whole Directory PROBLEMS :(

Hi Guys This is my first post so I am not sure how things go here. I'm sorry if I'm breaking the rule or something. Feel free to correct me about that :) So as I was saying... I'd been trying to grep this folder containing 900,000 txt files but seems no luck. I get either "No such file... (6 Replies)
Discussion started by: Nexeu
6 Replies
All times are GMT -4. The time now is 07:48 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy