Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help with pipe count in a flat file!!!

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help with pipe count in a flat file!!!
# 8  
Old 06-01-2007
Shell_Life,
I am working under cygwin bash, and the two commands egrep and sed works fine with yours datas :

Code:
$ cat infile
abc
abc|
abc|def|
abc|def|123|
abc|def|123|456|
abc|def|123|456|xyz|
abc|def|123|456|xyz
abc|def|123|456
abc|def|123
abc|def
abc
$ egrep -v '^([^|]*\|){2,2}[^|]*$' infile >errfile
$ diff -y -W 40 infile errfile
abc                     abc
abc|                    abc|
abc|def|            <
abc|def|123|            abc|def|123|
abc|def|123|456|        abc|def|123|456|
abc|def|123|456|        abc|def|123|456|
abc|def|123|456|        abc|def|123|456|
abc|def|123|456         abc|def|123|456
abc|def|123        <
abc|def                 abc|def
abc                     abc
$ sed  '/^\([^|]*|\)\{2\}[^|]*$/d' infile > errfile
$ diff -y -W 40 infile errfile
abc                     abc
abc|                    abc|
abc|def|           <
abc|def|123|            abc|def|123|
abc|def|123|456|        abc|def|123|456|
abc|def|123|456|        abc|def|123|456|
abc|def|123|456|        abc|def|123|456|
abc|def|123|456         abc|def|123|456
abc|def|123         <
abc|def                 abc|def
abc                     abc
$

As you can see, the result of commands contains all records with a number of fields different of 3 (2 separators |).

Jean-Pierre.
# 9  
Old 06-01-2007
Aigles,
I am using unix SunOS and just ran it again (for 2) and it displays
all the lines in the file, not the ones the "egrep" or "sed" asked for.
# 10  
Old 06-02-2007
Shell_Life,
I have tested the command at home on my PC with cygwin.
I will try Monday at work where i use AIX.

In the meantime, try this new command:
Code:
$ sed -n 'h;s/[^|]*//g; /^|\{2\}$/!{;g;p;}' infile > errfile
$ diff -y -W 40 infile errfile
abc                     abc
abc|                    abc|
abc|def|           <
abc|def|123|            abc|def|123|
abc|def|123|456|        abc|def|123|456|
abc|def|123|456|        abc|def|123|456|
abc|def|123|456|        abc|def|123|456|
abc|def|123|456         abc|def|123|456
abc|def|123         <
abc|def                 abc|def
abc                     abc
$

Jean-Pierre.
Last edited by aigles; 06-02-2007 at 03:14 PM..
# 11  
Old 06-02-2007
kumar - you have to use a POSIX shell: zsh, ksh, bash, etc. - not tcsh or csh for this:
Code:
#!/bin/ksh

while read pipes
do
       pipes=$(echo "$pipes" | tr -dc '|')
       if [[ ${#pipes} -lt 42 ]] ; then
       	  echo "$x ${#pipes}"
       fi
done < filename > shortlist.txt

# 12  
Old 06-02-2007
Another way:
Code:
#! /usr/bin/ksh
while IFS="" read line ; do
        oline="$line"
        count=0
        while [[ $line = *\|* ]] ; do
                line=${line#*\|}
                ((count=count+1))
        done
        if ((count != 42)) ; then
                echo $count "$oline"
        fi
done
exit 0

Since it is all in ksh with no external programs it may be faster for you.
# 13  
Old 06-03-2007
Yet anoyher way with ksh :
Code:
#!/usr/bin/ksh
oIFS="$IFS"
while IFS="" read line
do
   IFS='|'
   set -A fields -- $line
   IFS="$oIFS"
   [ ${#fields[*]} -ne 43 ] && echo ${line}
done < infile > errfile

or bash :
Code:
oIFS="$IFS"
while IFS="" read line
do
   IFS='|'
   declare -a fields=(${line}_); 
   IFS="$oIFS"
   [ ${#fields[*]} -ne 43 ] && echo ${line}
done < infile > errfile

Jean-Pierre.
# 14  
Old 06-04-2007
Shell_life,

I ran the egrep and the two sed commands under AIX without any problem.

Jean-Pierre.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Oracle table extract: all columns are not converting into pipe delimited in flat file

Hi All, I am writing a shell script to extract oracle table into a pipe dilemited flat file. Below is my code and I have attached two files that I have abled to generate so far. 1. Table.txt ==> database extract file 2. flat.txt ==> pipe delimited after some manipulation of the original db... (5 Replies)
Discussion started by: express14
5 Replies

2. Shell Programming and Scripting

How to ignore Pipe in Pipe delimited file?

Hi guys, I need to know how i can ignore Pipe '|' if Pipe is coming as a column in Pipe delimited file for eg: file 1: xx|yy|"xyz|zzz"|zzz|12... using below awk command awk 'BEGIN {FS=OFS="|" } print $3 i would get xyz But i want as : xyz|zzz to consider as whole column... (13 Replies)
Discussion started by: rohit_shinez
13 Replies

3. Shell Programming and Scripting

Unique count from flat file

Hello Guys I have a flat file with '|~|' delimited When I use to record count using below command awk -FS"+" ' {print $colno}' filename | wc -l the count is fine But when I am trying to find the unique number of record the o/p is always 1 awk -FS"+" ' {print $colno}'... (11 Replies)
Discussion started by: Pratik4891
11 Replies

4. Shell Programming and Scripting

count of null in pipe delimited txt file

Hi, I have a pipe delimited txt file which contains 17 fields per line/row. 16th field contains email id. I want to count the number of lines/rows that contains null in the 16th field. Plz find attached example data file. I'm looking for a command line/script which achieves this. ... (5 Replies)
Discussion started by: Sriranga
5 Replies

5. Shell Programming and Scripting

Searching for Log / Bad file and Reading and writing to a flat file

Need to develop a unix shell script for the below requirement and I need your assistance: 1) search for file.log and file.bad file in a directory and read them 2) pull out "Load_Start_Time", "Data_File_Name", "Error_Type" from log file 4) concatinate each row from bad file as... (3 Replies)
Discussion started by: mlpathir
3 Replies

6. UNIX for Dummies Questions & Answers

Grep char count & pipe to sed command

Hi I am having a 'grep' headache Here is the contents of my file: (PBZ,CP,(((ME,PBZ,BtM),ON),((ME,((PBZ,DG),(CW9,PG11))),CW9,TS2,RT1))) I would like to count out how many times 'PBZ' occurs and then place that number in the line above 3... (8 Replies)
Discussion started by: cavanac2
8 Replies

7. Shell Programming and Scripting

count alphabets in a flat file and print

I have a text file with a huge dataset, and each row in that dataset has some data(65479 rows). In that file I need to find the number of times a-z & A-Z Appears. How Can I Initialise Array into an Array to parse the count also I need to parse a,A into a single array preferably. example of... (3 Replies)
Discussion started by: vmsenthil
3 Replies

8. Shell Programming and Scripting

Getting Sum, Count and Distinct Count of a file

Hi all this is a UNIX question. I have a large flat file with millions of records. col1|col2|col3 1|a|b 2|c|d 3|e|f 3|g|h footer**** I am supposed to calculate the sum of col1 1+2+3+3=9, count of col1 1,2,3,3=4, and distinct count of col1 1,2,3=c3 I would like it if you avoid... (4 Replies)
Discussion started by: singhabhijit
4 Replies

9. Programming

compare XML/flat file with UNIX file system structure

Before i start doing something, I wanted to know whether the approach to compare XML file with UNIX file system structure. I have a pre-configured file(contains a list of paths to executables) and i need to check against the UNIX directory structure. what are the various approches should i use ? I... (6 Replies)
Discussion started by: shafi2all
6 Replies

10. UNIX for Dummies Questions & Answers

How to count the record count in an EBCDIC file.

How do I get the record count in an EBCDIC file on a Linux Box. :confused: (1 Reply)
Discussion started by: oracle8
1 Replies
Login or Register to Ask a Question