Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Using freq in script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using freq in script
# 1  
Old 03-11-2015
Using freq in script
Hi,

I have a requirement where I need to check a column data in a fixed length file.

Sample file.
Code:
1 10 20150201 TEST1 TEST1
2 20 20140201 TEST2 TEST2
3 30 20150810 TEST3 TEST3
4 10 20150201 TEST4 TEST4
5 10 20150201 TEST5 TEST5
6 30        0 TEST6 TEST6
7 20 20140201 TEST7 TEST7
8 10 20140201 TEST8 TEST8
9 20 20120609 TEST9 TEST9
1  0 20120609 TEST0 TEST0
2 20 20150201 TEST2 TEST2
3  0        0 TEST3 TEST3
4  0        0 TEST4 TEST4
5  0 20150201 TEST5 TEST5

The characters 6-13 is a date column for which the value has to be 8 digits. I don't need to validate the date format that its an actual date, only that the value is 8 digits.

I am able to do a freq on the file which will tell me the counts per value:
Code:
/>freq inputfile.txt 6 13
inputfile.txt
       0  3
20120609  2
20140201  3
20150201  5
20150810  1
Total    14

Can anyone help me as to how this can be checked in a shell script?

Thanks.
# 2  
Old 03-11-2015
Not sure what output you are after. Here I report if file contains any invalid rows:

Code:
if grep -qvE '^.{5}[0-9]{6}' inputfile.txt
then
    echo "File is invalid"
else
    echo "File is valid or empty"
fi

This User Gave Thanks to Chubler_XL For This Post:
member2014
# 3  
Old 03-11-2015
If you are reading the file in line by line in a loop, you could add this type of thing too:-
Code:
i=0
while read col1 col2 col3 col4 col5
do
   ((i=$i+1))
   if [ "${#col3}" -ne 8 ]
   then
      echo "Error on line $i"
      sed -n${i}p file           # Read out the illegal line directly from the file
   fi
done < file


Does that give you an option to the above suggestion? Which way suits your existing code?



Robin
This User Gave Thanks to rbatte1 For This Post:
member2014
# 4  
Old 03-11-2015
In Chubler_XL's proposal, 8 digits should be checked, and you could stop after the first invalid entry, saving some time on large files:
Code:
if grep -qvEm 1 '^.{5}[0-9]{8}' inputfile.txt

These 2 Users Gave Thanks to RudiC For This Post:
Chubler_XL member2014
# 5  
Old 03-11-2015
Thanks Chubler_XL, Robin and RudiC. This is really helpful.

I changed Chubler_XL's code to use 8 characters and it is working perfectly.

Robin - Thanks for your suggestion. The test file I have provided here is just a sample. The original file have over 150 columns and is a big file with around 1MM records per cycle.
I think the if statement will be better for this file so we dont' have to read all records.

RudiC - your suggestion on exit when you hit an invalid entry is really good. That way we don't have to go through the entire file.
# 6  
Old 03-11-2015
Nice pickup on the number of digits RadiC, must have miscounted there.

I'm fairly sure that -q will exit on first matching record so -m 1 will not add any more efficiency.
These 2 Users Gave Thanks to Chubler_XL For This Post:
member2014 RudiC
# 7  
Old 03-11-2015
Yes - should have read the entire -q entry in man grep...
Login or Register to Ask a Question

Previous Thread | Next Thread

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to block first bash script until second bash script script launches web server/site?

I'm new to utilities like socat and netcat and I'm not clear if they will do what I need. I have a "compileDeployStartWebServer.sh" script and a "StartBrowser.sh" script that are started by emacs/elisp at the same time in two different processes. I'm using Cygwin bash on Windows 10. My... (3 Replies)
Discussion started by: siegfried
3 Replies

2. Shell Programming and Scripting

Shell script works fine as a standalone script but not as part of a bigger script

Hello all, I am facing a weird issue while executing a code below - #!/bin/bash cd /wload/baot/home/baotasa0/sandboxes_finance/ext_ukba_bde/pset sh UKBA_publish.sh UKBA 28082015 3 if then echo "Param file conversion for all the areas are completed, please check in your home directory"... (2 Replies)
Discussion started by: ektubbe
2 Replies

3. UNIX for Dummies Questions & Answers

Calling a script from master script to get value from called script

I am trying to call a script(callingscript.sh) from a master script(masterscript.sh) to get string type value from calling script to master script. I have used scripts mentioned below. #masterscript.sh ./callingscript.sh echo $fileExist #callingscript.sh echo "The script is called"... (2 Replies)
Discussion started by: Raj Roy
2 Replies

4. Shell Programming and Scripting

Script will keep checking running status of another script and also restart called script at night

I am using blow script :-- #!/bin/bash FIND=$(ps -elf | grep "snmp_trap.sh" | grep -v grep) #check snmp_trap.sh is running or not if then # echo "process found" exit 0; else echo "process not found" exec /home/Ketan_r /snmp_trap.sh 2>&1 & disown -h ... (1 Reply)
Discussion started by: ketanraut
1 Replies

5. Shell Programming and Scripting

create a shell script that calls another script and and an awk script

Hi guys I have a shell script that executes sql statemets and sends the output to a file.the script takes in parameters executes sql and sends the result to an output file. #!/bin/sh echo " $2 $3 $4 $5 $6 $7 isql -w400 -U$2 -S$5 -P$3 << xxx use $4 go print"**Changes to the table... (0 Replies)
Discussion started by: magikminox
0 Replies
Login or Register to Ask a Question