Bash Script to read a file and parse each record


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash Script to read a file and parse each record
# 1  
Old 02-19-2010
Bash Script to read a file and parse each record

Hi Guys,

I am new to unix scripting and I am tasked to parse through a CSV file delimited by #.

Sample:

sample.csv
Code:
H#A#B#C
D#A#B#C
T#A#B#C

H = Header
D = Detail Record
T = Tail


What I need is to read the file and parse through it to get the columns.
I have no idea on how to do this in bash script.

I'd appreciate any assistance. Thanks.

Last edited by zaxxon; 02-19-2010 at 04:23 AM.. Reason: use code tags please, ty
# 2  
Old 02-19-2010
What output do you need (example inside code tags please)?
Maybe a while/read construct or awk is an option for you.
# 3  
Old 02-19-2010
Hi

something like this. its not working very well.

Code:
#!/bin/bash
i=0
cat sample.csv | while read fileline
do
echo "$fileline"
row[i]=$fileline
i=$i+1
done

*** parse all rows in row array ***

i would like to output the detail record [D] only

Code:
A
B
C

# 4  
Old 02-19-2010
If just the output is desired and column number is fixed:
Code:
awk -F"#" '/^D/ {print $2,$3,$4}' infile
A B C

If there are varying number of columns in a row, maybe something like this (could also use something with split function in awk):
Code:
$> cat infile
H#A#B#C
D#A#B#C#D#E
T#A#B#C
$> awk -F# '/^D/ {for(a=2; a<=NF; a++){print $a}}END{printf"\n"}' ORS=" " infile
A B C D E

If you want to have every element of the row as element of a bash array:
Code:
$> arr=(`awk -F"#" '/^D/ {print $2,$3,$4}' infile`)
$> echo ${arr[*]}
A B C
$> echo ${arr[1]}
B

If you like it more with usage of shell capabilities:
Code:
$> LINE=`grep ^D infile`
$> echo ${LINE#*#}| tr -s '#' ' '
A B C

and there are many more ways to do it.
# 5  
Old 02-19-2010
MySQL Just Run this in Command Prompt

Code:
cut -d# -f1 sample.csv >> firstcolumn.csv

Basically, it copies the first column and generated as in firstcolumn.csv
Code:
H
D
T
.
.
.

Alternatively, you can change the fields as
Code:
cut -d# -f2 sample.csv >> secondcolumn.csv

or
Code:
cut -d# -f1,2 sample.csv >> firstandsecondcolumn.csv

Good Bye.

Last edited by radoulov; 02-19-2010 at 06:42 AM.. Reason: Use code tags, please!
# 6  
Old 02-19-2010
Thanks for your reply guys. i appreciate it.

now i am trying something like this to work.
- read the csv file
- put the lines in a row array
- next loop would be to parse each row to get the columns

csv file
Code:
Record Type#Batch Job ID#Batch Number#FileCreation Date#FileCreation Time#Production/Test Fileindicator#File Character
H#0002#0002#20100218#17.25#P#barani

Record Type#A#B#C#D#E#F#G#H#J#K#L
1#2#3#4#5#6#7#8#9#10#11

Record Type#NumberOFRecords
T#4500

N

script:
Code:
#!/bin/bash
i=0
cat 1.csv | while read fileline
do
echo "$fileline"
row[i]=$fileline]
i=$(($i+1))
done

j=0
while [ $j -le $i ]
do
col1=$( cut –f 2-7  -d '#' –s $row[i])
echo "first char done"
col2=$( cut –f 1,3-7  -d '#' –s $row[i])
echo "sec char done"
col3=$( cut –f 1-2,4-7  -d '#' –s $row[i])
echo "3rd char done"
echo "$col1,$col2,$col3"
j=$(($j+1))
done

is this possible?
appreciate all your replies guys. thank you!

Last edited by 3vilwyatt; 02-19-2010 at 06:56 AM.. Reason: added csv file
# 7  
Old 02-21-2010
Code:
# you can parse the line using IFS and array
IFS="#" flds=( $fileline )
nrofflds=${#flds[@]}

fld1="${flds[0]}"  
fld=0
while (( fld<nrofflds ))
do
       #...  
       ((fld+=1))
done

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

BASH script to read external file to perform text replacements?

Hi all, I have a moderate size (300 lines) BASH Shell script that performs various tasks on different source reports (CSV files). One of the tasks that it performs, is to use SED to replace 'non-conforming' titles with conformant ones. For example "How to format a RAW Report" needs to become... (3 Replies)
Discussion started by: richardsantink
3 Replies

2. Shell Programming and Scripting

'Couldn't read file' error in bash script with expect, sed and awk!

Ok, so I have a bash script with an embedded expect statement. Inside of the expect statement, i'm trying to pull all of the non-comment lines from the /etc/oratab file one at a time. Here's my command: cat /etc/oratab |sed /^s*#/d\ | awk 'NR==1'|awk -F: '{print \"$1\"}'|. oraenv Now,... (0 Replies)
Discussion started by: alexdglover
0 Replies

3. Shell Programming and Scripting

Read record from the text file & assign those values to variables in the script

For eg: I have sample.txt file with 4 rows of record like: user1|password1 user2|password2 user3|password3 user4|password4 The username and password is sepsrated by '|' I want to get the 1st row value from the file and assign it to two different variables(username and password) in my... (1 Reply)
Discussion started by: priya001
1 Replies

4. Shell Programming and Scripting

Bash Script Help...search then read from file: change text and pipe back...

Hello, I am trying to make a bash script that can pull data from a file and then change one part of said data. I want to search by username and pull the full line. That way there is a way to replace just one part of that line then return it back to the file. My Data is stored like: ... (1 Reply)
Discussion started by: serverfull
1 Replies

5. Shell Programming and Scripting

Bash Shell Script to parse file

Raw Results: results|192.168.2|192.168.2.1|general/udp|10287|Security Note|For your information, here is the traceroute from 192.168.2.24 to 192.168.2.1 : \n192.168.2.24\n192.168.2.1\n\n results|192.168.2|192.168.2.1|ssh (22/tcp)|22964|Security Note|An SSH server is running on this port.\n... (2 Replies)
Discussion started by: jroberson
2 Replies

6. Shell Programming and Scripting

Bash script to read file location

I'm writing a bash script that reads a file location from a user, and I'm wondering how to get the script to accept tab to auto complete the directories that are input. (8 Replies)
Discussion started by: Prodiga1
8 Replies

7. Shell Programming and Scripting

Help w/ script to read file and parse log message

Hi, I am working on the script to parsing the specific message like "aaaa" in multiple log files like N1-***,N2-***,N3-***... The script is to find the list of lof files which contains the message "aaaa" and export the list into excel filE. Can anyone give help? Thanks (2 Replies)
Discussion started by: shyork2001
2 Replies

8. Shell Programming and Scripting

How to read and parse the content of csv file containing # as delimeter into fields using Bash?

#!/bin/bash i=0 cat 1.csv | while read fileline do echo "$fileline" IFS="#" flds=( $fileline ) nrofflds=${#flds} echo "noof fields$nrofflds" fld=0 while do echo "noof counter$fld" echo "$nrofflds" #fld1="${flds}" trying to store the content of line to fields but i... (4 Replies)
Discussion started by: barani75
4 Replies

9. UNIX for Dummies Questions & Answers

how to read record by record from a file in unix

Hi guys, i have a big file with the following format.This includes header(H),detail(D) and trailer(T) information in the file.My problem is i have to search for the character "6h" at 14 th and 15 th position in all the records .if it is there i have to write all those records into a... (1 Reply)
Discussion started by: raoscb
1 Replies

10. Shell Programming and Scripting

read record from file

Hi all I have file f1 like this: Set AM/PM indicator to PM started|14155| Generate Error Re|7| Projected Cash Ba|741| Roll System Date |4| Projected Cash Balances started|2| Process Mark To Market started|13429| Process paydowns started|14189| Process Fixed Inc|439| Process Mark To... (3 Replies)
Discussion started by: koti_rama
3 Replies
Login or Register to Ask a Question