reading a csv file and creating a flat file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting reading a csv file and creating a flat file
# 1  
Old 07-29-2012
reading a csv file and creating a flat file

hi

i have written a script for reading a csv file and creating a flat file, suggest if this script can be optimized

Code:
#----------------

FILENAME="$1"
SCRIPT=$(basename $0)

#-----------------------------------------//

function usage
{
echo "\nUSAGE: $THIS_SCRIPT  file_to_process\n"
echo "OR - To send the output to a file use: "
echo "\n$THIS_SCRIPT  file_to_process  > output_file_name 2>&1 \n"
exit 1
}

#-----------------------------------------//
append(){
lr=$1
val=$3
size=$2

if [ $1 = "l" ]
then
    echo "$val" >> 2           
else
    val=$(printf "%${size}s" "$val")
    echo "$val" >> 2
   echo "ap no"
fi

}

#-----------------------------------------//

function readline
{

    #while read line
sed 1d $FILENAME | while read line
    do
        echo "$line"
        
#Srl. No,Deleted ?,Field,Size,Alignment,Value

        del=`echo "$line" | cut -d , -f2`
del=${del:-"n"}
del=${del,,}
        size=`echo "$line" | cut -d , -f 4`
        lr=`echo "$line" | cut -d , -f 5`
#echo "$lr"
lr=${lr:-"l"}
lr=${lr,,}
#echo "$lr"

        val=`echo "$line" | cut -d , -f 6`
a=$del
#if [ ${a:-"n"} = "n" -o ${a:-"n"} = "N" ]
if [ ${a:-"n"} = "n" ]
then
append "$lr" "$size" "$val"

echo "okok"
fi
    #done < $FILENAME
done

}

# **************MAIN*************** #

# Looking for exactly one parameter
(( $# == 1 )) || usage

# Does the file exist as a regular file?
[[ -f $1 ]] || usage


readline

cat 2
tr -d '\n' <2 >3

# 2  
Old 07-30-2012
Why not use IFS to split up your values, as this saves calling cut 5 times for each line:

Code:
function readline
{
oldifs=$IFS
IFS=,
sed 1d $FILENAME | while read srl del fld size lr val
do
        del=${del:-"n"}
        del=${del,,}
        lr=${lr:-"l"}
        lr=${lr,,}
        a=$del
        if [ ${a:-"n"} = "n" ]
        then
            append "$lr" "$size" "$val"
            echo "okok"
        fi
done
IFS=$oldifs
}

Also consider using -v option of printf to assign variable and avoid calling another subshell:
Code:
append(){
lr=$1
val=$3
size=$2
if [ $1 = "l" ]
then
    echo "$val" >> 2
else
    printf -v val "%${size}s" "$val"
    echo "$val" >> 2
   echo "ap no"
fi
}


Last edited by Chubler_XL; 07-30-2012 at 12:38 AM..
This User Gave Thanks to Chubler_XL For This Post:
# 3  
Old 07-30-2012
this function will append the required space to the variable
Code:
function append(){
lr=$1
val=$3
pad=$2

#echo ${#val}
#echo $pad
#pad=${#pad}-${#val}
let pad=$pad-${#val}
echo $pad

while [ "$pad" -gt 0 ]
do
        if [ "$lr" = "l" ]
        then
                val=$val" "
                let pad=$pad-1
        else
                val=" "$val
                let pad=$pad-1
        fi
done

echo "[$val]"
echo "$val" >> 2
}

# 4  
Old 07-31-2012
@mprakasheee - Depends on your shell.

If you use ksh good. However, printf -v is builtin in bash so for bash; stick with my append()
This User Gave Thanks to Chubler_XL For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

Reading flat file content

is there any unix tools that can read the text files like through SQL queries? (ie in Hadoop, Impala DB support flat file query) (1 Reply)
Discussion started by: omykarshan
1 Replies

2. Shell Programming and Scripting

Converting csv file to flat file

Hi All, I have a csv file which is comma seperated. I need to convert to flat file with preferred column length country,id Australia,1234 Africa,12399999 Expected output country id Australia 1234 Africa 12399999 the flat file should predefined length on respective... (8 Replies)
Discussion started by: rohit_shinez
8 Replies

3. UNIX for Dummies Questions & Answers

Convert flat file to csv

Hi I have a file like this: a=1 b=2 c=3 a=4 b=2 d=3 a=3 c=4 How can I change this to csv format a,b,c,d 1,2,3,, 4,2,,3 3,,4,, Please use code tags next time for your code and data. Thanks (10 Replies)
Discussion started by: sandip_2014
10 Replies

4. Shell Programming and Scripting

Creating Dynamic Variables from a Flat File

Greetings all, Been trying to do my Googling and forum searches but can't seem to lock in on a solution. I have a script that parses a log and collects all the uniq events to a flat file. Some days might have 50 unique events, other days might have 75. (Hence my reference to dynamic.) ... (2 Replies)
Discussion started by: sjrupp
2 Replies

5. Shell Programming and Scripting

Reading XML data in a FLAT FILE

I have a requirement to read the xml file and split the files into two diffrent files in Unix shell script. Could anyone please help me out with this requirement. Sample file --------------- 0,<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <Information... (3 Replies)
Discussion started by: kmanivan82
3 Replies

6. Shell Programming and Scripting

Searching for Log / Bad file and Reading and writing to a flat file

Need to develop a unix shell script for the below requirement and I need your assistance: 1) search for file.log and file.bad file in a directory and read them 2) pull out "Load_Start_Time", "Data_File_Name", "Error_Type" from log file 4) concatinate each row from bad file as... (3 Replies)
Discussion started by: mlpathir
3 Replies

7. Shell Programming and Scripting

Reading a FLAT File - No Delimeters

Hi Folks, I have a file without any delimeters and it is a flat file. Example, my raw data looks: x25abcy26defz27ghi..... Now, could you please any one help me to program to split this into variable and create a text file. I want a output as below Name Age Number x 25 abc... (14 Replies)
Discussion started by: Jerald Nathan
14 Replies

8. Shell Programming and Scripting

Flat file to csv conversion

Hi Guy's can someone help me in converting the following I have a flat text file which has several thousand lines which I need to convert to a csv it's got a consistent format but basically want every time it hit's txt to create a new line with the subsequent lines comma delimited for example ... (6 Replies)
Discussion started by: p1_ben
6 Replies

9. Shell Programming and Scripting

Awk to convert a flat file to CSV file

Hi , I have a file with contents as below: Contract Cancellation Report UARCNCL LOS CODE DATE REAS TYPE AMOUNT AMOUNT LETTER BY ========= ======= ==== ==== ==== ========= ==== ==== 8174739 7641509 1S NONE CRCD 30-JUN-2008 NPAR N .00 .00 CCAN 8678696 8091709 1S NONE DDEB 30-JUN-2008... (14 Replies)
Discussion started by: rkumudha
14 Replies

10. UNIX for Dummies Questions & Answers

Creating flat text file (ASCII)

Hi everybody. I need help and I hope someone is willing to help me out here. My wholesale company is currently moving to new software. The old software is running on a UNIX platform. We need to migrate data from the UNIX system, but our former software provider refuses to assist the data... (5 Replies)
Discussion started by: Wdonero
5 Replies
Login or Register to Ask a Question