Help to parse csv file with shell script


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Help to parse csv file with shell script
# 1  
Old 01-18-2012
[Solved] Help to parse csv file with shell script

Hello !

I am very aware that this is not the first time this question is asked here, because I have already read a lot of previous answers, but none of them worked, so...
As said in the title, I want to read a csv file with a bash script.

Here is a sample of the file:

Code:
steev	YES	RGBA8888	0.55	2	2	2	2048
vahine	YES	RGBA8888	0.55	2	2	2	2048
winnick	YES	RGBA8888	0.55	2	2	2	2048

(Copied and pasted from Excel for Mac, but there should be ";")

And here is the simplified code I am trying to make work:

Code:
export IFS=";"

    file="Test.csv"
    more $file | while read name
    do
        echo $name
    done

And... it won't work. Nothing comes out.
If I just do "more $file", then I have a beautiful output:
Code:
vahine;YES;RGBA8888;0.55;2;2;2;2048^Mwinnick;YES;RGBA8888;0.55;2;2;2;2048^Mvenus;YES;RGBA8888;0.55;1;2;1;2048^Mvesper;YES;RGBA8888;0.55;1;2;1;2048

(I could not reproduce it here, but the "^M" are written in white on a black background.)

I have tried using cat or echo instead of more, without success, I have even tried sed and awk with some of the many solutions given on the internet, but they did not work neither, and I would prefer not to use awk if possible, since I do not really understand it yet.

Another strange thing is that if I do:
Code:
    fileContent=$(more $file)
    echo $fileContent

Then I have as output only the last line of the file and not the others...

I am kind of stuck here, without any additionnal idea.
I don't even absolutely want to use csv files, it is just that this format is convenient if many people have to change the source file, so if you have in mind a format easier to read with bash, let me know.

I hope I've given all the needed information and that one of you will have an idea...
# 2  
Old 01-18-2012
Sorry I'm not a bash guy, but this works using ksh93 on Solaris. Perhaps it is easlily translated to bash:

Input file:
Code:
$ cat efs.csv
steev,YES,RGBA8888,0.55,2,2,2,2048
vahine,YES,RGBA8888,0.55,2,2,2,2048
winnick,YES,RGBA8888,0.55,2,2,2,2048
$

Code:
Code:
#!/usr/dt/bin/dtksh

while IFS=',' read name val1 val2 rest_of_line
do
  print $name
done < efs.csv

exit 0

Output:
Code:
$ efs1
steev
vahine
winnick
$

# 3  
Old 01-19-2012
Thanks for the try, but it does not work :/ The code you provided does not give anything in output Smilie

What is more, when I do a simple cat on my csv file, only the last line is displayed... Maybe there are some libs I don't have to support csv reading with bash ?
# 4  
Old 01-19-2012
dos2unix - convert text file from DOS format to ISO format .. Try with this ..
Code:
 
export IFS=";"
file="Test.csv"
dos2unix $file $file 2>/dev/null
more $file | while read name
do
        echo $name
done

# 5  
Old 01-19-2012
Nothing more Smilie I've also tried mac2unix (since I'm on a Mac), but no result neither... If I write:

Code:
export IFS=";"
file="../assets/Test.csv"
dos2unix $file $file 2>/dev/null
echo "about to read"
more $file | while read name
do
echo "a line"
echo $name
done
echo "end"

The only output I have is:

Code:
about to read
end

Smilie

---------- Post updated at 03:45 PM ---------- Previous update was at 11:14 AM ----------

Wow, finally I got something that works... I could not explain why this works and not the other solution...

I used "-e" option of cat, which gives me the entire file, and then changed the "^M" with sed, and then splat it with tr... basically:

Code:
    cat -e $file | sed "s/\^M/\`/g") | while read name
    do
        echo $name | tr "\`" "\n" | while read -ra line
        do
            echo $line
        done
    done

And... it works... It's dirty, I don't understand it, but it works Smilie Thanks for the advises however!
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help with Shell Scrip in Masking particular columns in .csv file or .txt file using shell script

Hello Unix Shell Script Experts, I have a script that would mask the columns in .csv file or .txt file. First the script will untar the .zip files from Archive folder and processes into work folder and finally pushes the masked .csv files into Feed folder. Two parameters are passed ... (5 Replies)
Discussion started by: Mahesh G
5 Replies

2. Shell Programming and Scripting

Korn shell program to parse CSV text file and insert values into Oracle database

Enclosed is comma separated text file. I need to write a korn shell program that will parse the text file and insert the values into Oracle database. I need to write the korn shell program on Red Hat Enterprise Linux server. Oracle database is 10g. (15 Replies)
Discussion started by: shellguy
15 Replies

3. Shell Programming and Scripting

Bash Shell Script to parse file

Raw Results: results|192.168.2|192.168.2.1|general/udp|10287|Security Note|For your information, here is the traceroute from 192.168.2.24 to 192.168.2.1 : \n192.168.2.24\n192.168.2.1\n\n results|192.168.2|192.168.2.1|ssh (22/tcp)|22964|Security Note|An SSH server is running on this port.\n... (2 Replies)
Discussion started by: jroberson
2 Replies

4. Shell Programming and Scripting

Parse file from 2nd line in shell script

Hi, I need to parse input file from 2nd line. Input file contents are, ABC123;20100913115432;2000000;NO; 04;AAA;09;DDD;601020304;AAAA;1;OPTA1;OPTA2;;; 04;BBB;09;BBB;601020304;BBBB;0;OPTB1;OPTB2;OPTB3;OPTB4;OPTB5; 04;CCC;09;DDD;601020304;CCCC;1;;;;; For each line, 1] I need to check... (17 Replies)
Discussion started by: Poonamol
17 Replies

5. Shell Programming and Scripting

Parse XML file in shell script

Hi Everybody, I have an XML file containing some data and i want to extract it, but the specific issue in my file is that the data is repeated some times like the following example : <section1> <subsection1> X=... Y=... Z=... <\subsection1> <subsection2> X=... Y=... Z=...... (2 Replies)
Discussion started by: yassine
2 Replies

6. UNIX for Advanced & Expert Users

shell script to parse html file

hi all, i have a html file something similar to this. <tr class="evenrow"> <td class="data">added</td><td class="data">xyz@abc.com</td> <td class="data">filename.sql</td><td class="modifications-data">08/25/2009 07:58:40</td><td class="data">Added TK prof script</td> </tr> <tr... (1 Reply)
Discussion started by: sais
1 Replies

7. Shell Programming and Scripting

Parse XML file into CSV with shell?

Hi, It's been a few years since college when I did stuff like this all the time. Can someone help me figure out how to best tackle this problem? I need to parse a file full of entries that look like this: <eq action="A" sectyType="0" symbol="PGR" exch="CA" curr="VEF" sess="NORM"... (7 Replies)
Discussion started by: Pcushing
7 Replies

8. Shell Programming and Scripting

Help!!! Shell script to parse data file.

I am faced with a :confused: tricky problem to parse a data file ( May not be a tricky problem to the scripting guru's ). Here is what I am faced with. I have a file with multiple rows of data and the rows are not of fixed length. "|" is used as a delimiters for individual columns and each row... (3 Replies)
Discussion started by: yajaykumar
3 Replies

9. Shell Programming and Scripting

Unix Script to parse a CSV

I am writing a unix script that will parse a CSV and edit the values. My CSV looks like this 0,0,0,0,1,0,1,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,2,0,0,0,0,0,0 10,11,7,0,4,12,2,3,7,0,11,3,12,4,0,5,5,4,5,0,8,6,12,0,9,3,3,0,2,7,8... (16 Replies)
Discussion started by: RJ17
16 Replies

10. Shell Programming and Scripting

Parse a string in XML file using shell script

Hi! I'm just new here and don't know much about shell scripting. I just want to ask for help in creating a shell script that will parse a string or value of the status in the xml file. Please sample xml file below. Can you please help me create a simple script to get the value of status? Also it... (46 Replies)
Discussion started by: ayhanne
46 Replies
Login or Register to Ask a Question