Visit Our UNIX and Linux User Community


Read data from .csv file through shell script & modify


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Read data from .csv file through shell script & modify
# 1  
Old 03-18-2011
Question Read data from .csv file through shell script & modify

I need to read data from a file called "test.csv" through shell script where the file contains values like name,price,descriptor etc. There are rows where descriptor (& in some rows name) are written as string & other characters like "car_+" OR "bike*" etc where it should contains strings like "car" OR "bike".

Problem is I need to read that file & take out error rows & write those rows into a new file say error.csv & update test.csv with correct rows.

Please let me know how to read & check these things & then update.
OS can be either AIX/HP-UX/Sun-Solaris. Suppose script is bourne shell script.

Thanks,
Raj
# 2  
Old 03-18-2011
Quote:
Originally Posted by raj100
I need to read data from a file called "test.csv" through shell script where the file contains values like name,price,descriptor etc. There are rows where descriptor (& in some rows name) are written as string & other characters like "car_+" OR "bike*" etc where it should contains strings like "car" OR "bike".

Problem is I need to read that file & take out error rows & write those rows into a new file say error.csv & update test.csv with correct rows.

Please let me know how to read & check these things & then update.
OS can be either AIX/HP-UX/Sun-Solaris. Suppose script is bourne shell script.
...
Post the following:
(1) Your sample input file.
(2) Your desired output.

Give us something to work with.

tyler_durden
# 3  
Old 03-21-2011
Alright sample input file : (ofcourse this is not the original file but the original file I need to work is similar to this containing more rows & columns. Original file is office confidential.)

File test.csv :
----------------------------------------------------------------
name price descriptor
Car 50 Mustang
Car_ 70 Audi A5
Bike 30 BMW)
SUV@ 80 Fortuner+_
Car 75 Sierra
--------------------------------
1st row is fine
2nd row -> underscore(car_) need to remove it & put it in a file called error.csv
3rd row -> closing brace ) after BMW & again this row should go to error.csv
4th row -> error at 1st column & last column should go to error.csv
5th row is fine
-----

Now Rows 2, 3 & 4 contain error & they need to be written to error file called error.csv & update test.csv with correct rows. Error rows need not be present in test.csv after these checks.
# 4  
Old 03-21-2011
Quote:
Originally Posted by raj100
...
File test.csv :
----------------------------------------------------------------
name price descriptor
Car 50 Mustang
Car_ 70 Audi A5
Bike 30 BMW)
SUV@ 80 Fortuner+_
Car 75 Sierra
--------------------------------
...
"csv" means "comma separated values". There are no commas in your "test.csv" file, so I'll assume your file looks like the one in the following excerpt -

Code:
$
$
$ # show the content of the csv file "f3"
$
$ cat f3
name,price,descriptor
Car,50,Mustang
Car_,70,Audi A5
Bike,30,BMW)
SUV@,80,Fortuner+_
Car,75,Sierra
$
$ # run the awk one-liner to process "f3"
$
$ awk -F, 'NR>1 {if ($1 !~ /^[a-zA-Z0-9 ]+$/ || $3 !~ /^[a-zA-Z0-9 ]+$/){print $0 >"f3.err"} else {print $0 >"f3.tmp"}}' f3 ; mv f3.tmp f3
$
$ # show the content of the error file "f3.err"
$
$ cat f3.err
Car_,70,Audi A5
Bike,30,BMW)
SUV@,80,Fortuner+_
$
$ # show the content of the updated csv file "f3"
$
$ cat f3
Car,50,Mustang
Car,75,Sierra
$
$

tyler_durden

Previous Thread | Next Thread
Test Your Knowledge in Computers #504
Difficulty: Medium
Typically, programs operate in a reverse-linear flow of control.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Is there a way to handle commas inside the data when generating a csv file from shell script?

I am extracting data via sql query and some of the data has commas. Output File must be csv and I cannot update the data in the db (as it is used by other application). Example table FavoriteThings Person VARCHAR2(25), Favorite VARCHAR2(100) Sample Data Greta rain drop on... (12 Replies)
Discussion started by: patk625
12 Replies

2. Shell Programming and Scripting

Read csv file, convert the data and make one text file in UNIX shell scripting

I have input data looks like this which is a part of a csv file 7,1265,76548,"0102:04" 8,1266,76545,"0112:04" I need to make the output data should look like this and the output data will be part of text file: 7|1265000 |7654899 |A| 8|12660000 |76545999 |B| The logic behind the... (6 Replies)
Discussion started by: RJG
6 Replies

3. UNIX for Dummies Questions & Answers

Shell script to read lines in a text file and filter user data Shell Programming and Scripting

sxsaaas (3 Replies)
Discussion started by: VikrantD
3 Replies

4. Shell Programming and Scripting

Shell script to extract data from csv file

Hi everyone, I have a csv file which has data with different heading and column names as below. Static Data Ingested ,,,,,,,,,,,,Known Explained Rejections Column_1,column_2,Column_3,Column_4,,Column_6,Column_7,,% Column_8,,Column_9 ,Column_10 ,... (14 Replies)
Discussion started by: Vivekit82
14 Replies

5. UNIX for Dummies Questions & Answers

Shell script to extract data from csv file

Hi Guys, I am new to shell script.I need your help to write a shell script. I need to write a shell script to extract data from a .csv file where columns are ',' separated. The file has 7 columns having values say column 1,column 2.....column 7 as below along with their values. Name, Address,... (7 Replies)
Discussion started by: Vivekit82
7 Replies

6. UNIX for Dummies Questions & Answers

Shell script to extract data from csv file based on certain conditions

Hi Guys, I am new to shell script.I need your help to write a shell script. I need to write a shell script to extract data from a .csv file where columns are ',' separated. The file has 5 columns having values say column 1,column 2.....column 5 as below along with their valuesm.... (1 Reply)
Discussion started by: Vivekit82
1 Replies

7. Shell Programming and Scripting

Calling Pl/sql function in shell script to modify csv

I need to 1.Open a csv 2.Process the csv i.e. Modify 2 column in the csv. To modify the column the value needs to be passed to a pl/sql function and the return value should be updated For eg: If column 2 E,then E will be passed in database function which will return Employee. 3. Write a... (5 Replies)
Discussion started by: Chinky23
5 Replies

8. Shell Programming and Scripting

Exporting data as a CSV file from Unix shell script

Friends...This is the first time i am trying the report generation using shell script... any suggestions are welcome. Is there a way to set the font size & color when i am exporting the data from unix shell script as a CSV file ? The following sample data is saved as a .csv file in the... (2 Replies)
Discussion started by: appu2176
2 Replies

9. Shell Programming and Scripting

Shell Script to Load data into the database using a .csv file and .ctl file

Since i'm new to scripting i'm findind it difficult to code a script. The script has to be an executable with 2 paramters passed to it.The Parameters are 1. The Control file name(.ctl file) 2. The Data file name(.csv file) Does anybody have an idea about it? :confused: (3 Replies)
Discussion started by: Csmani
3 Replies

10. Shell Programming and Scripting

Korn Shell Script - Read File & Search On Values

I am attempting to itterate through a file that has multiple lines and for each one read the entire line and use the value then to search in other files. The problem is that instead of an entire line I am getting each word in the file set as the value I am searching for. For example in File 1... (2 Replies)
Discussion started by: run_unx_novice
2 Replies

Featured Tech Videos