cat or cut to delimit csv?

Login or Register to Ask a Question and Join Our Community

cat or cut to delimit csv?

Tags

Login to Discuss or Reply to this Discussion in Our Community

Top Forums Shell Programming and Scripting cat or cut to delimit csv?

05-02-2009

Registered User

3, 0

Join Date: May 2009

Last Activity: 2 May 2009, 7:02 AM EDT

Posts: 3

Thanks Given: 0

Thanked 0 Times in 0 Posts

cat or cut to delimit csv?

Hi,

Im a pretty large noob to linux/perl etc and im trying to use mysql slurp to take a delimited file and import it into mysql using stdin (in the hope its faster)
mysqlslurp - slurp <STDIN> into a MySQL table - search.cpan.org
Christopher Brown / MySQL-Slurp - search.cpan.org

Using "cat": This seems to only take a tab delimited file. My import files have varying delimiters e.g. "|" with the quotes. Is there any way to get cat to work with other delimiters such as pipe.

I have tried using cut: This just puts everything in one column

Code:

cut -f1-9 -d="|" myfile.csv | mysqlslurp --database=my_db --table=my_table

I also tried using cat: This doesnt seem to work with mysqlimport options as mysqlslurp mentions that they can be passed through

Code:

cat myfile.csv | mysqlslurp --database=my_db --table=my_table --fields-terminated-by="|"

any help is much appreciated

Cheers

newtony

View Public Profile for newtony

Find all posts by newtony

05-02-2009

Registered User

738, 7

Join Date: Oct 2007

Last Activity: 21 August 2013, 5:20 AM EDT

Location: Bangalore

Posts: 738

Thanks Given: 0

Thanked 7 Times in 7 Posts

Can you paste a sample input line and your expected o/p that will eventually feed back to the my sqlslurp?

cheers,
Devaraj Takhellambam

devtakh

View Public Profile for devtakh

Find all posts by devtakh

05-02-2009

Registered User

3, 0

Join Date: May 2009

Last Activity: 2 May 2009, 7:02 AM EDT

Posts: 3

Thanks Given: 0

Thanked 0 Times in 0 Posts

absolutely

here is one line of delimited info:

Code:

"product name"|"http://example.com/productlink.htm"|"http://example.com/productimage.jpg"|"price $80"

The code I am sending to mysql slurp: (this converts pipes to tabs) I was hoping not to do any conversion and just to use cat but it has no delimiter feature. This also doesnt get rid of the quotes enclosing each field.

Code:

cat myfile.csv | tr '|' '\t' | mysqlslurp --database=mydb --table=mytable

I cant seem to work out how to use the --mysqlimport options. It mentions on the mysqlslurp site that its possible to use them. I was thinking if i can use the options i can use --fields-terminated-by and --fields-enclosed-by etc

from mysqlslurp: http://search.cpan.org/dist/MySQL-Sl...ipt/mysqlslurp

Code:

  mysqlslurp 
      -D | --database                   Database    
      --table                           Table
      [--tmp | --tmpdir | --temp ]      Temporary Directory

      [ --usage ]                       Prints this synopsis
      [ --man ]                         Prints full man page

      [options passed to mysqlimport] 
      ( see mysqlimport --help for a list of options )


  Example: 
    cat file | mysqlslurp --database=my_db --table=my_table

I have a huge amount of these files (millions), so i was hoping to do it all with cat with as little text operation as possible from awk or perl etc

I guess I am looking for the fastest method to upload this to the db.

Cheers for any help

Last edited by newtony; 05-02-2009 at 07:34 AM..

newtony

View Public Profile for newtony

Find all posts by newtony

05-02-2009

Registered User

3, 0

Join Date: May 2009

Last Activity: 2 May 2009, 7:02 AM EDT

Posts: 3

Thanks Given: 0

Thanked 0 Times in 0 Posts

no worries i worked it out - the mysqlimport options do work, i just got --fields-terminated-by=" wrong. I had to play around with this to get it to work

newtony

View Public Profile for newtony

Find all posts by newtony

Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to remove space and delimit a csv file?

Hi All , I am facing a small challenge i need unix command to remove space as well as replace "|" to "," in a csv file . original file : A | B | c | D E | F | G | H I | J | K | L P | M | N | O Expected o/p: A,B,c,D E,F,G,H I,J,K,L P,M,N,O

2. UNIX for Dummies Questions & Answers

Delimit the out of wc -l command.

Hi All, Greetings !!! :) Am having issues with wc -l command. My requirement is to delimit the output with comma seperated values and use that file as input in informatica. wc -l *.ksh Output : 16 espress_raise.ksh 173 espresso_monitoring.ksh 189 total ...

3. Shell Programming and Scripting

Perl script to delimit size of strings

Hello, I have a huge file of over 2,00,00,00 strings in UTF8 format. I have managed to write a script in Perl which sorts them neatly as per their Unicode ranges. However I am now stuck with a script which will pipe out all strings between 3 and 20 letters/characters. I am not very good at...

4. Shell Programming and Scripting

CSV with commas in field values, remove duplicates, cut columns

Hi Description of input file I have: ------------------------- 1) CSV with double quotes for string fields. 2) Some string fields have Comma as part of field value. 3) Have Duplicate lines 4) Have 200 columns/fields 5) File size is more than 10GB Description of output file I need:...

5. UNIX for Advanced & Expert Users

Delimit the Folder Size come under Webapps dir

Hi all, Great thanks to all for support till today..today i came here for 1 new issue :-( in our organization we are developing a job portal web application for a client. using Apache-tomcat we are hosting this application, now i need to delimit the applications directory comer under the webapps...

6. Solaris

how to delimit a colon

I want to know how to delimit a colon in a file name while scp ing from one host to other host. I used forward slash(\) but that is not working.

7. Shell Programming and Scripting

cat in the command line doesn't match cat in the script

Hello, So I sorted my file as I was supposed to: sort -n -r -k 2 -k 1 file1 | uniq > file2 and when I wrote > cat file2 in the command line, I got what I was expecting, but in the script itself ... sort -n -r -k 2 -k 1 averages | uniq > temp cat file2 It wrote a whole...

8. Shell Programming and Scripting

How to delimit a flat file with records with SED

Hi gurus, hoping someone can help with a sed line that can do the following... I have a flat file with about 1000 records, but in order to import into openoffice spreadsheet, I need to create a delimited file. I'd like to do 2 things with the SED command: 1- add a pipe character "|" at the end...

9. UNIX for Dummies Questions & Answers

Difference between cat , cat > , cat >> and touch !!!

Hi Can anybody tell the difference between Difference between cat , cat > , cat >> and touch command in UNIX? Thanks

Login or Register to Ask a Question