Output large volume of data to CSV file Post: 302956682

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Large volume file formatting

Hi, I have a file which is around 193 gb in size. This file has tonnes of spaces and I need to sanitize it. I tried to use awk script to split this file but it gave me an error like line to long... As of now I am using a sed command to search replace the spaces; however its too slow for such a...

2. Shell Programming and Scripting

Updating a line in a large csv file, with sed/awk?

I have an extremely large csv file that I need to search the second field, and upon matches update the last field... I can pull the line with awk.. but apparently you cant use awk to directly update the file? So im curious if I can use sed to do this... The good news is the field I want to...

3. Shell Programming and Scripting

select data from oracle table and save the output as csv file

Hi I need to execute a select statement in a solaris environment with oracle database. The select statement returns number of rows of data. I need the data to be inserted into a CSV file with proper format. For that we normally use "You have to select all your columns as one big string,...

4. Shell Programming and Scripting

Help Manipulating Large Csv File

Hello everyone, I am trying to manipulate a large .csv file where I have output similar to the following - http://imgur.com/TEXD8.png The result that I am looking for would be to consolidate the first column, but combine the second and third column so it still relates to the first. I...

5. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Background ------------- The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files. File-1 ------ Contains 50,000 rows with 2 fields in each row, separated by pipe. Row structure is like Object_Id|Object_Name, as following: 111|XXX 222|YYY 333|ZZZ ...

6. Shell Programming and Scripting

Compare 2 files of csv file and match column data and create a new csv file of them

Hi, I am newbie in shell script. I need your help to solve my problem. Firstly, I have 2 files of csv and i want to compare of the contents then the output will be written in a new csv file. File1: SourceFile,DateTimeOriginal /home/intannf/foto/IMG_0713.JPG,2015:02:17 11:14:07...

7. Shell Programming and Scripting

Save output of updated csv file as csv file itself, part 2

Hi, I have another problem. I want to sort another csv file by the first field. result.csv SourceFile,Airspeed,GPSLatitude,GPSLongitude,Temperature,Pressure,Altitude,Roll,Pitch,Yaw /home/intannf/foto5/2015_0313_090651_219.JPG,0.,-7.77223,110.37310,30.75,996.46,148.75,180.94,182.00,63.92 ...

8. Shell Programming and Scripting

Copy data to CSV file from txt output

Hi Everyone , Below is output from a TL1 , I want just the NE Name: and beside that the Temperature and the voltages in a csv file , Is this possible? > act-user:AB1S2a:ArshadFO:493::**********; AB1S2a 2016-02-07 10:13:24 M 493 COMPLD "ArshadFO:2016-02-07 10-04-55,0" ;...

9. Shell Programming and Scripting

Need a script to parse data and output to csv

I am not too savvy with arrays and am assuming that what I am looking for needs arrays. This is my requirement. So I have the raw data that gets updated to a log as shown below StudentInfo: FullInfo = { Address = Newark Age = 20 Name= John } StudentInfo:...

10. Programming

Python script to run multiple command and append data in output csv file

Experts, I am writing a script and able to write only small piece of code and not able to collect logic to complete this task. In input file have to look for name like like this (BGL_HSR_901_1AG_A_CR9KTR10) before sh iss neors. Record this (BGL_HSR_901_1AG_A_CR9KTR10) in csv file Now have to...

LEARN ABOUT DEBIAN

dbatbl

DBATBL(1)						      General Commands Manual							 DBATBL(1)

NAME

       dbatbl - Manage on-disk reference tables for DB-ALLe

SYNOPSIS

       dbatbl [command] [options] [args...]

DESCRIPTION

       This tool allows to index and query the tables that are needed for normal functioning of DB-ALLe.

       dbatbl always requires a non-switch argument, that indicates what is the operation that should be performed:

       help
	      Print a help summary.

       help manpage
	      Print this manpage.

       cat tableid [tableid [...]]
	      Output all the contents of a WMO B table..

       grep string
	      Output all the contents of the local B table whose description contains the given string..

       expand table-entry [table-entry [...]]
	      Describe a WMO B table entry or expand a WMO D table entry in its components..

       expandcode varcode [varcode [...]]
	      Expand the value of a packed variable code.

       index [options] filename index-id
	      Index the contents of a table file.

       describe [options] what [values]
	      Invoke the formatter to describe the given values.  Supported so far are: "level ltype l1 l2", "trange pind p1 p2".

OPTIONS

       dbatbl follows the usual GNU command line syntax, with long options starting with two dashes (`-').

   Option for command cat
       -?, --help
	      print an help message

       -c, --csv
	      output variables in CSV format

   Option for command grep
       -?, --help
	      print an help message

       -c, --csv
	      output variables in CSV format

   Option for command expand
       -?, --help
	      print an help message

   Option for command expandcode
       -?, --help
	      print an help message

   Option for command index
       -?, --help
	      print an help message

       -t type, --type=type
	      format of the table to index ('b', 'd', 'conv')

   Option for command describe
       -?, --help
	      print an help message

AUTHOR

       dbatbl has been written by Enrico Zini <enrico@enricozini.com> for ARPA Emilia Romagna, Servizio Idrometeorologico.

								   may 16, 2012 							 DBATBL(1)