Help Manipulating Large Csv File

Tags

shell scripts

Top Forums Shell Programming and Scripting Help Manipulating Large Csv File

09-01-2010

Registered User

5,690, 630

Join Date: Jan 2007

Last Activity: 9 January 2017, 4:40 AM EST

Location: Варна, България / Milano, Italia

Posts: 5,690

Thanks Given: 184

Thanked 630 Times in 587 Posts

OK,
you need something different:

Code:

awk -F, 'END {
  print p, c2, c3, x
  }
$1 != "" && p && p != $1 {
  print p, c2, c3, x
  c2 = c3 = x
  }
{ 
  r2 = $(NF - 1); r3 = $NF
  c2 = c2 ? c2 FS r2 : r2
  c3 = c3 ? c3 FS r3 : r3
  $1 == "" || p = $1
  }' OFS=\; infile

This User Gave Thanks to radoulov For This Post:

radoulov

View Public Profile for radoulov

Find all posts by radoulov

09-01-2010

Registered User

5, 0

Join Date: Sep 2010

Last Activity: 1 December 2010, 10:17 PM EST

Posts: 5

Thanks Given: 2

Thanked 0 Times in 0 Posts

This is exactly what I was looking for. Thank you!

xxwohxx

View Public Profile for xxwohxx

Find all posts by xxwohxx

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Output large volume of data to CSV file

I have a program that output the ownership and permission on each directory and file on the server to a csv file. I am getting error message when I run the program. The program is not outputting to the csv file. Error: the file access permissions do not allow the specified action cannot...

2. Shell Programming and Scripting

Compare 2 files of csv file and match column data and create a new csv file of them

Hi, I am newbie in shell script. I need your help to solve my problem. Firstly, I have 2 files of csv and i want to compare of the contents then the output will be written in a new csv file. File1: SourceFile,DateTimeOriginal /home/intannf/foto/IMG_0713.JPG,2015:02:17 11:14:07...

3. Shell Programming and Scripting

Comparing two large unsorted csv files

Hi All, My requirement is to write a shell script to compare two large csv files. I've created sample files for explaining my problem i.e., a.csv and b.csv contents of files: ----------------- a.csv ------ Type,Memory (Kb),Location HD,Size (Mb),Serial # XT,640,D402,0,MG0010...

4. Shell Programming and Scripting

Merging Very large CSV files in Unix

Hi, I have two very large CSV files, which I want to merge (equi-join) based on a key (column). One of the file (say F1) would have ~30 MM records and 700 columns. The other file (~f2) would have same # of records and lesser columns (say 50). I want to create an output file joining on a...

5. SCO

Need advice: Copying large CSV report files off SCO system

I have a SCO Unix server from 1999 running SCO 5.0.5 and some ancient accounting software called Real World A report writer program on the system is used to generate CSV files from accounting that we write with DOSCOPY commands to 3.5" floppies In the next 60 days we will be decommissioning...

6. Shell Programming and Scripting

Manipulating csv file

We need to convert a field in a csv file which is in cents to dollars.(divide by 100) in our shell script. Can some body help me?

7. Shell Programming and Scripting

Updating a line in a large csv file, with sed/awk?

I have an extremely large csv file that I need to search the second field, and upon matches update the last field... I can pull the line with awk.. but apparently you cant use awk to directly update the file? So im curious if I can use sed to do this... The good news is the field I want to...

8. Shell Programming and Scripting

Sed or awk script to remove text / or perform calculations from large CSV files

I have a large CSV files (e.g. 2 million records) and am hoping to do one of two things. I have been trying to use awk and sed but am a newbie and can't figure out how to get it to work. Any help you could offer would be greatly appreciated - I'm stuck trying to remove the colon and wildcards in...

9. Shell Programming and Scripting

manipulating csv to add extra row

hi how do i manipulate .csv file to add an extra row after each row using shell script? I need a blank line added for each 1000 records in my file? I will then need to copy and paste some data in the blank row created. thanks 4 ur support neil

Shell Programming and Scripting