Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Convert comma separated file to fix length

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Convert comma separated file to fix length
# 1  
Old 05-29-2013
Convert comma separated file to fix length
Hi, I am converting a comma separated file to fixed field lenght and I am using that:
Code:
COLUMNS="25 24 67 26 39 63 20 34 35 14 397"
(
cat $indir/input_file.dat | \
$AWK -v columns="$COLUMNS" '
  BEGIN {
    FS=",";
    OFS="";
    split(columns, arr, " ");
  }
  {
    for(i=1; i<=NF; i++)
      printf("%-*s%c", arr[i], $i, (i==NF) ? RS : OFS)
  }
') >> $outdir/output_file.dat

It is working fine, but most of the files are big and the performance is very slow. Any ideas how can I make it faster?

Thanks!
# 2  
Old 05-29-2013
Step 1: Cut the i/o in half by eliminating the unnecessary cat.
Step 2: Eliminate the for-loop and replace it with a single call to printf.
Step 3: Don't use gawk unless you must. It's the slowest awk implementation.

Regards,
Alister
# 3  
Old 05-29-2013
Actually it is:
Code:
AWK="/usr/xpg4/bin/awk"  #extended awk for solaris

# 4  
Old 05-29-2013
Quote:
Originally Posted by apenkov
Actually it is:
Code:
AWK="/usr/xpg4/bin/awk"  #extended awk for solaris

I wasn't asserting that you are using gawk. Your post did not specify the implementation, so I mentioned gawk's slow execution speed in case it was relevant.

Regards,
Alister
# 5  
Old 05-29-2013
Assuming ASCII data, try
Code:
perl -F, -lape 'BEGIN { ($tmpl = shift) =~ s/(\d+)/A$1/g }
$_ = pack($tmpl, @F)' "$COLUMNS" "$indir/input_file.dat" > "$outdir/output_file.dat"

Last edited by elixir_sinari; 05-29-2013 at 01:43 PM..
This User Gave Thanks to elixir_sinari For This Post:
apenkov
# 6  
Old 06-03-2013
Quote:
Originally Posted by elixir_sinari
Assuming ASCII data, try
Code:
perl -F, -lape 'BEGIN { ($tmpl = shift) =~ s/(\d+)/A$1/g }
$_ = pack($tmpl, @F)' "$COLUMNS" "$indir/input_file.dat" > "$outdir/output_file.dat"

Hi, thanks. I tried it, but unfortunately it is the same Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Convert fixed value fields to comma separated values

Hi All, Hope you are doing Great!!!. Today i have came up with a problem to say exactly it was for performance improvement. I have written code in perl as a solution for this to cut in specific range, but it is taking time to run for files thousands of lines so i am expecting a sed... (9 Replies)
Discussion started by: mad man
9 Replies

2. Shell Programming and Scripting

awk to parse comma separated field and removing comma in between number and double quotes

Hi Experts, Please support I have below data in file in comma seperated, but 4th column is containing comma in between numbers, bcz of which when i tried to parse the file the column 6th value(5049641141) is being removed from the file and value(222.82) in column 5 becoming value of column6. ... (3 Replies)
Discussion started by: as7951
3 Replies

3. Shell Programming and Scripting

Convert column to quote and comma separated row

Hi, I have a list of tables in a file.txt C_CLAIM C_HLD C_PROVIDER I want the output to be 'C_CLAIM','C_HLD','C_PROVIDER' Currently I'm usin awk and getting output which is almost correct but still has minor defects awk -vORS="','" '{ print $1 }' file.txt The output of... (4 Replies)
Discussion started by: wahi80
4 Replies

4. UNIX for Dummies Questions & Answers

[solved] Comma separated values to space separated

Hi, I have a large number of files which are written as csv (comma-separated values). Does anyone know of simple sed/awk command do achieve this? Thanks! ---------- Post updated at 10:59 AM ---------- Previous update was at 10:54 AM ---------- Guess I asked this too soon. Found the... (0 Replies)
Discussion started by: lost.identity
0 Replies

5. Shell Programming and Scripting

How to split the comma separated file?

Hi, I have a filein unix like ABC,CDE BCD,KHL and the output i need is like column1 column2 ABC,CDE ABC ABC,CDE CDE BCD,KHL BCD BCD,KHL KHL. Can some body help me out? Hi, The code is working fine. But in my file each row does not have always 1 comma. It may... (6 Replies)
Discussion started by: jagdishrout
6 Replies

6. Shell Programming and Scripting

Need Help - comma inside double quote in comma separated csv,

Hello there, I have a comma separated csv , and all the text field is wrapped by double quote. Issue is some text field contain comma as well inside double quote. so it is difficult to process. Input in the csv file is , 1,234,"abc,12,gh","GH234TY",34 I need output like below,... (8 Replies)
Discussion started by: Uttam Maji
8 Replies

7. Shell Programming and Scripting

Comma separated file

Hi all, I have the following files types: FileA: 100, 23, 33, FileB: 22, 45, 78, and i want to make File C: 100,22 23,45 33,78 any nice suggestions for making it easy. (3 Replies)
Discussion started by: hen1610
3 Replies

8. Shell Programming and Scripting

parsing comma separated file

Hi, I have a file with th elist of patches separated by comma, like below: patch1, patch 2, patch 3................ t\The number of patches is not known as it changes every time. I need assistance in writing a routine such as it will take patch1 as first variable and performs the... (4 Replies)
Discussion started by: avikaljain
4 Replies

9. Shell Programming and Scripting

How to format file into comma separated field

Guys, Need you help, i have a a file content that look like this. Nokia 3330 <spaces><spaces><more spaces>+76451883874 Nokia 3610 +87467361615 so on and so forth, - there are so many spaces in between. - e.g.... (5 Replies)
Discussion started by: shtobias
5 Replies

10. UNIX for Dummies Questions & Answers

Convert a tab delimited/variable length file to fixed length file

Hi, all. I need to convert a file tab delimited/variable length file in AIX to a fixed lenght file delimited by spaces. This is the input file: 10200002<tab>US$ COM<tab>16/12/2008<tab>2,3775<tab>2,3783 19300978<tab>EURO<tab>16/12/2008<tab>3,28523<tab>3,28657 And this is the expected... (2 Replies)
Discussion started by: Everton_Silveir
2 Replies
Login or Register to Ask a Question