Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Single command for add 2 columns and remove 2 columns in unix/performance tuning

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Single command for add 2 columns and remove 2 columns in unix/performance tuning
# 1  
Old 07-01-2010
Single command for add 2 columns and remove 2 columns in unix/performance tuning
Hi all,
I have created a script which adding two columns and removing two columns for all files.

Filename: Cust_information_1200_201010.txt

Source Data:

Code:
"1","Cust information","123","106001","street","1-203 high street"
"1","Cust information","124","105001","street","1-203 high street"

2) Extract SID and PID from a file

Filename:Cust_information_1200_201010.txt
Code:
SID=1200
PID=201010

Output required - ( first and second columns removed and last two columns added)

Code:
"123","106001","street","1-203 high street","1200","201010"
"124","105001","street","1-203 high street","1200","201010"

Currently I am using the below script to execute 40 GB file. It is taking lot of time and also creating temporary files(lst files).
it is occupying double the space.

I am unable to tune the script using with in a single command can i add two columns and remove two columns in unix.



Code:
for file4 in `ls $TempDir|egrep '(.*Detail.*txt)'`
do
        filename=`echo $file4 |cut -d "." -f1`
        Var3=`echo $file4 |cut -d "." -f1 |awk -F[_-] '{print $(NF)}'`;Var4=`echo $file4 |cut -d "." -f1 |awk -F[_-] '{print $(NF-1)}'`; sed -e "s/^.*$/&,$Var3,$Var4/g" $TempDir/$file4 >$TempDir/$filename".lst"
        awk -F, '{ OFS=","; $1=""; $2=""; print }' $TempDir/$filename".lst"|sed s/^,,//g >$TempDir/$filename".csv"
        rm -f $file4
done
rm -f $TempDir/*Detail_*.lst


Any help greatly appreciated
Last edited by Scott; 07-01-2010 at 05:52 AM.. Reason: Code tags again
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

UNIX Command to Match columns from two csv files

I am joining two CSV files based on 'Server_Name' column, 1st column of first file and 2nd column of second file. If matches, output 1st and 2nd column from first file, 3rd,4th,5th,6th columns from second file. I am expecting output CSV file as below. Could you please send me help me with... (6 Replies)
Discussion started by: Anadmbt
6 Replies

2. UNIX for Beginners Questions & Answers

Group by columns and add sum in new columns

Dear Experts, I have input file which is comma separated, has 4 columns like below, BRAND,COUNTRY,MODEL,COUNT NIKE,USA,DUMMY,5 NIKE,USA,ORIGINAL,10 PUMA,FRANCE,DUMMY,20 PUMA,FRANCE,ORIGINAL,15 ADIDAS,ITALY,DUMMY,50 ADIDAS,ITALY,ORIGINAL,50 SPIKE,CHINA,DUMMY,1O And expected output add... (2 Replies)
Discussion started by: ricky1991
2 Replies

3. Shell Programming and Scripting

Add new columns based on existing columns

Hi all, I am kind of stuck with printing my desired output. Please help me if you know how it can work. My input file(tab separated): NW_0068.1 41,16 100,900 NW_0699.1 4,2,19 200,700,80 My Output file (desired): NW_0068.1 41,16 100,900 100 - 141 NW_0068.1 41,16 100,900 ... (3 Replies)
Discussion started by: sam_2921
3 Replies

4. Shell Programming and Scripting

Evaluate 2 columns, add sum IF two columns satisfy the condition

HI All, I'm embedding SQL query in Script which gives following output: Assignee Group Total ABC Group1 17 PQR Group2 5 PQR Group3 6 XYZ Group1 10 XYZ Group3 5 I have saved the above output in a file. How do i sum up the contents of this output so as to get following output: ... (4 Replies)
Discussion started by: Khushbu
4 Replies

5. Shell Programming and Scripting

Evaluate 2 columns, add sum IF two columns match on two rows

Hi all, I know this sounds suspiciously like a homework course; but, it is not. My goal is to take a file, and match my "ID" column to the "Date" column, if those conditions are true, add the total number of minutes worked and place it in this file, while not printing the original rows that I... (6 Replies)
Discussion started by: mtucker6784
6 Replies

6. Shell Programming and Scripting

How to remove spaces between the columns in UNIX script?.

Hi guru's, I am trying to write a script to generate a csv file by connecting to database run a query and put the values into csv file. But the problem i face is i am getting lot of space after one value.how can i remove those values?. Please help. #!/bin/bash export... (2 Replies)
Discussion started by: karingulanagara
2 Replies

7. Shell Programming and Scripting

command to remove multiple commands in particular columns

Hi Experts, I actually need to remove multiple commas within the column not the entire row. Its comma delimited file Actually the value seems to look like 1,006,000, when we open this in notepad or word pad the value look s like “1,006,000” Actually our Sed command removes single comma and... (7 Replies)
Discussion started by: bshivali
7 Replies

8. UNIX for Advanced & Expert Users

Unix Bash: substitute columns in .csv using other .csv columns

Hi All, I have two .csv's input.csv having values as (7 columns) ABC,A19907103,ABC DEV YUNG,2.17,1000,2157,07/07/2006 XYZ,H00213850,MM TRUP HILL,38.38,580,23308,31/08/2010 output.csv having (25 columns) A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y... (4 Replies)
Discussion started by: abhivyas
4 Replies

9. AIX

UNIX/AIX Performance Tuning = some basics please

Hi all, From Googling, I found that the basics used for troubleshooting UNIX/AIX performance issues are commands like vmstat, iostat and sar. I believe these are generic commands regardless of what UNIX flavour is in used, only difference being is the format of the output. In a real case... (2 Replies)
Discussion started by: newbie_01
2 Replies

10. SCO

SCO unix server performance tuning

Hello, Can someone share his experience with SCO unix server performance tuning? - how to make some performance analysis - how to tune up a server for better performance Thank you! (0 Replies)
Discussion started by: forumuser7
0 Replies
Login or Register to Ask a Question