Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Single command for add 2 columns and remove 2 columns in unix/performance tuning
Top Forums Shell Programming and Scripting Single command for add 2 columns and remove 2 columns in unix/performance tuning Post 302433988 by onesuri on Thursday 1st of July 2010 04:40:12 AM
Old 07-01-2010
Single command for add 2 columns and remove 2 columns in unix/performance tuning
Hi all,
I have created a script which adding two columns and removing two columns for all files.

Filename: Cust_information_1200_201010.txt

Source Data:

Code:
"1","Cust information","123","106001","street","1-203 high street"
"1","Cust information","124","105001","street","1-203 high street"

2) Extract SID and PID from a file

Filename:Cust_information_1200_201010.txt
Code:
SID=1200
PID=201010

Output required - ( first and second columns removed and last two columns added)

Code:
"123","106001","street","1-203 high street","1200","201010"
"124","105001","street","1-203 high street","1200","201010"

Currently I am using the below script to execute 40 GB file. It is taking lot of time and also creating temporary files(lst files).
it is occupying double the space.

I am unable to tune the script using with in a single command can i add two columns and remove two columns in unix.



Code:
for file4 in `ls $TempDir|egrep '(.*Detail.*txt)'`
do
        filename=`echo $file4 |cut -d "." -f1`
        Var3=`echo $file4 |cut -d "." -f1 |awk -F[_-] '{print $(NF)}'`;Var4=`echo $file4 |cut -d "." -f1 |awk -F[_-] '{print $(NF-1)}'`; sed -e "s/^.*$/&,$Var3,$Var4/g" $TempDir/$file4 >$TempDir/$filename".lst"
        awk -F, '{ OFS=","; $1=""; $2=""; print }' $TempDir/$filename".lst"|sed s/^,,//g >$TempDir/$filename".csv"
        rm -f $file4
done
rm -f $TempDir/*Detail_*.lst


Any help greatly appreciated
Last edited by Scott; 07-01-2010 at 05:52 AM.. Reason: Code tags again
 

10 More Discussions You Might Find Interesting

1. SCO

SCO unix server performance tuning

Hello, Can someone share his experience with SCO unix server performance tuning? - how to make some performance analysis - how to tune up a server for better performance Thank you! (0 Replies)
Discussion started by: forumuser7
0 Replies

2. AIX

UNIX/AIX Performance Tuning = some basics please

Hi all, From Googling, I found that the basics used for troubleshooting UNIX/AIX performance issues are commands like vmstat, iostat and sar. I believe these are generic commands regardless of what UNIX flavour is in used, only difference being is the format of the output. In a real case... (2 Replies)
Discussion started by: newbie_01
2 Replies

3. UNIX for Advanced & Expert Users

Unix Bash: substitute columns in .csv using other .csv columns

Hi All, I have two .csv's input.csv having values as (7 columns) ABC,A19907103,ABC DEV YUNG,2.17,1000,2157,07/07/2006 XYZ,H00213850,MM TRUP HILL,38.38,580,23308,31/08/2010 output.csv having (25 columns) A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y... (4 Replies)
Discussion started by: abhivyas
4 Replies

4. Shell Programming and Scripting

command to remove multiple commands in particular columns

Hi Experts, I actually need to remove multiple commas within the column not the entire row. Its comma delimited file Actually the value seems to look like 1,006,000, when we open this in notepad or word pad the value look s like “1,006,000” Actually our Sed command removes single comma and... (7 Replies)
Discussion started by: bshivali
7 Replies

5. Shell Programming and Scripting

How to remove spaces between the columns in UNIX script?.

Hi guru's, I am trying to write a script to generate a csv file by connecting to database run a query and put the values into csv file. But the problem i face is i am getting lot of space after one value.how can i remove those values?. Please help. #!/bin/bash export... (2 Replies)
Discussion started by: karingulanagara
2 Replies

6. Shell Programming and Scripting

Evaluate 2 columns, add sum IF two columns match on two rows

Hi all, I know this sounds suspiciously like a homework course; but, it is not. My goal is to take a file, and match my "ID" column to the "Date" column, if those conditions are true, add the total number of minutes worked and place it in this file, while not printing the original rows that I... (6 Replies)
Discussion started by: mtucker6784
6 Replies

7. Shell Programming and Scripting

Evaluate 2 columns, add sum IF two columns satisfy the condition

HI All, I'm embedding SQL query in Script which gives following output: Assignee Group Total ABC Group1 17 PQR Group2 5 PQR Group3 6 XYZ Group1 10 XYZ Group3 5 I have saved the above output in a file. How do i sum up the contents of this output so as to get following output: ... (4 Replies)
Discussion started by: Khushbu
4 Replies

8. Shell Programming and Scripting

Add new columns based on existing columns

Hi all, I am kind of stuck with printing my desired output. Please help me if you know how it can work. My input file(tab separated): NW_0068.1 41,16 100,900 NW_0699.1 4,2,19 200,700,80 My Output file (desired): NW_0068.1 41,16 100,900 100 - 141 NW_0068.1 41,16 100,900 ... (3 Replies)
Discussion started by: sam_2921
3 Replies

9. UNIX for Beginners Questions & Answers

Group by columns and add sum in new columns

Dear Experts, I have input file which is comma separated, has 4 columns like below, BRAND,COUNTRY,MODEL,COUNT NIKE,USA,DUMMY,5 NIKE,USA,ORIGINAL,10 PUMA,FRANCE,DUMMY,20 PUMA,FRANCE,ORIGINAL,15 ADIDAS,ITALY,DUMMY,50 ADIDAS,ITALY,ORIGINAL,50 SPIKE,CHINA,DUMMY,1O And expected output add... (2 Replies)
Discussion started by: ricky1991
2 Replies

10. UNIX for Beginners Questions & Answers

UNIX Command to Match columns from two csv files

I am joining two CSV files based on 'Server_Name' column, 1st column of first file and 2nd column of second file. If matches, output 1st and 2nd column from first file, 3rd,4th,5th,6th columns from second file. I am expecting output CSV file as below. Could you please send me help me with... (6 Replies)
Discussion started by: Anadmbt
6 Replies

LEARN ABOUT FREEBSD

tabs

TABS(1) 						    BSD General Commands Manual 						   TABS(1)

NAME

     tabs -- set terminal tabs

SYNOPSIS

     tabs [-n | -a | -a2 | -c | -c2 | -c3 | -f | -p | -s | -u] [+m[n]] [-T type]
     tabs [-T type] [+[n]] n1[,n2,...]

DESCRIPTION

     The tabs utility displays a series of characters that clear the hardware terminal tab settings then initialises tab stops at specified posi-
     tions, and optionally adjusts the margin.

     In the first synopsis form, the tab stops set depend on the command line options used, and may be one of the predefined formats or at regular
     intervals.

     In the second synopsis form, tab stops are set at positions n1, n2, etc.  If a position is preceded by a '+', it is relative to the previous
     position set.  No more than 20 positions may be specified.

     If no tab stops are specified, the ``standard'' UNIX tab width of 8 is used.

     The options are as follows:

     -n      Set a tab stop every n columns.  If n is 0, the tab stops are cleared but no new ones are set.

     -a      Assembler format (columns 1, 10, 16, 36, 72).

     -a2     Assembler format (columns 1, 10, 16, 40, 72).

     -c      COBOL normal format (columns 1, 8, 12, 16, 20, 55)

     -c2     COBOL compact format (columns 1, 6, 10, 14, 49)

     -c3     COBOL compact format (columns 1, 6, 10, 14, 18, 22, 26, 30, 34, 38, 42, 46, 50, 54, 58, 62, 67).

     -f      FORTRAN format (columns 1, 7, 11, 15, 19, 23).

     -p      PL/1 format (columns 1, 5, 9, 13, 17, 21, 25, 29, 33, 37, 41, 45, 49, 53, 57, 61).

     -s      SNOBOL format (columns 1, 10, 55).

     -u      Assembler format (columns 1, 12, 20, 44).

     +m[n], +[n]
	     Set an n character left margin, or 10 if n is omitted.

     -T type
	     Output escape sequence for the terminal type type.

ENVIRONMENT

     The LANG, LC_ALL, LC_CTYPE and TERM environment variables affect the execution of tabs as described in environ(7).

     The -T option overrides the setting of the TERM environment variable.  If neither TERM nor the -T option are present, tabs will fail.

EXIT STATUS

     The tabs utility exits 0 on success, and >0 if an error occurs.

SEE ALSO

     expand(1), stty(1), tput(1), unexpand(1), termcap(5)

STANDARDS

     The tabs utility conforms to IEEE Std 1003.1-2001 (``POSIX.1'').

HISTORY

     A tabs utility appeared in PWB UNIX.  This implementation was introduced in FreeBSD 5.0.

BUGS

     The current termcap(5) database does not define the 'ML' (set left soft margin) capability for any terminals.

BSD
								   May 20, 2002 							       BSD
All times are GMT -4. The time now is 07:33 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy