Format the file by deleting empty field


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Format the file by deleting empty field
# 1  
Old 10-08-2010
Format the file by deleting empty field

I have the test data with 10 column separated by comma and each column has more than 1000000 rows. Can anyone help me to find empty field in all columns and delete that empty field alone and lift that column up by one row.

Data with empty field:
Code:
A74203XYZ,A21718XYZ,A72011XYZ,A41095XYZ,
A11215WXZ,A41001XYZ,A11050XYZ,         ,
        ,A48515XYZ,A72200RWS,A56009WXZ,
A16406RWS,A24515WXZ,A13955RWS,A13102WXZ,
A27440WXZ,A71176WXZ,A16518RWS,A74607WXZ,
L18692WXZ,A71439RWS,         ,A21795WXZ,
A41420XYZ,A11220WXZ,A13049WXZ,A44577XYZ,
A52734WXZ,A24461RWS,A11458WXZ,A21211XYZ,

Updated data by deleting empty field:-
Code:
A74203XYZ,A21718XYZ,A72011XYZ,A41095XYZ,
A11215WXZ,A41001XYZ,A11050XYZ,A56009WXZ,
A16406RWS,A48515XYZ,A72200RWS,A13102WXZ,
A27440WXZ,A24515WXZ,A13955RWS,A74607WXZ,
L18692WXZ,A71176WXZ,A16518RWS,A21795WXZ,
A41420XYZ,A71439RWS,A13049WXZ,A44577XYZ,
A52734WXZ,A11220WXZ,A11458WXZ,A21211XYZ,

# 2  
Old 10-08-2010
Bug

Hi Zooby

Try this code I think this will work :

Code:
#!/bin/ksh
cut -d ',' -f 1 YourFile.txt|tr -s '\n' '' > finalData.tmp
for var in  2 3 4 5 6 7 8 9 10
do
cut -d ',' -f ${var} YourFile.txt|tr -s '\n' '' > data.tmp
paste -d ',' finalData.tmp data.tmp > tmp1.tmp
mv tmp1.tmp finalData.tmp
done
rm data.tmp

# 3  
Old 10-10-2010
Try this:
Code:
awk -F, '{gsub(" ","")}
$1{print $1  > "temp1"}
$2{print $2  > "temp2"}
$3{print $3  > "temp3"}
$4{print $4 FS > "temp4"}
' yourfile

paste temp1 temp2 temp3 temp4 | tr '\t' ',' > newfile
rm temp1 temp2 temp3 temp4

# 4  
Old 10-10-2010
Using Perl:-

Code:
perl -F, -wlane '
push @col1,$F[0] if ($F[0] =~ /\S+/ ) ;
push @col2,$F[1] if ($F[1] =~ /\S+/ ) ;
push @col3,$F[2] if ($F[2] =~ /\S+/ ) ;
push @col4,$F[3] if ($F[3] =~ /\S+/ ) ;
END{
for ($i=0;$i<=$#col1;$i++) {$,="," ; print $col1[$i],$col2[$i],$col3[$i],$col4[$i] ; }
}
' infile.txt

# 5  
Old 10-10-2010
If you have enough memory:
Code:
awk '{ for(i=1;i<=NF;i++)
         if($i!~/ /)
         {
           N[i]++
           A[N[i],i]=$i
         }
     }
     END {
       for (i=1;i<=N[1];i++)
       {
         for (j=1;j<NF;j++)
           printf A[i,j]FS
         print A[i,NF]
       }
     }' FS=, infile

# 6  
Old 10-10-2010
Quote:
Originally Posted by Scrutinizer
If you have enough memory:
Code:
awk '{ for(i=1;i<=NF;i++)
         if($i!~/ /)
         {
           N[i]++
           A[N[i],i]=$i
         }
     }
     END {
       for (i=1;i<=N[1];i++)
       {
         for (j=1;j<NF;j++)
           printf A[i,j]FS
         print A[i,NF]
       }
     }' FS=, infile

Scrutinizer it is not working for me , it gives empty spaces.


===============
sorry man; it worked after I used /usr/xpg4/bin/awk only and it failed with nawk!!!!
BR
# 7  
Old 10-10-2010
This variation should use less memory. It prints rows once they are full and it discards array elements that are no longer required:
Code:
awk '{ p=1
       for(i=1;i<=NF;i++)
       {
         if($i!~/ /)
         {
           N[i]++
           A[N[i],i]=$i
         }
         if(N[i]<=n)
           p=0
         fi
       }
       if(p)n++
       for (i=m+1;i<=n;i++)
         for (j=1;j<=NF;j++)
         {
           s=(j==NF)?RS:FS
           printf A[i,j]s
           delete A[i,j]
         }
       m=n
     }' FS=, infile


Last edited by Scrutinizer; 10-10-2010 at 11:45 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Command to extract empty field in a large UNIX file?

Hi All, I have records in unix file like below. In this file, we have empty fields from 4th Column to 22nd Column. I have some 200000 records in a file. I want to extract records only which have empty fields from 4th field to 22nd filed. This file is comma separated file. what is the unix... (2 Replies)
Discussion started by: rakeshp
2 Replies

2. Shell Programming and Scripting

How to remove empty field in a text file?

Hi all, I want to remove empty field in a text file. I tried to used sed. But it failed. Input: LG10_PM_map_19_LEnd 1000560 G AG AG LG10_PM_map_19_LEnd 1005621 G AG LG10_PM_map_19_LEnd 1011214 A AG AG LG10_PM_map_19_LEnd 1011673 T CT CT ... (3 Replies)
Discussion started by: huiyee1
3 Replies

3. Linux

How do I format a Date field of a .CSV file with multiple commas in a string field?

I have a .CSV file (file.csv) whose data are all enclosed in double quotes. Sample format of the file is as below: column1,column2,column3,column4,column5,column6, column7, Column8, Column9, Column10 "12","B000QRIGJ4","4432","string with quotes, and with a comma, and colon: in... (3 Replies)
Discussion started by: dhruuv369
3 Replies

4. Shell Programming and Scripting

Deleting empty folders

Hey, I need help with writing a shell script that deletes empty folders..anyone? :) Thank you! (5 Replies)
Discussion started by: putukas
5 Replies

5. Shell Programming and Scripting

How to format file into comma separated field

Guys, Need you help, i have a a file content that look like this. Nokia 3330 <spaces><spaces><more spaces>+76451883874 Nokia 3610 +87467361615 so on and so forth, - there are so many spaces in between. - e.g.... (5 Replies)
Discussion started by: shtobias
5 Replies

6. Shell Programming and Scripting

how I can add a constant to a field without changing the file format

Hi, I need to edit a file Protein Data Bank (pdb) and then open that file with the program VMD but when I edit the file with awk, it changes pdb format and the VMD program can not read it. I need to subtract 34 to field 6 ($ 6). this is a pdb file : ATOM 918 N GLY B 103 -11.855 8.675... (8 Replies)
Discussion started by: bio_
8 Replies

7. Shell Programming and Scripting

awk - if field is empty, move line to new file

I have a script with this statement: /usr/xpg4/bin/awk -F"" 'NR==FNR{s=$2;next}{printf "%s\"%s\"\n", $0, s}' LOOKUP.TXT finallistnew.txt >test.txt I want to include logic or an additional step that says if there is no data in field 3, move the whole line out of test.txt into an additional... (9 Replies)
Discussion started by: scriptr2be
9 Replies

8. Shell Programming and Scripting

Deleting the empty file

Hi, I have a TEST.dat file. As a clean up process, I have to delete the TEST.dat file if it is empty. Basically, I don't want to delete TEST.dat if it contains anything in it but want to delete TEST.dat if it contains any spaces or nothing in it. Is there a command to check if the file... (2 Replies)
Discussion started by: rkumar28
2 Replies

9. Shell Programming and Scripting

deleting the empty files

as of our requiremnt some x no of files will be created from a third party tool ,out of them one or two files will be empty i.e size is 0. so i want to remove those files which are empty. naming of the files which are created will be like this abc_.txt 0 size abc_1.txt 4000 size abc_2.txt... (1 Reply)
Discussion started by: srivsn
1 Replies

10. Shell Programming and Scripting

change the empty function from the old format to the new format

I have about 300 files which has the function getDBBackend(). How to write a program to change the empty function from the old format to the new format? Old empty function format are either: function getDBBackend() { // Not available } // getDBBackend or: function... (0 Replies)
Discussion started by: powah
0 Replies
Login or Register to Ask a Question