Extracting a portion of data from a very large tab delimited text file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extracting a portion of data from a very large tab delimited text file
# 1  
Old 04-11-2010
Extracting a portion of data from a very large tab delimited text file

Hi All

I wanted to know how to effectively delete some columns in a large tab delimited file.

I have a file that contains 5 columns and almost 100,000 rows

Code:
3456 f g t t
3456 g h
456   f  h
4567 f g h z
345   f g
567   h j k l

This is a very large data file and tab delimited.

I need to extract the rows that have values in all the 5 columns. At present, there are several rows that contain only 3 values.

please let me know the best way to extract the rows with all 5 values

Thanks.
LA
# 2  
Old 04-11-2010
Quote:
Originally Posted by Lucky Ali
Hi All

I wanted to know how to effectively delete some columns in a large tab delimited file.

I have a file that contains 5 columns and almost 100,000 rows

Code:
3456 f g t t
3456 g h
456   f  h
4567 f g h z
345   f g
567   h j k l

This is a very large data file and tab delimited.

I need to extract the rows that have values in all the 5 columns. At present, there are several rows that contain only 3 values.

please let me know the best way to extract the rows with all 5 values

Thanks.
LA
Code:
awk 'NF==5' file > newfile

# 3  
Old 04-11-2010
If Franklin52's solution does not work, perhaps it is because there are always five fields (four tabs delimiting possibly empty fields). In that case, you could try:
Code:
awk -F'\t' '{for (i=1;i<=NF;i++) if (!length($i)) next; print}' file

Regards,
Alister

Last edited by alister; 04-11-2010 at 03:44 PM.. Reason: Fix erroneous suggestions
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Need to convert a pipe delimited text file to tab delimited

Hi, I have a rquirement in unix as below . I have a text file with me seperated by | symbol and i need to generate a excel file through unix commands/script so that each value will go to each column. ex: Input Text file: 1|A|apple 2|B|bottle excel file to be generated as output as... (9 Replies)
Discussion started by: raja kakitapall
9 Replies

2. Shell Programming and Scripting

How to read data from tab delimited file after a specific position?

Hi Experts, I have a tab deliminated file as below myfile.txt Local Group Memberships *Administrators *Guests I need data in below format starting from 4th position. myfile1.txt Administrators Guests the above one is just an example and there could... (15 Replies)
Discussion started by: Litu1988
15 Replies

3. UNIX for Dummies Questions & Answers

How to convert a text file into tab delimited format?

I have a text file that made using text editor in Ubuntu. However the text file is not being recognized as space or tab delimited, the formatting seems to be messed up. How can I convert the text file into tab delimited format? (3 Replies)
Discussion started by: evelibertine
3 Replies

4. UNIX for Dummies Questions & Answers

Add a new column to a tab delimited text file

I want to add a new column to a tab delimited text file. It will be the first column and it will just be 1's. How do I go about doing that? Thanks! (1 Reply)
Discussion started by: evelibertine
1 Replies

5. UNIX for Dummies Questions & Answers

Deleting columns from a tab delimited text file?

I have a tab limited text file with 10000+ columns. I want to delete columns 6 through 23, how do I go about doing that? Thanks! (1 Reply)
Discussion started by: evelibertine
1 Replies

6. UNIX for Dummies Questions & Answers

How to convert text to columns in tab delimited text file

Hello Gurus, I have a text file containing nearly 12,000 tab delimited characters with 4000 rows. If the file size is small, excel can convert the text into coloumns. However, the file that I have is very big. Can some body help me in solving this problem? The input file example, ... (6 Replies)
Discussion started by: Unilearn
6 Replies

7. Shell Programming and Scripting

Extracting a portion of a data file with identifier

Hi, I do have a TAB delimted text file with the following format. 1 (- identifier of each group. this text is not present in the file only number) 1 3 4 65 56 WERTF 2 3 4 56 56 GHTYHU 3 3 5 64 23 VMFKLG 2 1 3 4 65 56 DGTEYDH 2 3 4 56 56 FJJJCKC 3 3 5 64 23 FNNNCHD 3 1 3 4 65 56 JDHJDH... (9 Replies)
Discussion started by: Lucky Ali
9 Replies

8. Shell Programming and Scripting

Delete first column in tab-delimited text-file

I have a large text-file with tab-delimited genetic data that looks like: KSC112 KSC234 0 0 1 1 A G C T I simply wan to delete the first column, but since the file has 600 000 columns, it is not possible with awk (seems to be limited at 32k columns). Does anyone have an idea how to do this? (2 Replies)
Discussion started by: andmal
2 Replies

9. Shell Programming and Scripting

Removing blanks in a text tab delimited file

Hi Experts I am very new to perl and need to make a script using perl. I would like to remove blanks in a text tab delimited file in in a specfic column range ( colum 21 to column 43) sample input and output shown below : Input: 117 102 650 652 654 656 117 93 95... (3 Replies)
Discussion started by: Faisal Riaz
3 Replies

10. Shell Programming and Scripting

Check whether a given file is in ASCII format and data is tab-delimited

Hi All, Please help me out with a script which checks whether a given file say abc.txt is in ASCII format and data is tab-delimited. If the condition doesn't satisfy then it should generate error code "100" for file not in ASCII format and "105" if it is not in tab-delimited format. If the... (9 Replies)
Discussion started by: Mandab
9 Replies
Login or Register to Ask a Question