Deleting specific rows in large files having rows greater than 100000


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Deleting specific rows in large files having rows greater than 100000
# 1  
Old 12-11-2009
Deleting specific rows in large files having rows greater than 100000

Hi Guys,

I need help in modifying a large text file containing more than 1-2 lakh rows of data using unix commands. I am quite new to the unix language

the text file contains data in a pipe delimited format

Code:
sdfsdfs
sdfsdfsd
START_ROW
sdfsd|sdfsdfsd|sdfsdfasdf|sdfsadf|sdfasdf
sdfsd|sdfsdfsd|sdfsdfasdf||sdfasdf
sdfsd||sdfsdfasdf|sdfsadf|sdfasdf
END_ROW
sdfsd
sdfsfsdf

i want to remove the header and the footer, so the final file would look like below.

Code:
sdfsd|sdfsdfsd|sdfsdfasdf|sdfsadf|sdfasdf
sdfsd|sdfsdfsd|sdfsdfasdf||sdfasdf
sdfsd||sdfsdfasdf|sdfsadf|sdfasdf

I tried varous vb methods to do it .However when i use it for large files it hangs and closes.

Thanks very much.

Last edited by Yogesh Sawant; 12-11-2009 at 03:46 AM.. Reason: added code tags
# 2  
Old 12-11-2009
Try this with grep:

Code:
grep \| < infile >outfile

# 3  
Old 12-11-2009
if your data file contains only one START END block:
Code:
sed -n -e '/START_ROW/,/END_ROW/ {p} ; /END_ROW/ q' file.txt > newfile.txt

# 4  
Old 12-11-2009
can i use the grep command to print lines which are greater than a specific length..... let say lines having length > 25
# 5  
Old 12-11-2009
Quote:
Originally Posted by manish2009
can i use the grep command to print lines which are greater than a specific length..... let say lines having length > 25
Yes. But in this case, perfromance will not be good, as the output of grep shoule be piped to another command which will select records >25 chars.

In this context, sed would be faster.

Code:
sed -n '/|/ { /.\{25\}/p }' < infile > outfile

# 6  
Old 12-11-2009
Code:
sed '1,/START_ROW/d;/END_ROW/,$d' infile

# 7  
Old 12-11-2009
below i m getting sed:garbage after command and a blank file generates

sed -n -e '/START_ROW/,/END_ROW/ {p} ; /END_ROW/ q' file.txt > newfile.txt

thanks
Manish
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extracting data from specific rows and columns from multiple csv files

I have a series of csv files in the following format eg file1 Experiment Name,XYZ_07/28/15, Specimen Name,Specimen_001, Tube Name, Control, Record Date,7/28/2015 14:50, $OP,XYZYZ, GUID,abc, Population,#Events,%Parent All Events,10500, P1,10071,95.9 Early Apoptosis,1113,11.1 Late... (6 Replies)
Discussion started by: pawannoel
6 Replies

2. UNIX for Dummies Questions & Answers

Deleting rows where the value in a specific column match

Hi, I have a tab delimited text file where I want to delete all rows that have the same string for column 1. How do I go about doing that? Thanks! Example Input: aa 1 aa 2 aa 3 bb 4 bc 5 bb 6 cd 8 Output: bc 5 cd 8 (4 Replies)
Discussion started by: evelibertine
4 Replies

3. Shell Programming and Scripting

deleting rows under a certain condition

there are 20 variables and I would like to delete the rows if 13th-20th columns are all NA. Thank you! FID IID aspirpre statihos fibrahos ocholhos arbhos betabhos alphbhos cacbhos diurehos numbcig.x toast1 toast2 toast3 toast4 ischoth1 ischoth2 ischoth3 ischoth4 101 101 1 1 1 1 1 2 1 2... (2 Replies)
Discussion started by: johnkim0806
2 Replies

4. UNIX for Dummies Questions & Answers

Deleting specific rows from a text file

How do I go about deleting specific rows from a text file (given row number)? (5 Replies)
Discussion started by: evelibertine
5 Replies

5. UNIX for Dummies Questions & Answers

Help with deleting specific rows from a text file

I know this is a complicated question but I will try to illustrate it with some data. I have a data file that looks like the following: 1341 NA06985 0 0 2 46.6432798439 1341 NA06991 NA06993 NA06985 2 48.8478948517 1341 NA06993 0 0 1 45.8022601455 1340 NA06994 0 0 1 48.780669145 1340... (1 Reply)
Discussion started by: evelibertine
1 Replies

6. Shell Programming and Scripting

Large file - columns into rows etc

I have done a couple of searches on this and have found many threads but I don't think I've found one that is useful to me - probably because I have very basic comprehension of perl and beginners shell so trying to manipulate a script already posted maybe beyond my capabilities.... Anyway - I... (26 Replies)
Discussion started by: Myrona
26 Replies

7. Shell Programming and Scripting

deleting rows that have certain characters

Hi, I want to delete rows whenever column one has the letters 'rpa'. The file is tab seperated. e.g. years 1 bears 1 cats 2 rpat 3 rpa99 4 rpa011 5 then removing 'rpa' containing rows based on the first column years 1 bears 1 cats 2 thanks (7 Replies)
Discussion started by: phil_heath
7 Replies

8. Shell Programming and Scripting

Deleting of Specific Rows.

Fruit : Price : Quantity apple : 20 : 40 chiku : 40 :30 Hey guys, i have written a code using sed to delete a specific char which is being typed in. But the problem i am having is , how can i expand my coding to actually allow it do delete the whole row. For example,... (21 Replies)
Discussion started by: gregarion
21 Replies

9. Shell Programming and Scripting

Deleting rows that begin with #

Hi, I have a file that has rows that start with # and ends with #. For example.. # hi text JK NM JK NM JK K JK NM # no # yes So I want to remove the #'s and put them into another file. so the output will be two files.. File 1: JK NM JK NM JK K JK NM (3 Replies)
Discussion started by: phil_heath
3 Replies

10. Shell Programming and Scripting

Deleting specific files greater then 90 days

I have a directory that contains a number of history files for a process. Every file starts with the string "EVACK". FOr example EVACK0101.000001 EVACK0102.095940 EVACKmmdd.hhmiss I want to erase in the specific directory ONLY(no sub-directories) all EVACK files older then 90 days. I... (2 Replies)
Discussion started by: beilstwh
2 Replies
Login or Register to Ask a Question