Sponsored Content
Top Forums Shell Programming and Scripting Deleting specific rows in large files having rows greater than 100000 Post 302379524 by manish2009 on Friday 11th of December 2009 02:42:57 AM
Old 12-11-2009
Deleting specific rows in large files having rows greater than 100000

Hi Guys,

I need help in modifying a large text file containing more than 1-2 lakh rows of data using unix commands. I am quite new to the unix language

the text file contains data in a pipe delimited format

Code:
sdfsdfs
sdfsdfsd
START_ROW
sdfsd|sdfsdfsd|sdfsdfasdf|sdfsadf|sdfasdf
sdfsd|sdfsdfsd|sdfsdfasdf||sdfasdf
sdfsd||sdfsdfasdf|sdfsadf|sdfasdf
END_ROW
sdfsd
sdfsfsdf

i want to remove the header and the footer, so the final file would look like below.

Code:
sdfsd|sdfsdfsd|sdfsdfasdf|sdfsadf|sdfasdf
sdfsd|sdfsdfsd|sdfsdfasdf||sdfasdf
sdfsd||sdfsdfasdf|sdfsadf|sdfasdf

I tried varous vb methods to do it .However when i use it for large files it hangs and closes.

Thanks very much.

Last edited by Yogesh Sawant; 12-11-2009 at 03:46 AM.. Reason: added code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Deleting specific files greater then 90 days

I have a directory that contains a number of history files for a process. Every file starts with the string "EVACK". FOr example EVACK0101.000001 EVACK0102.095940 EVACKmmdd.hhmiss I want to erase in the specific directory ONLY(no sub-directories) all EVACK files older then 90 days. I... (2 Replies)
Discussion started by: beilstwh
2 Replies

2. Shell Programming and Scripting

Deleting rows that begin with #

Hi, I have a file that has rows that start with # and ends with #. For example.. # hi text JK NM JK NM JK K JK NM # no # yes So I want to remove the #'s and put them into another file. so the output will be two files.. File 1: JK NM JK NM JK K JK NM (3 Replies)
Discussion started by: phil_heath
3 Replies

3. Shell Programming and Scripting

Deleting of Specific Rows.

Fruit : Price : Quantity apple : 20 : 40 chiku : 40 :30 Hey guys, i have written a code using sed to delete a specific char which is being typed in. But the problem i am having is , how can i expand my coding to actually allow it do delete the whole row. For example,... (21 Replies)
Discussion started by: gregarion
21 Replies

4. Shell Programming and Scripting

deleting rows that have certain characters

Hi, I want to delete rows whenever column one has the letters 'rpa'. The file is tab seperated. e.g. years 1 bears 1 cats 2 rpat 3 rpa99 4 rpa011 5 then removing 'rpa' containing rows based on the first column years 1 bears 1 cats 2 thanks (7 Replies)
Discussion started by: phil_heath
7 Replies

5. Shell Programming and Scripting

Large file - columns into rows etc

I have done a couple of searches on this and have found many threads but I don't think I've found one that is useful to me - probably because I have very basic comprehension of perl and beginners shell so trying to manipulate a script already posted maybe beyond my capabilities.... Anyway - I... (26 Replies)
Discussion started by: Myrona
26 Replies

6. UNIX for Dummies Questions & Answers

Help with deleting specific rows from a text file

I know this is a complicated question but I will try to illustrate it with some data. I have a data file that looks like the following: 1341 NA06985 0 0 2 46.6432798439 1341 NA06991 NA06993 NA06985 2 48.8478948517 1341 NA06993 0 0 1 45.8022601455 1340 NA06994 0 0 1 48.780669145 1340... (1 Reply)
Discussion started by: evelibertine
1 Replies

7. UNIX for Dummies Questions & Answers

Deleting specific rows from a text file

How do I go about deleting specific rows from a text file (given row number)? (5 Replies)
Discussion started by: evelibertine
5 Replies

8. Shell Programming and Scripting

deleting rows under a certain condition

there are 20 variables and I would like to delete the rows if 13th-20th columns are all NA. Thank you! FID IID aspirpre statihos fibrahos ocholhos arbhos betabhos alphbhos cacbhos diurehos numbcig.x toast1 toast2 toast3 toast4 ischoth1 ischoth2 ischoth3 ischoth4 101 101 1 1 1 1 1 2 1 2... (2 Replies)
Discussion started by: johnkim0806
2 Replies

9. UNIX for Dummies Questions & Answers

Deleting rows where the value in a specific column match

Hi, I have a tab delimited text file where I want to delete all rows that have the same string for column 1. How do I go about doing that? Thanks! Example Input: aa 1 aa 2 aa 3 bb 4 bc 5 bb 6 cd 8 Output: bc 5 cd 8 (4 Replies)
Discussion started by: evelibertine
4 Replies

10. Shell Programming and Scripting

Extracting data from specific rows and columns from multiple csv files

I have a series of csv files in the following format eg file1 Experiment Name,XYZ_07/28/15, Specimen Name,Specimen_001, Tube Name, Control, Record Date,7/28/2015 14:50, $OP,XYZYZ, GUID,abc, Population,#Events,%Parent All Events,10500, P1,10071,95.9 Early Apoptosis,1113,11.1 Late... (6 Replies)
Discussion started by: pawannoel
6 Replies
Locale::Codes::LangFam(3pm)				 Perl Programmers Reference Guide			       Locale::Codes::LangFam(3pm)

NAME
Locale::Codes::LangFam - standard codes for language extension identification SYNOPSIS
use Locale::Codes::LangFam; $lext = code2langfam('apa'); # $lext gets 'Apache languages' $code = langfam2code('Apache languages'); # $code gets 'apa' @codes = all_langfam_codes(); @names = all_langfam_names(); DESCRIPTION
The "Locale::Codes::LangFam" module provides access to standard codes used for identifying language families, such as those as defined in ISO 639-5. Most of the routines take an optional additional argument which specifies the code set to use. If not specified, the default ISO 639-5 language family codes will be used. SUPPORTED CODE SETS
There are several different code sets you can use for identifying language families. A code set may be specified using either a name, or a constant that is automatically exported by this module. For example, the two are equivalent: $lext = code2langfam('apa','alpha'); $lext = code2langfam('apa',LOCALE_LANGFAM_ALPHA); The codesets currently supported are: alpha This is the set of three-letter (lowercase) codes from ISO 639-5 such as 'apa' for Apache languages. This is the default code set. ROUTINES
code2langfam ( CODE [,CODESET] ) langfam2code ( NAME [,CODESET] ) langfam_code2code ( CODE ,CODESET ,CODESET2 ) all_langfam_codes ( [CODESET] ) all_langfam_names ( [CODESET] ) Locale::Codes::LangFam::rename_langfam ( CODE ,NEW_NAME [,CODESET] ) Locale::Codes::LangFam::add_langfam ( CODE ,NAME [,CODESET] ) Locale::Codes::LangFam::delete_langfam ( CODE [,CODESET] ) Locale::Codes::LangFam::add_langfam_alias ( NAME ,NEW_NAME ) Locale::Codes::LangFam::delete_langfam_alias ( NAME ) Locale::Codes::LangFam::rename_langfam_code ( CODE ,NEW_CODE [,CODESET] ) Locale::Codes::LangFam::add_langfam_code_alias ( CODE ,NEW_CODE [,CODESET] ) Locale::Codes::LangFam::delete_langfam_code_alias ( CODE [,CODESET] ) These routines are all documented in the Locale::Codes::API man page. SEE ALSO
Locale::Codes The Locale-Codes distribution. Locale::Codes::API The list of functions supported by this module. http://www.loc.gov/standards/iso639-5/id.php ISO 639-5 . AUTHOR
See Locale::Codes for full author history. Currently maintained by Sullivan Beck (sbeck@cpan.org). COPYRIGHT
Copyright (c) 2011-2012 Sullivan Beck This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.16.2 2012-10-11 Locale::Codes::LangFam(3pm)
All times are GMT -4. The time now is 03:06 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy