Adding lines to a large file Post: 302918060

Sponsored Content

Top Forums Shell Programming and Scripting Adding lines to a large file Post 302918060 by LMHmedchem on Saturday 20th of September 2014 01:30:16 PM

09-20-2014

Registered User

Adding lines to a large file

Hello,

I have a relatively large text file (25,000K) consisting of records of data. For each record, I need to create a new line based on what is already there.

Every record has a block that looks like,

Code:

M  END
>  <ID>
1

>  <SOURCE>
KEGG

>  <SOURCE_ID>
C00002

>  <NAME>
ATP; Adenosine 5'-triphosphate

>  <SMILES>
Nc1ncnc2n(cnc12)[C@@H]1O[C@H](COP(O)(=O)OP(O)(=O)OP(O)(O)=O)[C@@H](O)[C@H]1O

>  <MIMW>
506.995745159

>  <FORMULA>
C10H16N5O13P3

$$$$

The data tag lines > <ID>, etc, are the same for each record (or should be). The data on the line below the tag varies. I need to make a new field called

> <SOURCE_SOURCE_ID>

That is the data from > <SOURCE> concatenated with > <SOURCE_ID> separated with an underscore.

The record above would look like,

Code:

M  END
>  <ID>
1

>  <SOURCE>
KEGG

>  <SOURCE_ID>
C00002

>  <SOURCE_SOURCE_ID>
KEGG_C00002

>  <NAME>
ATP; Adenosine 5'-triphosphate

>  <SMILES>
Nc1ncnc2n(cnc12)[C@@H]1O[C@H](COP(O)(=O)OP(O)(=O)OP(O)(O)=O)[C@@H](O)[C@H]1O

>  <MIMW>
506.995745159

>  <FORMULA>
C10H16N5O13P3

$$$$

This is quite a bit beyond the things I normally do with shell scripts and I'm not sure where to start. I presume this would be some kind of while read line that looks for > <SOURCE> and captures the next line, looks for > <SOURCE_ID> and captures the next line, makes up the new variable, and makes an insert. All other lines would just be printed. This seems like manipulating an output stream, which I know how to do in cpp, but not in bash.

Suggestions would be greatly appreciated.

LMHmedchem

LMHmedchem

View Public Profile for LMHmedchem

Find all posts by LMHmedchem

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Help with selecting specific lines in a large file

Hello, I need to select the 3 lines above as well as below a search string, including the search string. I have been trying various combinations using sed command without any success. Can anuone help please. Thanking

2. Shell Programming and Scripting

Adding strings to lines in a file

Hi all, I have a positional text file that comes from some source application. Before it is processed by destination application I have to add some header (suffix) to every record(line) in the file. e.g. Actual File ............... AccountDetails AcNO Name Amount 1234 John 26578 5678...

3. Shell Programming and Scripting

Search for multiple lines in large file

Hi, I have a requirement to search for a string in a large log file along with few lines before and after the the string. The following script was sufficient to search such an entry. STRING_TO_GREP="$1" FILE_TO_GREP="$2" NUMBER_OF_LINES_BEFORE=$3 NUMBER_OF_LINES_AFTER=$4 for i in `grep...

4. Shell Programming and Scripting

Adding new lines to a file + adding suffix to a pattern

I need some help with adding lines to file and substitute a pattern. Ok I have a file: #cat names.txt name: John Doe stationed: 1 name: Michael Sweets stationed: 41 . . . And would like to change it to: name: John Doe employed permanently stationed: 1-office

5. Shell Programming and Scripting

Parse large file on line count (random lines)

I have a file that needs to be parsed into multiple files every time there line contains a number 1. the problem i face is the lines are random and the file size is random. an example is that on line 4, 65, 187, 202 & 209 are number 1's so there has to be file breaks between all those to create 4...

6. UNIX for Dummies Questions & Answers

Adding missing lines in file

Dear all, I have a file with two columns - the first column is increasing every 50, the second column is just count (e.g. 5). However, when count is zero, no line is present. Sample: 1950 7 2000 14 2050 7 2100 13 2150 10 2200 9 2250 7 2300 8 2350 7...

7. UNIX for Advanced & Expert Users

Count number of lines between a pattern in a large file

1000CUS E Y4NYRETAIL 10010004HELIOPOLIS 110000500022360591000056XX EG 1101DEBY XXAD ZSSKY TSSROS 1102HANYNNYY@HOTMAIL.COM 210030/05/201301/06/2013AED 3100 OPE 3100 CLO 3100 The 1000CUS E Y NYCORPORATE 10010004HELIOPOLIS 110000500025270504550203XX EG 1101XXXQ FOR CTING AND...

8. Shell Programming and Scripting

Adding lines at a particular location in a file.

Hi Experts, Let us take a text file,say items.txt having the following data jar bottle gum tube cereal bag I want to add the content of items.txt to another file say #many lines not necessary ingredients #many line not necesary ingredients I want to append the data in...

9. UNIX for Advanced & Expert Users

How to split a large file with the first 100 lines of each condition?

I have a huge file with the following input: Case1 Specific_Info Specific_Info Case1 Specific_Info Specific_Info Case3 Specific_Info Specific_Info Case4 Specific_Info Specific_Info Case1 Specific_Info Specific_Info Case2 Specific_Info Specific_Info Case2 Specific_Info Specific_Info...

10. Shell Programming and Scripting

Bash : Checking Large file for specific lines

Morning .. I have a file with approximately 1000 lines. I want to check that the file contains, for example, 100 lines. Something like whats given below is ugly. And even if I create a function I have to call it 100 times. I may need to look through multiple files at times. Is there a...

LEARN ABOUT OSX

cap_mkdb

CAP_MKDB(1)						    BSD General Commands Manual 					       CAP_MKDB(1)

NAME

     cap_mkdb -- create capability database

SYNOPSIS

     cap_mkdb [-v] [-f outfile] file1 [file2 ...]

DESCRIPTION

     Cap_mkdb builds a hashed database out of the getcap(3) logical database constructed by the concatenation of the specified files .

     The database is named by the basename of the first file argument and the string ``.db''.  The getcap(3) routines can access the database in
     this form much more quickly than they can the original text file(s).

     The ``tc'' capabilities of the records are expanded before the record is stored into the database.

     The options as as follows:

     -f outfile
	     Specify a different database basename.

     -v      Print out the number of capability records in the database.

FORMAT

     Each record is stored in the database using two different types of keys.

     The first type is a key which consists of the first capability of the record (not including the trailing colon (``:'')) with a data field
     consisting of a special byte followed by the rest of the record.  The special byte is either a 0 or 1, where a 0 means that the record is
     okay, and a 1 means that there was a ``tc'' capability in the record that couldn't be expanded.

     The second type is a key which consists of one of the names from the first capability of the record with a data field consisting a special
     byte followed by the the first capability of the record.  The special byte is a 2.

     In normal operation names are looked up in the database, resulting in a key/data pair of the second type.	The data field of this key/data
     pair is used to look up a key/data pair of the first type which has the real data associated with the name.

RETURN VALUE

     The cap_mkdb utility exits 0 on success and >0 if an error occurs.

SEE ALSO

     dbopen(3), getcap(3), termcap(5)

BSD
								   June 2, 2019 							       BSD

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Help with selecting specific lines in a large file

Discussion started by: tansha

2. Shell Programming and Scripting

Adding strings to lines in a file

Discussion started by: sharath160

3. Shell Programming and Scripting

Search for multiple lines in large file

Discussion started by: praveen123

4. Shell Programming and Scripting

Adding new lines to a file + adding suffix to a pattern

Discussion started by: hemo21