The files that I am processing are called sdf and they contain information about chemical structures with each structure contained in a record. The first section of the record holds the chemical structure and the second section holds (can hold) other associated information such as the compound name, identification numbers, measured data, etc, in the form of attribute tags where the tag is on one line and the value on the next. Unfortunately, the standard is rather loose and the same information can be located in more than one place and there is no requirement that every record have the same attribute tags or have them in the same order.

Software that reads this type of file is also all over the place as far as where any individual application will be looking for specific information or what limitations there will be. Since these applications cannot be modified (by me), it is often necessary to modify the input and, as expected, I tend to come to linux when that happens.

In this case, there is an issue with the chemical name. IUPAC, which creates the nomenclature for chemical names has not yet come around to understanding that chemical names should formatted such that they can have a linear notation in standard ACSII or similar. There are many chemical names that cannot be copied and pasted into a computer file name or flat text file. When you get such a name, you need to do something else for that compound. With the data I am working with, someone substituted a different value called the InChi (International Chemical Identifier). This is a computer compatible string but is unfortunately still not compatible with some applications (it's too long).

Years of working with such data has taught me to avoid names that begin with a number, have special characters, or are longer than 300 characters but not everyone has come to those same conclusions. I am working with files that are 50+ MB and have thousands of records. There are generally between 50 and 75 records that need to be changed. That's to many to do by hand.

The specific case I am looking for is when the InChi value was used for the chemical name. This is identified by the line following > <CompoundName> containing the string InChi=. Where this is not the case, nothing needs to be done to the record. Where that is the case, I need to create a substitute name from something reasonable. I am using the Identifier value, which is to be found on the line following > <Identifier>.

In short, if the line following > <CompoundName> contains InChi=, I save the value on the line following > <Identifier> and use it to create a new name. That name is written to both the first line of the record (one place where apps look for the name) and to the line following > <CompoundName>. The version on the first line is a bit different but that isn't very important.

My script works, but can take an hour to do a long file. I thought that I could speed things up by storing the output in an array and then dumping it at the end as I think this is more or less what apps like awk do. I couldn't get that working.

The number of records that need to be modified is relatively small but the files are big enough to be difficult to manage. The solution should write records that do not comply with the criteria in an unaltered fashion. I have tried to write a version that knows exactly which records need to be modified and so does not process the rest (just writes to output) but that version isn't working yet. It won't be much of an improvement if I can't store the output and have to write it to a file line by line.

LMHmedchem