I have some files in a directory and a short list of strings. I want to loop through the files and remove lines containing the string and renumber.
There are some issues. The first is the strings that can contain troublesome characters like single quotes and parenthesis. Here is one list of strings,
It is very likely that the list will contain the same string more than once. I either need to clean that up or have the script allow for instances where the string is not found.
The other complexity is that the line numbering doesn't start until the 15th line of the file.
I was thinking of something like,
This code works for the line removal but is rather inefficient since it has to make separate calls to grep for each item in the remove list and do that for every file. This does not have to be particularly fast, but I would prefer if it was not quite so moronic.
As for the line renumbering starting with the 15th line, I have no idea.
Suggestions would be appreciated.
---------- Post updated at 07:17 PM ---------- Previous update was at 06:22 PM ----------
This is part of one of the files. You can see that the numbering starts on the line following f*order. If it helps, the numbers start on the first line that begins with a number. The f*order field can have value from f0-f9. The number of columns and rows in the files vary. This example shows the first 8 columns and 10 data rows.
The only thing I can think of at the moment would be to copy the first 14 lines to a temp file and then delete them. Then I would renumber the rest of the file and then cat the file back together.
Do the lines in the remove list exist (unquoted) in a file? (If so, a single grep -Fvf string_file file would seem better for this problem than one grep invocation for each fixed string. But awk is probably better yet since it can do both the line removal and the renumbering.) Do any of these strings ever contain any whitespace characters?
Does the renumbering apply only to the 1st field in the lines to be renumbered? Or, does the 2nd field also need to be modified? (If so, how?)
You said that the number of rows and columns vary from file to file. Does the field to be matched also vary, or is it always the 3rd field? If it isn't always the 3rd field, is it always in a field with the string Name as the header in line 1 in that file? (An awk script will run faster if we know which field to match.)
This User Gave Thanks to Don Cragun For This Post:
This is a script that currently works for this task. It is messy an un-elegant, but I am posting it since I sometimes think that working code is often a better explanation than a description given in prose, even where the code leaves allot to be desired.
To answer your questions, the strings do not exist in a file, but that would be easy enough to create if it would be helpful.
Only the first column of the data section needs to be renumbered, the numbers in column 2 can remain as is.
The remove string will always refer to the 3rd field, and this will always be the name column.
The headers are on line 1 and then are duplicated on line 14.
There could be cases in the future where there are more than 14 rows before the data begins. In all cases, you could look for the second instance of the header row (some row that matches row 1) to know that the data starts on the next row.
Well the script I posted works, but it takes more than 38 minutes to run on the directory tree I tested. The test directory has 2000 files spread over 40 directories with 2446 lines per file and a remove list of 28 strings.
The script posted by Don Cragun finished in,
real 1m50.953s
user 3m0.410s
sys 0m49.394s
I like the fact that this script looks for the second instance of the header row to know where to start the renumbering. There are versions of my data where that would be useful.
For the Perl script posted by Aia, these files are in a directory structure of 40 different directories. I don't know perl well enough to set up the looping to troll through all of that and test the script. It seems as if I would have to run my script and then call your script and pass $FILE_LIST along with the file with the patterns to remove. Is that right?
From the post by Scrutinizer, I also do quite see how to loop through my directory structure and create lists of files to pass to the code.
Once again, it is amazing how much better a well formed script will perform.
In vi I would like to remove a line containing a string. I thought after reading this I could do this.
https://www.unix.com/302297288-post3.html
:'3560,3572/gcc/d'
It keeps complaining vi mark not set. And sometimes it complains E488: Trailing characters.
I don't understand what mark... (5 Replies)
I would like to identify every line with a specific string, in this case: "Mamma".
I would like to remove that line, and also the line above it and below it. So the below
Where are all amazing Flats
Look At The Great Big White
Hey There Hot Mamma
You Are So hot Baby
I wish You were Mine... (5 Replies)
I am outputting a line like this
print $2 "/" $4The last character though is a ":" and I want to remove it. Is there any neat way to remove it? Or am I forced to do something like this:
print $2 "/" substr($4, 1, length($4) - 1)Thanks. (6 Replies)
Hi,
I have a file like :
I want to remove last string in last line (here total string is "05550"~). And last line end with ~ character.
Output should be :
Please help me
Thanks in advance (3 Replies)
I am trying to write a script that will allow me to recursively look at my directories, and output all filenames to a txt document. I am almost finished, however I am hitting one last snag. Here is my script so far:
find . | grep .jpg | awk -F"/" '{print $NF}' > output.txtThis will give me an... (7 Replies)
Hi Folks,
I am new to ksh, i have informatica parameter file that i need to update everyday with shell script. i need your help updating this file with new parameters.
sample data
$$TABLE1_DATE=04-27-2011
$$TABLE2_DATE=04-23-2011
$$TABLE3_DATE=03-19-2011
.......Highligned... (4 Replies)
I want to add a "-r <remote_host>" option to my ksh script, causing the script to run a script of the same name on the specified remote host. The remote invocation should itself include all the command-line options of the original invocation, less the -r option.
For example, this invocation:
... (7 Replies)
Hi
Can anyone tell me how can i remove new line character from a string.
My requirement is to read a line from a file and store it to a string.
read line
string1=$line
read line
string2=$line
echo $string1$string2
The result i am getting in different line. i want the output in the same... (1 Reply)
I am looking for a way to remove any line in a text file that contains the string "Mac address". I guess you would grep and sed, but I am not sure how to do this. Thanks for you help. (3 Replies)