Use foreach with sed in a bash script

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Use foreach with sed in a bash script
# 1  
Old 02-15-2018
Use foreach with sed in a bash script

I want to extract data from a ASCII file that looks like the one provided here (see input.txt). For this purpose I used sed commands. I want to chain the sed commands into a script that I can call with custom variables, instead of having to run it multiple times (Need to run the code for 30*24 = 720 times).

Here is the first few lines of the - the block starting 1NAME repeats for a given number of times:
Code:
AVERAGE   MODELNAME -- RUNNAME
 0  1  11121    0. 11122   24.
       -9700000         4000000   0   -241200000000   -1620000
1.00000      1000.00000  10 10   1   2   0    15.    11.     0.
    1    1  500  400
NAME
          11121      0.00     11121      1.00
   1NAME
 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
NAME
          11121      1.00     11121      2.00
   1NAME
 1.0000000E+00 45.0000000E+00 01.0000000E+00 115.0000000E+00 5.0000000E+00
 2.0000000E+00 66.0000000E+00 09.0000000E+00 180.0000000E+00 4.0000000E+00
 3.0000000E+00 80.0000000E+00 70.0000000E+00 130.0000000E+00 5.0000000E+00

What I want to do is grab a value from a given location (after the occurrence 1NAME) from this file. As an example, say I want to grab the value 66.0000000E+00 - that occurs at position 7 (5 from row 1 + 2 from row 2).
  1. Here is what I did. Listed here are some challenges and what i did to fix it (see lines starting with >>): 1. The format is rather strange and has multiple whitespaces. So I used sed to remove multiple spaces and replace with one space. And then using the piped output from this step, removed all blank lines. This resulted in all the data in the file arranged as one value per row.

>>
Code:
sed 's/\s\+/\n/g' <input.txt>| sed '/^$/d

  1. I then used sed command to extract data from a given row (say row 11 and row 50).

Code:
sed -n -e 11p -e 50p

  1. I tried to put all these commands as a bash script with custom row number. Since I have multiple data to extract, I used foreach fuction. The script I wrote is unfortunately giving me errors. I also want outputs from the last sed command to be piped to a single output file:

Code:
#!/bin/bash #Set input file name set FILE=$cwd/sample_or_2day foreach GRID (23729) foreach GRIDTIME(28 4100) sed 's/\s+/\n/g' $FILE | sed '/^$/d' | sed '1,36d' > temp_out rm -f $cwd/out_$GRID sed -n -e "$GRIDTIME" temp_out | tee $cwd/out_$GRID

In this code, the last sed command is only throwing out one value, even though my foreach has two inputs. I thought piping the output using tee would let the file add output from sed instead of overwrite the file itself.
  1. Also, I want to modify this code so that the foreach array is self generated using an equation. How can I use the if loop for this purpose (or another command) and how can I pass the results from the if loop into foreach? As an example: foreach (1 100 199 298) could be set as: for n=99, i = 1 + 99 (looped from 1 - 298).

Thanks for your patience. I am a nervous programmer and trying to master basics. I spent time looking at sed insturction pages, and user support forums. Any recommendations are welcome - especially with explicit instructions. Thanks!
# 2  
Old 02-15-2018
Welcome to the forum.

First comment on your title: there's no foreach in bash - it's from csh. bash provides similar but not identical commands.

Now to your specification. Please form sentences in plain English, carefully, detailedly, and completely. And, make sure the written request and sample code are consistent (you want "to grab the value 66.0000000E+00", but your sed scripts deliver -9700000 and 0.0000000E+00). Describe the problem and desired result. Add your (failed?) attempts: the steps you have taken and the tools you have used.

Let me try to paraphrase what I more guessed than understood from what you said above:
We have an input file with a five line general header, and followed by multiple records, each with a two line sub header, the keyword 1NAME in an extra line, and three data lines of five floating point numbers, i.e. 15 numbers.
From every record, we need to extract one certain number pointed to by a variable or an algorithm (to be defined).


Now there's some questions to be asked:
- Are any (sub) header data relevant and need to be taken into accont or passed back?
- how will the pointer be passed to this script / tool / program?
- are data from every single record to be extracted?
- how would the named algorithm be defined?
- what should be passed back to the caller?

Last edited by RudiC; 02-15-2018 at 05:33 PM.. Reason: improve phrasing
This User Gave Thanks to RudiC For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need bash script to use a sed command as a variable

I need to be able to use a sed command as a variable in a bash script. I have the sed command that almost works the way I want it. the command is sed -n '/inet/,/}/p' config.boot This gets me this result: inet 192.168.1.245 } I need to get the IP address into a variable so I... (9 Replies)
Discussion started by: edlentz
9 Replies

2. Shell Programming and Scripting

Replacing filename with sed in bash script

I need to cat two files with similar names. I am using the following script: #!/bin/bash if ] then file=$1 file2="${file%R1.fastq}R2.fastq" echo fetching data from R2 file ... sleep 3 cat $file $file2 > infile else echo "Input_file passed... (2 Replies)
Discussion started by: Xterra
2 Replies

3. Shell Programming and Scripting

sed problems - Bash Script

Hi I keep getting the following error sed: -e expression #1, char 32: unterminated `s' command sed: -e expression #1, char 35: unterminated `s' command sed: -e expression #1, char 35: unterminated `s' command whenever I use the following bash script #! /bin/bash... (2 Replies)
Discussion started by: spbr
2 Replies

4. Shell Programming and Scripting

Text manipulation with sed/awk in a bash script

Guys, I have a variable in a script that I want to transform to into something else Im hoping you guys can help. It doesn't have to use sed/awk but I figured these would be the simplest. DATE=20160120 I'd like to transform $DATE into "01-20-16" and move it into a new variable called... (8 Replies)
Discussion started by: dendenyc
8 Replies

5. Shell Programming and Scripting

sed not working in a bash script

Hi friends, I have two files - input and commands I want to read the input and replace a value in it with the contents in commands. My script is like this. Instead of printing the value in the commands file, it is simply printing $cmd in the output file. Any pointers are highly... (1 Reply)
Discussion started by: jacobs.smith
1 Replies

6. UNIX for Dummies Questions & Answers

Shell script foreach help

I am writing a shell script to uncompress files in a directory, then call a Perl script to search the files for given terms, store those terms in a different output file , and compress the output. I get a syntax error with my use of foreach. Below is my script. #!/bin/csh -fxv if (!... (2 Replies)
Discussion started by: dgrayman
2 Replies

7. Shell Programming and Scripting

foreach in bash (simple)

#!/bin/bash foreach valuex ( word1 word2 word3 word4 ) echo $valuex done the output should be: word1 word2 word3 word4 HOW DO I MAKE THIS WORK? (1 Reply)
Discussion started by: ajp7701
1 Replies

8. Shell Programming and Scripting

XML- Sed || Awk Bash script... Help!

Hi ! I'm working into my first bash script to make some xml modification and it's going to make me crazy lol .. so I decide to try into this forum to take some ideas from people that really know about this! This is my situation I've and xml file with a lots of positional values with another tags... (9 Replies)
Discussion started by: juampal
9 Replies

9. Shell Programming and Scripting

for / foreach syntax issues (in bash or tcsh)

So I am new to unix, and actually anything outside drag and drop with the mouse (been learning for about a week so far) . I have been using the foreach command in tcsh because I am working on a group of files. Basically what I need is to insert part of the filename as the first line in the file.... (0 Replies)
Discussion started by: thepolypore
0 Replies

10. Shell Programming and Scripting

Using sed with a foreach loop

So I am back again beating my head against the wall with a shell script and getting a headache! I want to change each year in a file (1980, 1981, 1982, 1983, etc.) to the same year followed by a tab. The input is "blah blah (1980) blah blah". I want to get "blah blah (1980 ) blah blah".... (2 Replies)
Discussion started by: Peggy White
2 Replies
Login or Register to Ask a Question