Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Parse for 2 numbers in large single line

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parse for 2 numbers in large single line
# 1  
Old 08-25-2018
Parse for 2 numbers in large single line
Hi All,

I am writing a script in which I need to gather 2 numbers for 'total' and 'successful'. The goal is to compare the two numbers and if they are not equal, rerun the task until all are successful. I'm thinking the best way will be with awk or sed, but I really don't know where to begin with this one.

The line that is outputted from the task is a very large single line with no spaces. However the beginning of the line appears to be consistent. Here is the output from my 4 manual runs:

Code:
{"_shards":{"total":1717,"successful":1712,"failed":5}
{"_shards":{"total":1717,"successful":1715,"failed":2}
{"_shards":{"total":1717,"successful":1714,"failed":3}
{"_shards":{"total":1717,"successful":1717,"failed":0}

As mentioned, it looks like the easiest way to test the need for rerunning is comparing the total and successful numbers.

Can anyone provide any guidance as to how to gather the 2 numbers and then do a loop to rerun the task until there are no errors?

Any help is greatly appreciated.

Thanks in advance,

HB
# 2  
Old 08-26-2018
Does the task not use an exit code ? You can test for it until it is succesful
Code:
until task
do
  :
done

This will run indefinitely if task will never be successful

Alternatively (bash/ksh) you can limit the number of attempts and report about them.
Code:
for i in {1..10}
do
  if task
  then
    echo "successful after ${i} attempt(s)"
    break
  fi
  if (( i >= 10 )); then
    echo "Max number of attempts exceeded; task was unsuccessful"
    exit 1
  fi
done


--
If there is no return code and testing the output is the only option, then I suggest testing for the number of fails :
Code:
until [[ $result =~ ^\{\"_shards\":\{\"total\":[0-9]+,\"successful\":[0-9]+,\"failed\":0\} ]]
do
  result=$(task)
done

Likewise, you can use
Code:
result=$(task)
if [[ $result =~ ^\{\"_shards\":\{\"total\":[0-9]+,\"successful\":[0-9]+,\"failed\":0\} ]]

in the second example...
Last edited by Scrutinizer; 08-26-2018 at 04:41 AM..
This User Gave Thanks to Scrutinizer For This Post:
hburnswell
# 3  
Old 08-26-2018
Like Scrutinizer, I'd go for the "failed" count:
Code:
while IFS=":}" read -a INARR  <<< $(task) && [ 0 -ne "${INARR[${#INARR[@]}-1]}" ]; do :; done

This User Gave Thanks to RudiC For This Post:
hburnswell
# 4  
Old 08-26-2018
If task doesn't return a useful exit code (i.e., always returns a zero exit status), one could still skip parsing counts and just loop until success is found:
Code:
while line=$(task)
do	if [ "$line" = "${line%:0\}}" ]
	then
		echo "One or more tests failed: $line"
	else
		echo "All tests passed: $line"
		break
	fi
done

This User Gave Thanks to Don Cragun For This Post:
hburnswell
# 5  
Old 08-26-2018
Thank you for the responses.

Yes, unfortunately the exit code is always zero no matter the failures. I should have given more information, sorry.

The task is actually a 'curl' command for use with an Elasticsearch environment:

Code:
curl -u elastic -X POST "localhost:9200/_flush/synced"

As mentioned, the output is a very long single line, i.e. <truncated>:

Code:
{"_shards":{"total":1717,"successful":1717,"failed":0},"metricbeat-2018.08.01":{"total":2,"successful":2,"failed":0},"metricbeat-2018.08.02":{"total":2,"successful":2,"failed":0},"metricbeat-2018.08.03":{"total":2,"successful":2,"failed":0},....

The command will receive a 200 HTTPD response and a zero exit code.

I agree Scrutinizer & RudiC, the 'failed' count is what I will use.

I am trying a couple renditions now and will report back..

Thanks for the guidance.

HB
Last edited by hburnswell; 08-26-2018 at 02:54 PM..
# 6  
Old 08-26-2018
Go for

Code:
$(curl ... |  grep -Eo "shards[^}]*failed\":[0-9]*")

These 2 Users Gave Thanks to RudiC For This Post:
hburnswell jim mcnamara
# 7  
Old 08-26-2018
RudiC thank you for that suggestion of: 'grep -Eo'... It allowed me to get it working with:

Code:
synced_flush () {

        $curl -s -u ${creds} -X POST "localhost:9200/_flush/synced" | \
        grep -Eo "shards[^}]*failed\":[0-9]*" | \
        sed -e 's/.*://'

}

Code:
until [[ $(synced_flush) == "0" ]]

        do

                synced_flush

        done

This appears to be doing the trick. If there is something really wrong with this approach please let me know. Like most of the scripts I write, they start off 'working' and I improve the efficiency with time ;-) ...

Thanks again scrutinizer, Don Cragun, and RudiC.. Much appreciated.

HB
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Need to parse the multiple definitions from a single line and assign

Hi, I need a help on my requirement that eg: NEED="TEST=Name WORK=Ps DEL=let" Here the definition can be n number, could anybody have an idea to get the output as, TEST=Name WORK=Ps DEL=let .. .. till the 'n' definitions listed. Any suggestions please..... Regards, ricky (6 Replies)
Discussion started by: ricky-row
6 Replies

2. Shell Programming and Scripting

How to add line breaks to perl command with large text in single quotes?

Below code extracts multiple field values from XML into array and prints all in one line. perl -nle '@r=/(?: jndiName| authDataAlias| value| minConnections| maxConnections| connectionTimeout| name)="(+)/g and print join ",",$ENV{tIPnSCOPE},$ENV{pr ovider},$ENV{impClassName},@r' server.xml ... (4 Replies)
Discussion started by: kchinnam
4 Replies

3. Shell Programming and Scripting

Adding Long List Of Large Numbers

Hi All, I have a file with long list of numbers. This file contains only one column. These numbers are very large. I am using following command: cat myfile.txt | awk '{ sum+=$1} END {print sum}' The output is coming in scientific notation. How do I get the result in proper format? ... (4 Replies)
Discussion started by: angshuman
4 Replies

4. UNIX for Dummies Questions & Answers

Hope to create a file with two large column, with several numbers

I hope to create a file made up of 2 columns - first column print out number 0~61000 every 50 of it - second column just contains 0 delineated by space such as 0 0 50 0 100 0 150 0 200 0 ... 60900 0 60950 0 61000 0 Which command should I need to use? I think I might need to use... (5 Replies)
Discussion started by: exsonic
5 Replies

5. Shell Programming and Scripting

Parse a single line file and store value.

I have a single line file like this : Average Fragmentation Quotient : 3.084121 Now I want to store the value which comes after ":" i,e 3.084121 into a variable. And if this variable crosses above 6 i want to call another script... can any one help me on this... (7 Replies)
Discussion started by: Hyp_Todd
7 Replies

6. Shell Programming and Scripting

Parse large file on line count (random lines)

I have a file that needs to be parsed into multiple files every time there line contains a number 1. the problem i face is the lines are random and the file size is random. an example is that on line 4, 65, 187, 202 & 209 are number 1's so there has to be file breaks between all those to create 4... (6 Replies)
Discussion started by: darbs121
6 Replies

7. Programming

Working with extremely large numbers in C

Hi All, I am just curious, not programming anything of my own. I know there are libraries like gmp which does all such things. But I really need to know HOW they do all such things i.e. working with extremely large unimaginable numbers which are beyond the integer limit. They can do add,... (1 Reply)
Discussion started by: shoaibjameel123
1 Replies

8. Shell Programming and Scripting

A mistake in awk command I used to parse numbers

Hi I have a big file with a certain pattern (shown below) from which I need to parse out some digits in tabular format. The format of the file is: '-' indicates text which doesn't to be parsed # Output of huzzle for sequence file 1000.Clade1.html - - - -- -------... (2 Replies)
Discussion started by: Lucky Ali
2 Replies

9. AIX

Backup single large file

Hi I have a single large file 11gb that I need to copy/backup to tape then restore on another system. I tried tar but that complained about the file being too large Anyone have any suggestions how I can do this with AIX 5.2 Much appreciated. (3 Replies)
Discussion started by: Alvescot
3 Replies

10. Shell Programming and Scripting

How to parse large numbers of shell scripts

I am trying to parse hundreds of shell scripts to determine how they related to each other. Ideally for every script, I would get an output of: What other scripts it calls What files it reads Environment variables it accesses Any ideas on how to do this? TIA! (2 Replies)
Discussion started by: bliss
2 Replies
Login or Register to Ask a Question