Shell Programming and Scripting

#!/bin/bash
FILENAME=$1
COUNT=0
DOCS=$(wc -l < $FILENAME)

if [ -f words.txt ] 
    then rm words.txt
fi
if [ -f words_uniq.txt ]
    then rm words_uniq.txt
fi

while [ $COUNT -le 1000 ] 
do
      let COUNT++
      if [ $(($COUNT % 100)) -eq 0 ]
            then echo "$COUNT texts processed"
      fi
      RAND=$((RANDOM % DOCS + 1))
      sed -n "$RAND{p;q;}" "$FILENAME" | sed "s/<.*>//g" | tr -cs "[:alpha:]" "[\n*]" | tr "[:upper:]" "[:lower:]" >> words.txt
done

n=$(wc -w < words.txt)
cat words.txt | sort | uniq > words_uniq.txt
V=$(wc -w < words_uniq.txt)
echo "V = $V, n = $n"

awk -v FS='>' '{
        $2=toupper($2);
        split($2, A, " ");
        for(key in A)   count[A[key]]++;
}
END {
        for(key in count)
                printf("%s\t%d\n", key, count[key]);
}' < inputfile

<http://dbpedia.org/resource/Albedo> <http://dbpedia.org/ontology/abstract> "The albedo of an object is a measure of how strongly it reflects light from light sources such as the Sun. It is therefore a more specific form of the term reflectivity. Albedo is defined as the ratio of total-reflected to incident electromagnetic radiation. It is a unitless measure indicative of a surface's or body's diffuse reflectivity. The word is derived from Latin albedo \"whiteness\", in turn from albus \"white\", and was introduced into optics by Johann Heinrich Lambert in his 1760 work Photometria. The range of possible values is from 0 (dark) to 1 (bright). The albedo is an important concept in climatology and astronomy, as well as in computer graphics and computer vision. In climatology it is sometimes expressed as a percentage. Its value depends on the frequency of radiation considered: unqualified, it usually refers to some appropriate average across the spectrum of visible light. In general, the albedo depends on the direction and directional distribution of incoming radiation. Exceptions are Lambertian surfaces, which scatter radiation in all directions in a cosine function, so their albedo does not depend on the incoming distribution. In realistic cases, a bidirectional reflectance distribution function (BRDF) is required to characterize the scattering properties of a surface accurately, although albedos are a very useful first approximation."@en .
<http://dbpedia.org/resource/Anarchism> <http://dbpedia.org/ontology/abstract> "Anarchism is a political philosophy which considers the state undesirable, unnecessary and harmful, and instead promotes a stateless society, or anarchy. It seeks to diminish or even abolish authority in the conduct of human relations. Anarchists may widely disagree on what additional criteria are required in anarchism. The Oxford Companion to Philosophy says, \"there is no single defining position that all anarchists hold, and those considered anarchists at best share a certain family resemblance. \" There are many types and traditions of anarchism, not all of which are mutually exclusive. Strains of anarchism have been divided into the categories of social and individualist anarchism or similar dual classifications. Anarchism is often considered to be a radical left-wing ideology, and much of anarchist economics and anarchist legal philosophy reflect anti-statist interpretations of communism, collectivism, syndicalism or participatory economics. However, anarchism has always included an individualist strain supporting a market economy and private property, or unrestrained egoism that bases right on might. Others, such as panarchists and anarchists without adjectives, neither advocate nor object to any particular form of organization as long as it is not compulsory. Differing fundamentally, some anarchist schools of thought support anything from extreme individualism to complete collectivism. The central tendency of anarchism as a social movement have been represented by communist anarchism, with individualist anarchism being primarily a philosophical or literary phenomenon. Some anarchists fundamentally oppose all forms of aggression, supporting self-defense or non-violence, while others have supported the use of some coercive measures, including violent revolution and terrorism, on the path to an anarchist society."@en .
<http://dbpedia.org/resource/Achilles> <http://dbpedia.org/ontology/abstract> "In Greek mythology, Achilles was a Greek hero of the Trojan War, the central character and the greatest warrior of Homer's Iliad. Achilles also has the attributes of being the most handsome of the heroes assembled against Troy. Later legends (beginning with a poem by Statius in the first century AD) state that Achilles was invulnerable in all of his body except for his heel. Since he died due to an arrow shot into his heel, the \"Achilles' heel\" has come to mean a person's principal weakness."@en .
<http://dbpedia.org/resource/Abraham_Lincoln> <http://dbpedia.org/ontology/abstract> "Abraham Lincoln (February 12, 1809 \u2013 April 15, 1865) served as the 16th President of the United States from March 1861 until his assassination in April 1865. He successfully led his country through its greatest internal crisis, the American Civil War, preserving the Union and ending slavery. Before his election in 1860 as the first Republican president, Lincoln had been a country lawyer, an Illinois state legislator, a member of the United States House of Representatives, and twice an unsuccessful candidate for election to the U.S. Senate. As an outspoken opponent of the expansion of slavery in the United States,"@en .

awk -v FS='>' '{
        # Get stuff after second >, and convert it all to uppercase
        $0=toupper($3);
        # Substitute every non-alpha char with space
        gsub(/[^A-Z]/, " ");
        # Split on spaces, into array A
        split($0, A, " ");
        # count words
        for(key in A)   count[A[key]]++;
}
END {
        # print words
        for(key in count)
                printf("%s\t%d\n", key, count[key]);
}' < filename