I have a text file downloaded from the web, I want to count the unique words used in the file, and a person's speaking length during conversation by counting the words between the opening and closing quotation marks which differ from the standard ASCII code. Also I found out the file contains some weird blank characters that are invisible from stdout which are the entry that has 118391 and the one has 6380 occurrence in the example.
It seems to me the file was processed with Mac PC by the single/double quotes I can guess, but I am not sure. Here is the output of my Ubuntu terminal:
1) How do I find out the invisible "blank/empty" characters in the file so that I can get rid of them in order to count the words?
2) How do I count the speaking duration of a person at conversations by the opening/closing double quotation pair? What I tried is:
This regex is too greedy that sometime combines adjacent dialogues into single one.
Thanks!
Hello,
I read a file whose in lines are datas and between thses datas there is blank characters (10, 12 or 5 or 1 .......)
So when i use the command while read line in the script(see under) there is also only one character between the datas and the others blank characters are not here.
... (3 Replies)
Hi,
I am trying to do two things in my script. I will really appreciate any help in this regards.
Is there a way to delete a last line from a pipe delimited flat file if the last line is blank. If the line is not blank then do nothing.....
Is there a way to count a word that are starting... (4 Replies)
Does anyone of you know how to turn off color and weird characters on bash shell when using the command "script"? Everytime users on my server used that command to record their script, they either couldn't print it because lp kept giving the "unknown format character" messages or the print paper... (1 Reply)
Hi.
I have files in my OS that has weird file names with not-conventional ascii characters.
I would like to run them but I can't refer them.
I know the ascii # of the problematic characters.
I can't change their name since it belongs to a 3rd party program... but I want to run it.
is there... (2 Replies)
Hi everyone,
I'm trying to write a shell script that process a log file. The log format is generally:
(8 digit hex of unix time),(system ID),(state)\n
My shell script gets the file from the web, saves it in a local text directory. I then want to change the hex to decimal, convert from unix time... (7 Replies)
I just finish the shell script .
This shell can replace weird characters (such as #$%^@!'"...) in file or directory name by "_"
I spent long time on replacing apostrophe in file/directory name
added: 2012-03-14
the 124th line (/usr/bin/perl -i -e "s#\'#\\'#g" /tmp/rpdir_level$i.tmp) is... (5 Replies)
Hi,
I was trying to remove the blank from beginning of a line.
when I try:
sed 's/^ +//' filename
it does not work
but when I try
sed 's/^ *//' filename
it works
But I think the first command should have also replaced any line with one or more blanks.
Kindly help me in understanding... (5 Replies)
Dear all,
I have the files: xaa xab xac
and I try to paste them using $paste -d, xaa xab xac
I see:
output
3e-130
,6e-78
,5e-74
6e-124
,0,007
,0,026
2e-119
When I type: $ paste -d, xaa xab xac |less
I see:
output
3e-130^M,6e-78^M,5e-74
6e-124^M,0,007^M,0,026 (2 Replies)
I am using Korn shell on Linux 2.6x platform , and I am suing the following code to capture the lines which contain CONTROL CHARACTERS in my file :
awk '/]/ {print NR}' EROLLMENT_INPUT.txt
The problem is that this code shows the file has control characters when the file is in folder A ,... (2 Replies)
Hi All
Need Help
I have a file with the below format (ABC.TXT) :
®¿¿ABCDHEJJSJJ|XCBJSKK01|M|7348974982790
HDFLJDKJSKJ|KJALKSD02|M|7378439274898
KJHSAJKHHJJ|LJDSAJKK03|F|9898982039999
(cont......)
I need to write a script where it will check for : blank lines (between rows,before... (6 Replies)
Discussion started by: chatwithsaurav
6 Replies
LEARN ABOUT V7
regex
regex(1F) FMLI Commands regex(1F)NAME
regex - match patterns against a string
SYNOPSIS
regex [-e] [ -v "string"] [ pattern template] ... pattern [template]
DESCRIPTION
The regex command takes a string from the standard input, and a list of pattern / template pairs, and runs regex() to compare the string
against each pattern until there is a match. When a match occurs, regex writes the corresponding template to the standard output and
returns TRUE. The last (or only) pattern does not need a template. If that is the pattern that matches the string, the function simply
returns TRUE. If no match is found, regex returns FALSE.
The argument pattern is a regular expression of the form described in regex(). In most cases, pattern should be enclosed in single quotes
to turn off special meanings of characters. Note that only the final pattern in the list may lack a template.
The argument template may contain the strings $m0 through $m9, which will be expanded to the part of pattern enclosed in ( ... )$0 through
( ... )$9 constructs (see examples below). Note that if you use this feature, you must be sure to enclose template in single quotes so
that FMLI does not expand $m0 through $m9 at parse time. This feature gives regex much of the power of cut(1), paste(1), and grep(1), and
some of the capabilities of sed(1). If there is no template, the default is $m0$m1$m2$m3$m4$m5$m6$m7$m8$m9.
OPTIONS
The following options are supported:
-e Evaluates the corresponding template and writes the result to the standard output.
-v "string" Uses string instead of the standard input to match against patterns.
EXAMPLES
Example 1: Cutting letters out of a string
To cut the 4th through 8th letters out of a string (this example will output strin and return TRUE):
`regex -v "my string is nice" '^.{3}(.{5})$0' '$m0'`
Example 2: Validating input in a form
In a form, to validate input to field 5 as an integer:
valid=`regex -v "$F5" '^[0-9]+$'`
Example 3: Translating an environment variable in a form
In a form, to translate an environment variable which contains one of the numbers 1, 2, 3, 4, 5 to the letters a, b, c, d, e:
value=`regex -v "$VAR1" 1 a 2 b 3 c 4 d 5 e '.*' 'Error'`
Note the use of the pattern '.*' to mean "anything else".
Example 4: Using backquoted expressions
In the example below, all three lines constitute a single backquoted expression. This expression, by itself, could be put in a menu defini-
tion file. Since backquoted expressions are expanded as they are parsed, and output from a backquoted expression (the cat command, in this
example) becomes part of the definition file being parsed, this expression would read /etc/passwd and make a dynamic menu of all the login
ids on the system.
`cat /etc/passwd | regex '^([^:]*)$0.*$' '
name=$m0
action=`message "$m0 is a user"`'`
DIAGNOSTICS
If none of the patterns match, regex returns FALSE, otherwise TRUE.
NOTES
Patterns and templates must often be enclosed in single quotes to turn off the special meanings of characters. Especially if you use the
$m0 through $m9 variables in the template, since FMLI will expand the variables (usually to "") before regex even sees them.
Single characters in character classes (inside []) must be listed before character ranges, otherwise they will not be recognized. For exam-
ple, [a-zA-Z_/] will not find underscores (_) or slashes (/), but [_/a-zA-Z] will.
The regular expressions accepted by regcmp differ slightly from other utilities (that is, sed, grep, awk, ed, and so forth).
regex with the -e option forces subsequent commands to be ignored. In other words, if a backquoted statement appears as follows:
`regex -e ...; command1; command2`
command1 and command2 would never be executed. However, dividing the expression into two:
`regex -e ...``command1; command2`
would yield the desired result.
ATTRIBUTES
See attributes(5) for descriptions of the following attributes:
+-----------------------------+-----------------------------+
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
+-----------------------------+-----------------------------+
|Availability |SUNWcsu |
+-----------------------------+-----------------------------+
SEE ALSO awk(1), cut(1), grep(1), paste(1), sed(1), regcmp(3C), attributes(5)SunOS 5.10 12 Jul 1999 regex(1F)