Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Extracting only words from a log file Post 302445267 by agama on Sunday 15th of August 2010 12:40:15 AM
Old 08-15-2010
Sed seems the best choice, then pipe output to sort -u.

This is split onto multiple lines so that comments can be added. There also might be an easier and/or cleaner way, but this does work.

Code:
sed -r '
s/<//g;                         # ditch existing < and > from input
s/>//g;
s/[ \t]([a-zA-Z])/ <\1/g;       # mark start of word with <
s/^[a-zA-Z]/<&/;              # special case to mark first word on line
s/<[^ \t]*/&> /g;             # mark end of words with >
s/^[^<]*</</;                   # delete all up to the first <
s/ //g;                         # delete blanks
s/>[^<]*/>/g;                   # delete non-words (things between > and <)
s/>//g;                         # delete all >
s/^<//;                        # ditch leading <
s/</\n/g;                       # split words with newlines
' |sort -u

If you're running on a BSD based system, then you might need to change the -r option to -E. Certainly -E is needed if you are using sed from AT&T's AST distribution.

This treats all tokens delimited by spaces/tabs that begin with an alpha character as a word. So something like Jul09 is also printed.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

extracting some words

i run a command that submits a word to WordNET which stores the search results in a document which looks like this... i searched "car" in this instance and id like to extract auto, automobile, machine, and store it in a file with the , , stripped away just the words. WordNET's results' template... (2 Replies)
Discussion started by: mark_nsx
2 Replies

2. Shell Programming and Scripting

Extracting Text Between Two Words

Hi all! Im trying to extract a portion of text from a KML and put it into a new file. Im trying to get all of the points out of it, ignoring everything else so I need only the text between <Placement> and </Placement>. Is there a way to make it extract all instances of these points and not just... (2 Replies)
Discussion started by: Grizzly
2 Replies

3. Shell Programming and Scripting

Extracting part of line between two words

Hi, I have a file few hundred MB's with text like one below in single line. 20091117 abc xyg 20091117 def ghi 20091118 ppp ttt 20091118 zzz zzz xxx I need to extract part of line from 1st occurence of pattern 20091117 till first occurence of another pattern 20091118. I tried... (3 Replies)
Discussion started by: artistic94555
3 Replies

4. Shell Programming and Scripting

words extracting

Hi, Pls assist. dn: uid=test,ou=test,dc=com description: password sunIdentityServerDeviceStatus: Active uid: test objectClass: sunIdentityServerDevice objectClass: iplanet-am-user-service objectClass: top objectClass: iPlanetPreferences sunIdentityServerDeviceType: blabla cn: default... (3 Replies)
Discussion started by: hudson03051nh
3 Replies

5. Shell Programming and Scripting

Help with extracting words from fixed length files

I am very new to scripting and need to write a script that will extract the account number from a line that begins with HDR. For example, the file is as follows HDR2010072600300405505100726 00300405505 LBJ FREEWAY DALLAS TELEGRAPH ... (9 Replies)
Discussion started by: bds052189
9 Replies

6. Shell Programming and Scripting

Extracting words from file

I am having a file from which i need to extract different length words into different file. For example 2 letter word into file2, 3 letter word into file3 and so on.... I did one using grep and shell script.. for (( i=1; i<7; i++)) do egrep -o '\<\(?{$i}\)?\>' $1 | sort -u -f|tr >file$i... (4 Replies)
Discussion started by: akhay_ms
4 Replies

7. Shell Programming and Scripting

grep - Extracting multiple key words from stdout

Hello. From command line, the command zypper info nxclient return a bloc of data : linux local # zypper info nxclient Loading repository data... Reading installed packages... Information for package nxclient: Repository: zypper_local Name: nxclient Version: 3.5.0-7 Arch: x86_64... (7 Replies)
Discussion started by: jcdole
7 Replies

8. Shell Programming and Scripting

Extracting Words from Text

Hi there, Unix Gurus Back in September last year you helped me find a way to extract the words in brackets in a textfile to a new one. In that case my textfile was made up of sentences containing an only bracketed word per sentence/line: 1. If the boss's son had been , someone would... (9 Replies)
Discussion started by: eldeingles
9 Replies

9. Shell Programming and Scripting

Extracting words and lines based on keywords

Hello! I'm trying to process a text file and am stuck at 2 extractions. Hoping someone can help me here: 1. Given a line in a text file and given a keyword, how can I extract the word preceeding the keyword using a shell command/script? For example: Given a keyword "world" in the line: ... (2 Replies)
Discussion started by: seemad
2 Replies

10. Shell Programming and Scripting

How to grep a log file for words listed in separate text file?

Hello, I want to grep a log ("server.log") for words in a separate file ("white-list.txt") and generate a separate log file containing each line that uses a word from the "white-list.txt" file. Putting that in bullet points: Search through "server.log" for lines that contain any word... (15 Replies)
Discussion started by: nbsparks
15 Replies
SORT(1) 						      General Commands Manual							   SORT(1)

NAME
sort - sort and/or merge files SYNOPSIS
sort [ -cmuMbdfinrwtx ] [ +pos1 [ -pos2 ] ... ] ... [ -k pos1 [ ,pos2 ] ] ... [ -o output ] [ -T dir ... ] [ option ... ] [ file ... ] DESCRIPTION
Sort sorts lines of all the files together and writes the result on the standard output. If no input files are named, the standard input is sorted. The default sort key is an entire line. Default ordering is lexicographic by runes. The ordering is affected globally by the following options, one or more of which may appear. -M Compare as months. The first three non-white space characters of the field are folded to upper case and compared so that precedes etc. Invalid fields compare low to -b Ignore leading white space (spaces and tabs) in field comparisons. -d `Phone directory' order: only letters, accented letters, digits and white space are significant in comparisons. -f Fold lower case letters onto upper case. Accented characters are folded to their non-accented upper case form. -i Ignore characters outside the ASCII range 040-0176 in non-numeric comparisons. -w Like -i, but ignore only tabs and spaces. -n An initial numeric string, consisting of optional white space, optional plus or minus sign, and zero or more digits with optional decimal point, is sorted by arithmetic value. -g Numbers, like -n but with optional e-style exponents, are sorted by value. -r Reverse the sense of comparisons. -tx `Tab character' separating fields is x. The notation +pos1 -pos2 restricts a sort key to a field beginning at pos1 and ending just before pos2. Pos1 and pos2 each have the form m.n, optionally followed by one or more of the flags Mbdfginr, where m tells a number of fields to skip from the beginning of the line and n tells a number of characters to skip further. If any flags are present they override all the global ordering options for this key. A missing .n means .0; a missing -pos2 means the end of the line. Under the -tx option, fields are strings separated by x; otherwise fields are non-empty strings separated by white space. White space before a field is part of the field, except under option -b. A b flag may be attached independently to pos1 and pos2. The notation -k pos1[,pos2] is how POSIX sort defines fields: pos1 and pos2 have the same format but different meanings. The value of m is origin 1 instead of origin 0 and a missing .n in pos2 is the end of the field. When there are multiple sort keys, later keys are compared only after all earlier keys compare equal. Lines that otherwise compare equal are ordered with all bytes significant. These option arguments are also understood: -c Check that the single input file is sorted according to the ordering rules; give no output unless the file is out of sort. -m Merge; assume the input files are already sorted. -u Suppress all but one in each set of equal lines. Ignored bytes and bytes outside keys do not participate in this comparison. -o The next argument is the name of an output file to use instead of the standard output. This file may be the same as one of the inputs. -Tdir Put temporary files in dir rather than in /tmp. EXAMPLES
Print in alphabetical order all the unique spellings in a list of words where capitalized words differ from uncapitalized. Print the users file sorted by user name (the second colon-separated field). Print the first instance of each month in an already sorted file. Options -um with just one input file make the choice of a unique representative from a set of equal lines predictable. grep -n '^' input | sort -t: +1f +0n | sed 's/[0-9]*://' A stable sort: input lines that compare equal will come out in their original order. FILES
/tmp/sort.<pid>.<ordinal> SOURCE
/sys/src/cmd/sort.c SEE ALSO
uniq(1), look(1) DIAGNOSTICS
Sort comments and exits with non-null status for various trouble conditions and for disorder discovered under option -c. BUGS
An external null character can be confused with an internally generated end-of-field character. The result can make a sub-field not sort less than a longer field. Some of the options, e.g. -i and -M, are hopelessly provincial. SORT(1)
All times are GMT -4. The time now is 11:17 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy