Sponsored Content
Top Forums Shell Programming and Scripting How to select only the most frequent instances of a variable string in a file? Post 302355794 by Scrutinizer on Wednesday 23rd of September 2009 07:56:17 PM
Old 09-23-2009
How big is your access file? If it is very big then the times the input file gets read is proportional to the amount of ip addresses. Which may take time for very large files.

This script will ony read the file once:

Code:
#!/bin/ksh
typeset -A ACCESS
while read ipaddr dummy; do
  (( ACCESS[$ipaddr]++ ))
done<access_log
for ip in ${!ACCESS[@]}; do
  echo $ip ${ACCESS[$ip]}
done|sort -rn -k2|head -10

or the equivalent awk:
Code:
awk '{access[$1]++} END { for ( i in access ) print i " " access[i] }' access_log |sort -rn -k2|head -10

Note that sort options "-rn -k2" stands for "reverse numerical sort on the second field". The syntax may vary per Unix platform; use "man sort" to find out the appropriate options. Head determines the number of IP addresses to list.

Last edited by Scrutinizer; 09-24-2009 at 07:24 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to replace all string instances found by find+grep

Hello all Im performing find + grep operation that looks like this : find . -name "*.dsp" | xargs grep -on Project.lib | grep -v ':0' and I like to add to this one liner the possibility to replace the string " Project.lib" that found ( more then once in file ) with "Example.lib" how can I do... (0 Replies)
Discussion started by: umen
0 Replies

2. Shell Programming and Scripting

Replacing string in all instances (both filenames and file contents) in a directory

Hi, I have a set of files stored in a single directory that I use to set parameters for a physics code, and I would like to streamline the process of updating them all when I change a parameter. For instance, if the files are called A2000p300ini, A2000p300sub, A2000p300run, and the text in each... (3 Replies)
Discussion started by: BlueChris
3 Replies

3. Shell Programming and Scripting

How to select the path that contains a certain string from a certain file?

Hi, I am new to this world of shell programming. I am facing a problem that is : I have directory which has many sub directories at different depth. say A/B/C/files A/B/files A/B/C/D/files In this directory structure there exists a file called ".project" in some of the sub... (2 Replies)
Discussion started by: bhaskar_m
2 Replies

4. Programming

Optimizing frequent file transfer?

Hi I have written a simple client/server(socket programming) application using TCP/IP. My server code runs on Linux and client is on windows. The concept is that the client request for files(on demand basis) to the server and the server sends it back to the client. As the client is attached to... (3 Replies)
Discussion started by: akilan
3 Replies

5. Shell Programming and Scripting

Using sed to replace a string in file with a string in a variable that contains spaces

Hi, i call my shell like: my_shell "my project name" my script: #!/bin/bash -vx projectname=$1 sed s/'PROJECT_NAME ='/'PROJECT_NAME = '$projectname/ <test_config_doxy >temp cp temp test_config_doxy the following error occurres: sed s/'PROJECT_NAME ... (2 Replies)
Discussion started by: vivelafete
2 Replies

6. Shell Programming and Scripting

Appending string, variable to file at the start and string at end

Hi , I have below file with 13 columns. I need 2-13 columns seperated by comma and I want to append each row with a string "INSERT INTO xxx" in the begining as 1st column and then a variable "$node" and then $2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13 and at the end another string " ; COMMIT;" ... (4 Replies)
Discussion started by: Vaddadi
4 Replies

7. Shell Programming and Scripting

how to test input variable is a string in a select loop

Okay -- I hope I ask this correctly. I'm working on my little shell script to write vendor names and aliases to files from user input. If a user choose to add to a file, he can do that as well. I'm using a select loop for this function to list all the possible files the user can choose from.... (7 Replies)
Discussion started by: Straitsfan
7 Replies

8. Shell Programming and Scripting

replace (sed?) a string in file with multiple lines (string) from variable

Can someone tell me how I can do this? e.g: a=$(echo -e wert trewt ertert ertert ertert erttert erterte rterter tertertert ert) How do i replace the STRING with $a? I try this: sed -i 's/STRING/'"$a"'/g' filename.ext but this don' t work (2 Replies)
Discussion started by: jforce
2 Replies

9. Shell Programming and Scripting

Counting Instances of a String with AWK

I have a list of URLs and I want to be able to count the number of instances of addresses ending in a certain TLD and output and sort it like so. 5 bdcc.com 48 zrtzr.com 49 rvo.com Input is as so ync.org sduzj.edu sduzj.edu sduzj.edu sduzj.edu sduzj.edu sduzj.edu sduzj.edu... (1 Reply)
Discussion started by: Pjstaab
1 Replies

10. Shell Programming and Scripting

Printing most frequent string in column

I am trying to put together an script that will output the most frequent string in a column. This is what I have: awk '{count++} END {for ( i in count ) print i, count }' Of course, my script is outputting all different strings and counts. However, I just need the most frequent one (there... (7 Replies)
Discussion started by: Xterra
7 Replies
shell_builtins(1)														 shell_builtins(1)

NAME
shell_builtins, case, for, foreach, function, if, repeat, select, switch, until, while - shell command interpreter built-in commands The shell command interpreters csh(1), ksh(1), and sh(1) have special built-in commands. The commands case, for, foreach, function, if, repeat, select, switch, until, and while are commands in the syntax recognized by the shells. They are described in the Commands section of the manual pages of the respective shells. The remaining commands listed in the table below are built into the shells for reasons such as efficiency or data sharing between command invocations. They are described on their respective manual pages. | Command | Shell alias |csh, ksh bg |csh, ksh, sh break |csh, ksh, sh case |csh, ksh, sh cd |csh, ksh, sh chdir |csh, sh continue |csh, ksh, sh dirs |csh echo |csh, ksh, sh eval |csh, ksh, sh exec |csh, ksh, sh exit |csh, ksh, sh export |ksh, sh false |ksh fc |ksh fg |csh, ksh, sh for |ksh, sh foreach |csh function |ksh getopts |ksh, sh glob |csh goto |csh hash |ksh, sh hashstat |csh history |csh if |csh, ksh, sh jobs |csh, ksh, sh kill |csh, ksh, sh let |ksh limit |csh login |csh, ksh, sh logout |csh, ksh, sh nice |csh newgrp |ksh, sh nohup |csh notify |csh onintr |csh popd |csh print |ksh pushd |csh pwd |ksh, sh read |ksh, sh readonly |ksh, sh rehash |csh repeat |csh return |ksh, sh select |ksh set |csh, ksh, sh setenv |csh shift |csh, ksh, sh source |csh stop |csh, ksh, sh suspend |csh, ksh, sh switch |csh test |ksh, sh time |csh times |ksh, sh trap |ksh, sh true |ksh type |ksh, sh typeset |ksh ulimit |ksh, sh umask |csh, ksh, sh unalias |csh, ksh unhash |csh unlimit |csh unset |csh, ksh, sh unsetenv |csh until |ksh, sh wait |csh, ksh, sh whence |ksh while |csh, ksh, sh Bourne Shell, sh, Special Commands Input/output redirection is now permitted for these commands. File descriptor 1 is the default output location. When Job Control is enabled, additional Special Commands are added to the shell's environment. In addition to these built-in reserved command words, sh also uses: : No effect; the command does nothing. A zero exit code is returned. .filename Read and execute commands from filename and return. The search path specified by PATH is used to find the directory con- taining filename. C shell, csh Built-in commands are executed within the C shell. If a built-in command occurs as any component of a pipeline except the last, it is exe- cuted in a subshell. In addition to these built-in reserved command words, csh also uses: : Null command. This command is interpreted, but performs no action. Korn Shell, ksh, Special Commands Input/Output redirection is permitted. Unless otherwise indicated, the output is written on file descriptor 1 and the exit status, when there is no syntax error, is zero. Commands that are preceded by one or two * (asterisks) are treated specially in the following ways: 1. Variable assignment lists preceding the command remain in effect when the command completes. 2. I/O redirections are processed after variable assignments. 3. Errors cause a script that contains them to abort. 4. Words, following a command preceded by ** that are in the format of a variable assignment, are expanded with the same rules as a vari- able assignment. This means that tilde substitution is performed after the = sign and word splitting and file name generation are not performed. In addition to these built-in reserved command words, ksh also uses: * : [ arg ... ] The command only expands parameters. * .file [ arg ..Read the complete file then execute the commands. The commands are executed in the current shell environment. The search path specified by PATH is used to find the directory containing file. If any arguments arg are given, they become the posi- tional parameters. Otherwise, the positional parameters are unchanged. The exit status is the exit status of the last com- mand executed. the loop termination test. intro(1), alias(1), break(1), cd(1), chmod(1), csh(1), echo(1), exec(1), exit(1), find(1), getoptcvt(1), getopts(1), glob(1), hash(1), his- tory(1), jobs(1), kill(1), ksh(1), let(1), limit(1), login(1), logout(1), newgrp(1), nice(1), nohup(1), print(1), pwd(1), read(1), read- only(1), set(1), sh(1), shift(1), suspend(1), test(1B), time(1), times(1), trap(1), typeset(1), umask(1), wait(1), chdir(2), chmod(2), creat(2), umask(2), getopt(3C), profile(4), environ(5) 29 Jun 2005 shell_builtins(1)
All times are GMT -4. The time now is 04:17 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy