Sponsored Content
Full Discussion: Best search technique
Top Forums Shell Programming and Scripting Best search technique Post 302408352 by chakrapani on Monday 29th of March 2010 09:17:03 AM
Old 03-29-2010
Hi Thanks for your reply.

Currently I am using code:

Code:
#!/bin/bash

function lookfor {
y=$1
s=$2
ONE="1"
echo -e "NOW Processing file: $s \n\n"
for file in $(cat $s )
do
# Make sure the lenght is bigger of looking number is bigger than 5
if [ "${#file}" -ge 5 ]
 then
 VAL=$( grep "$file" $y | awk '{ print $2}' | sort | uniq | wc -l )
 if [ -n "$VAL" ]
   then
    if [ "x$VAL" == "x$ONE" ]
       then
        echo  -en "$file"
        grep "|$file|" $y >> fnd.txt
     fi
   fi
echo -en "." # Show some activity
else
  echo -en "-" # Show some activity that I rejected this number
fi
done
}

lookfor "hugelog1.log" "firstRowUniq.1" ;
lookfor "hugelog2.log" "firstRowUniq.2" ;
lookfor "hugelog3.log" "firstRowUniq.3" ;

I have broken hugelog in some files and firstRowUniq is the file having uniq
of the first column from hugelog file. in Linux what I mean

$ awk '{ print $1 }' hugelog1.log | sort | uniq > firstRowUniq.1

Please note log file hugelog*.log is not sorted. the size of all hugefile combined is around 3 GB
 

8 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Difference Technique's???

Is there any better way of doing this? I only want to find a status of a diff, ie diff the file and return to me whether it is different or not or non-existant. This example works, however I think it could be less messier: workd=`pwd`;find $workd -name "*.sum" | while read line ; do... (1 Reply)
Discussion started by: Shakey21
1 Replies

2. UNIX for Dummies Questions & Answers

FORK/EXEC technique

Hi! Can someone explain me exactly this technique? Why a process (PARENT) creates a copy of itself with FORK (CHILD)? What's the reason of this behaviour? Sorry, but I cannot understand the logic behind it. Thanks. (4 Replies)
Discussion started by: marshmallow
4 Replies

3. Shell Programming and Scripting

Perl - use search keywords from array and search a file and print 3rd field when matched

Hi , I have been trying to write a perl script to do this job. But i am not able to achieve the desired result. Below is my code. my $current_value=12345; my @users=("bob","ben","tom","harry"); open DBLIST,"<","/var/tmp/DBinfo"; my @input = <DBLIST>; foreach (@users) { my... (11 Replies)
Discussion started by: chidori
11 Replies

4. Shell Programming and Scripting

Password Obscuring Technique

Hi, We have a unix shell script which tries login to database. The user name and password to connect to database is stored in a file connection.sql. Now connection.sql has contents def ora_user =&1 CONNECT A_PROXY/abc123@DEV01 When on UNIX server we connect to database and set spool on... (7 Replies)
Discussion started by: Gangadhar Reddy
7 Replies

5. Linux

Best Compression technique ?

Hi all, I am working on a sample backup code, where i read the files per 7200 bytes and send it to server. Before sending to server, i compress each 7200 bytes using zlib compression algorithm using dictionary max length of 1.5 MB . I find zlib is slow. Can anyone recommend me a... (3 Replies)
Discussion started by: selvarajvss
3 Replies

6. Shell Programming and Scripting

Perl - start search by using search button or by pressing the enter key

#Build label and text box $main->Label( -text => "Input string below:" )->pack(); $main->Entry( -textvariable => \$text456 )->pack(); $main->Button( -text => "Search", -command => sub { errchk ($text456) ... (4 Replies)
Discussion started by: popeye
4 Replies

7. Shell Programming and Scripting

Search pattern on logfile and search for day/dates and skip duplicate lines if any

Hi, I've written a script to search for an Oracle ORA- error on a log file, print that line and the .trc file associated with it as well as the dateline of when I assumed the error occured. In most it is the first dateline previous to the error. Unfortunately, this is not a fool proof script.... (2 Replies)
Discussion started by: newbie_01
2 Replies

8. What is on Your Mind?

YouTube: Search Engine Optimization | How To Fix Soft 404 Errors and A.I. Tales from Google Search

Getting a bit more comfortable making quick YT videos in 4K, here is: Search Engine Optimization | How To Fix Soft 404 Errors and A.I. Tales from Google Search Console https://youtu.be/I6b9T2qcqFo (0 Replies)
Discussion started by: Neo
0 Replies
UNHIDE(8)						      System Manager's Manual							 UNHIDE(8)

NAME
unhide -- forensic tool to find hidden processes SYNOPSIS
unhide [OPTIONS] TEST_LIST DESCRIPTION
unhide is a forensic tool to find processes hidden by rootkits, Linux kernel modules or by other techniques. It detects hidden processes using six techniques. OPTIONS
-f Write a log file (unhide.log) in the current directory. -h Display help -m Do more checks. As of 2010-11-21 version, this option has only effect for the procfs, procall, checkopendir and checkchdir tests. Implies -v -r Use alternate version of sysinfo check in standard tests -V Show version and exit -v Be verbose, display warning message (default : don't display). This option may be repeated more than once. TEST_LIST The checks to do consist of one or more of the following tests. The standard tests are the aggregation of one or more elementary test(s). Standard tests : The brute technique consists of bruteforcing the all process IDs. This technique is only available on Linux 2.6 kernels. The proc technique consists of comparing /proc with the output of /bin/ps. The procall technique combinates proc and procfs tests. This technique is only available on Linux 2.6 kernels. The procfs technique consists of comparing information gathered from /bin/ps with information gathered by walking in the procfs. With -m option, this test makes more checks, see checkchdir test. This technique is only available on Linux 2.6 kernels. The quick technique combines the proc, procfs and sys techniques in a quick way. It's about 20 times faster but may give more false posi- tives. This technique is only available on Linux 2.6 kernels. The reverse technique consists of verifying that all threads seen by ps are also seen in procfs and by system calls. It is intended to ver- ify that a rootkit has not killed a security tool (IDS or other) and make ps showing a fake process instead. This technique is only available on Linux 2.6 kernels. The sys technique consists of comparing information gathered from /bin/ps with information gathered from system calls. Elementary tests : The checkbrute technique consists of bruteforcing the all process IDs. This technique is only available on Linux 2.6 kernels. The checkchdir technique consists of comparing information gathered from /bin/ps with information gathered by making chdir() in the procfs. With the -m option, it also verify that the thread appears in its "leader process" threads list. This technique is only available on Linux 2.6 kernels. The checkgetaffinity technique consists of comparing information gathered from /bin/ps with the result of call to the sched_getaffinity() system function. This technique is only available on Linux 2.6 kernels. The checkgetparam technique consists of comparing information gathered from /bin/ps with the result of call to the sched_getparam() system function. This technique is only available on Linux 2.6 kernels. The checkgetpgid technique consists of comparing information gathered from /bin/ps with the result of call to the getpgid() system func- tion. This technique is only available on Linux 2.6 kernels. The checkgetprio technique consists of comparing information gathered from /bin/ps with the result of call to the getpriority() system function. This technique is only available on Linux 2.6 kernels. The checkRRgetinterval technique consists of comparing information gathered from /bin/ps with the result of call to the sched_rr_get_inter- val() system function. This technique is only available on Linux 2.6 kernels. The checkgetsched technique consists of comparing information gathered from /bin/ps with the result of call to the sched_getscheduler() system function. This technique is only available on Linux 2.6 kernels. The checkgetsid technique consists of comparing information gathered from /bin/ps with the result of call to the getsid() system function. This technique is only available on Linux 2.6 kernels. The checkkill technique consists of comparing information gathered from /bin/ps with the result of call to the kill() system function. Note : no process is really killed by this test. This technique is only available on Linux 2.6 kernels. The checknoprocps technique consists of comparing the result of the call to each of the system functions. No comparison is done against /proc or the output of ps. This technique is only available on Linux 2.6 kernels. The checkopendir technique consists of comparing information gathered from /bin/ps with information gathered by making opendir() in the procfs. This technique is only available on Linux 2.6 kernels. The checkproc technique consists of comparing /proc with the output of /bin/ps. This technique is only available on Linux 2.6 kernels. The checkquick technique combines the proc, procfs and sys techniques in a quick way. It's about 20 times faster but may give more false positives. This technique is only available on Linux 2.6 kernels. The checkreaddir technique consists of comparing information gathered from /bin/ps with information gathered by making readdir() in /proc and /proc/pid/task. This technique is only available on Linux 2.6 kernels. The checkreverse technique consists of verifying that all threads seen by ps are also seen in procfs and by system calls. It is intended to verify that a rootkit has not killed a security tool (IDS or other) and make ps showing a fake process instead. This technique is only available on Linux 2.6 kernels. The checksysinfo technique consists of comparing the number of process seen by /bin/ps with information obtained from sysinfo() system call. This technique is only available on Linux 2.6 kernels. The checksysinfo2 technique is an alternate version of checksysinfo test. It might (or not) work better on kernel patched for RT, preempt or latency and with kernel that don't use the standard scheduler. It's also invoked by standard tests when using the -r option This technique is only available on Linux 2.6 kernels. Exit status: 0 if OK, 1 if a hidden or fake thread is found. BUGS
Report unhide bugs on the bug tracker on sourceforge (http://sourceforge.net/projects/unhide/) SEE ALSO
unhide-tcp (8). AUTHOR
This manual page was written by Francois Marier francois@debian.org and Patrick Gouin. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU General Public License, Version 3 or any later version published by the Free Software Foun- dation. LICENSE
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Administration commands October 2010 UNHIDE(8)
All times are GMT -4. The time now is 04:20 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy