Sponsored Content
Top Forums Shell Programming and Scripting Search for patterns in thousands of files Post 302785631 by danish0909 on Tuesday 26th of March 2013 04:52:31 AM
Old 03-26-2013
Search for patterns in thousands of files

Hi All,


I want to search for a certain string in thousands of files and these files are distributed over different directories created daily. For that I created a small script in bash but while running it I am getting the below error:

/ms.sh: xrealloc: subst.c:5173: cannot allocate 268435456 bytes (536977408 bytes allocated)

Pasting the code that I wrote:

Code:
#!/usr/local/bin/bash

for i in `cat msisdn_u.txt`
do

cd /comptel4/elink/backup1/output/vas/NG0/20130301
find ./*GPX.Z|xargs zcat|grep $i; cd ..
cd /comptel4/elink/backup1/output/vas/NG0/20130302
find ./*GPX.Z|xargs zcat|grep $i; cd ..
cd /comptel4/elink/backup1/output/vas/NG0/20130303
find ./*GPX.Z|xargs zcat|grep $i; cd ..
cd /comptel4/elink/backup1/output/vas/NG0/20130304
find ./*GPX.Z|xargs zcat|grep $i; cd ..
cd /comptel4/elink/backup1/output/vas/NG0/20130305
find ./*GPX.Z|xargs zcat|grep $i; cd ..
cd /comptel4/elink/backup1/output/vas/NG0/20130306
find ./*GPX.Z|xargs zcat|grep $i; cd ..
cd /comptel4/elink/backup1/output/vas/NG0/20130307
find ./*GPX.Z|xargs zcat|grep $i; cd ..
..
..
..
done

This is in the patterns file:

Code:
more msisdn_u.txt
0564891888
0500555401
0563433343
0561132174
0562714661
0543210172
0503588147
0541400224
0564445889
0544998887
0564543055
0544095240
0563211334

Please advise as I need to find out and report it to the management.

Thanks

Danish

Last edited by radoulov; 03-26-2013 at 08:21 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Finding a specific pattern from thousands of files ????

Hi All, I want to find a specific pattern from approximately 400000 files on solaris platform. Its very heavy for me to grep that pattern to each file individually. Can anybody suggest me some way to search for specific pattern (alpha numeric) from these forty thousand files. Please note that... (6 Replies)
Discussion started by: aarora_98
6 Replies

2. UNIX Desktop Questions & Answers

how to search files efficiently using patterns

hi friens, :) if i need to find files with extension .c++,.C++,.cpp,.Cpp,.CPp,.cPP,.CpP,.cpP,.c,.C wat is the pattern for finding them :confused: (2 Replies)
Discussion started by: arunsubbhian
2 Replies

3. UNIX for Advanced & Expert Users

Copying Thousands of Tiny or Empty Files?

There is a procedure I do here at work where I have to synchronize file systems. The source file system always has three or four directories of hundreds of thousands of tiny (1k or smaller) or empty files. Whenever my rsync command reaches these directories, I'm waiting for hours for those files... (3 Replies)
Discussion started by: deckard
3 Replies

4. UNIX for Advanced & Expert Users

Best way to search for patterns in huge text files

I have the following situation: a text file with 50000 string patterns: abc2344536 gvk6575556 klo6575556 .... and 3 text files each with more than 1 million lines: ... 000000 abc2344536 46575 0000 000000 abc2344536 46575 4444 000000 abc2344555 46575 1234 ... I... (8 Replies)
Discussion started by: andy2000
8 Replies

5. Shell Programming and Scripting

help to parallelize work on thousands of files

I need to find a smarter way to process about 60,000 files in a single directory. Every night a script runs on each file generating a output on another directory; this used to take 5 hours, but as the data grows it is taking 7 hours. The files are of different sizes, but there are 16 cores... (10 Replies)
Discussion started by: vhope07
10 Replies

6. UNIX for Dummies Questions & Answers

script to search patterns inside list of files

>testfile while read x do if then echo $x >> testfile else fi if then echo $x >> testfile else fi done < list_of_files is there any efficient way to search abc.dml and xyz.dml ? (2 Replies)
Discussion started by: dr46014
2 Replies

7. Shell Programming and Scripting

to read two files, search for patterns and store the output in third file

hello i have two files temp.txt and temp_unique.text the second file consists the unique fields from the temp.txt file the strings stored are in the following form 4,4 17,12 15,65 4,4 14,41 15,65 65,89 1254,1298i'm able to run the following script to get the total count of a... (3 Replies)
Discussion started by: vaibhavkorde
3 Replies

8. SuSE

Search all files based on first and in all listed files search the second patterns

Hello Linux Masters, I am not a linux expert therefore i need help from linux gurus. Well i have a requirement where i need to search all files based on first patterns and after seraching all files then serach second pattern in all files which i have extracted based on first pattern.... (1 Reply)
Discussion started by: Black-Linux
1 Replies

9. Shell Programming and Scripting

Bash-awk to process thousands of files

Hi to all, I have thousand of files in a folder with names with format "FILE-YYYY-MM-DD-HHMM" for what I want to send the following AWK command awk '/Code.*/' FILE-2014* I'd like to separate all files that have the same date to a folder named with the corresponding date. For example, if I... (7 Replies)
Discussion started by: Ophiuchus
7 Replies

10. Shell Programming and Scripting

Bash - Find files excluding file patterns and subfolder patterns

Hello. For a given folder, I want to select any files find $PATH1 -f \( -name "*" but omit any files like pattern name ! -iname "*.jpg" ! -iname "*.xsession*" ..... \) and also omit any subfolder like pattern name -type d \( -name "/etc/gconf/gconf.*" -o -name "*cache*" -o -name "*Cache*" -o... (2 Replies)
Discussion started by: jcdole
2 Replies
PERIODIC.CONF(5)					      BSD File Formats Manual						  PERIODIC.CONF(5)

NAME
periodic.conf -- periodic job configuration information DESCRIPTION
The file periodic.conf contains a description of how daily, weekly and monthly system maintenance jobs should run. It resides in the /etc/defaults directory and parts may be overridden by a file of the same name in /etc, which itself may be overridden by the /etc/periodic.conf.local file. The periodic.conf file is actually sourced as a shell script from each of the periodic scripts and is intended to simply provide default con- figuration variables. The following variables are used by periodic(8) itself: local_periodic (str) List of directories to search for periodic scripts. This list is always prefixed with /etc/periodic, and is only used when an argument to periodic(8) is not an absolute directory name. <dir>_output (path or list) What to do with the output of the scripts executed from the directory dir. If this variable is set to an absolute path name, output is logged to that file, otherwise it is taken as one or more space separated email addresses and mailed to those users. If this variable is not set or is empty, output is sent to standard output. For an unattended machine, suitable values for daily_output, weekly_output, and monthly_output might be ``/var/log/daily.log'', ``/var/log/weekly.log'', and ``/var/log/monthly.log'' respectively, as newsyslog(8) will rotate these files (if they exists) at the appropriate times. <dir>_show_success <dir>_show_info <dir>_show_badconfig (bool) These variables control whether periodic(8) will mask the output of the executed scripts based on their return code (where dir is the base directory name in which each script resides). If the return code of a script is '0' and <dir>_show_success is set to ``NO'', periodic(8) will mask the script's output. If the return code of a script is '1' and <dir>_show_info is set to ``NO'', periodic(8) will mask the script's output. If the return code of a script is '2' and <dir>_show_badconfig is set to ``NO'', periodic(8) will mask the script's output. If these variables are set to neither ``YES'' nor ``NO'', they default to ``YES'', ``YES'' and ``NO'' respectively. Refer to the periodic(8) manual page for how script return codes are interpreted. The following variables are used by the standard scripts that reside in /etc/periodic/daily: daily_clean_tmps_enable (bool) Set to ``YES'' if you want to clear temporary directories daily. daily_clean_tmps_dirs (str) Set to the list of directories to clear if daily_clean_tmps_enable is set to ``YES''. daily_clean_tmps_days (num) When daily_clean_tmps_enable is set, this must also be set to the number of days old that a file's access and modification times must be before it is deleted. daily_clean_tmps_ignore (str) Set to the list of files that should not be deleted when daily_clean_tmps_enable is set to ``YES''. Wild card characters are permitted. daily_clean_tmps_verbose (bool) Set to ``YES'' if you want the removed files to be reported in your daily output. daily_clean_msgs_enable (bool) Set to ``YES'' if you wish old system messages to be purged. daily_clean_msgs_days (num) Set to the number of days that files must not have been modified before they are deleted. If this variable is left blank, the msgs(1) default is used. daily_clean_rwho_enable (bool) Set to ``YES'' if you wish old files in /var/who to be purged. daily_clean_rwho_days (num) Set to the number of days that files must not have been modified before they are deleted. daily_clean_rwho_verbose (bool) Set to ``YES'' if you want the removed files to be reported in your daily output. daily_accounting_enable (bool) Set to ``YES'' if you want to rotate your daily accounting files. No rotations are necessary unless accounting_enable is enabled in rc.conf(5). daily_accounting_compress (bool) Set to ``YES'' if you want your daily accounting files to be compressed using gzip(1). daily_accounting_save (num) When daily_accounting_enable is set, this may also be set to the number of daily accounting files that are to be saved. The default is ``3''. daily_accounting_flags (str) Set to the arguments to pass to the sa(8) utility (in addition to -s) when daily_accounting_enable is set to ``YES''. The default is -q. daily_status_disks_enable (bool) Set to ``YES'' if you want to run df(1) (with the arguments supplied in daily_status_disks_df_flags). daily_status_disks_df_flags (str) Set to the arguments for the df(1) utility when daily_status_disks_enable is set to ``YES''. daily_status_network_enable (bool) Set to ``YES'' if you want to run netstat -i. daily_status_network_usedns (bool) Set to ``YES'' if you want to run netstat(1) without the -n option (to do DNS lookups). daily_status_rwho_enable (bool) Set to ``YES'' if you want to run uptime(1) (or ruptime(1) if rwhod_enable is set to ``YES'' in /etc/rc.conf). daily_status_mailq_enable (bool) Set to ``YES'' if you want to run mailq(1). daily_status_mailq_shorten (bool) Set to ``YES'' if you want to shorten the mailq(1) output when daily_status_mailq_enable is set to ``YES''. daily_status_include_submit_mailq (bool) Set to ``YES'' if you also want to run mailq(1) on the submit mail queue when daily_status_mailq_enable is set to ``YES''. This may not work with MTAs other than sendmail(8). daily_local (str) Set to a list of extra scripts that should be run after all other daily scripts. All scripts must be absolute path names. The following variables are used by the standard scripts that reside in /etc/periodic/weekly: weekly_whatis_enable (bool) Set to ``YES'' if you want to run /usr/libexec/makewhatis.local. This script regenerates the database used by the apropos(1) command. weekly_local (str) Set to a list of extra scripts that should be run after all other weekly scripts. All scripts must be absolute path names. The following variables are used by the standard scripts that reside in /etc/periodic/monthly: monthly_accounting_enable (bool) Set to ``YES'' if you want to do login accounting using the ac(8) command. monthly_local (str) Set to a list of extra scripts that should be run after all other monthly scripts. All scripts must be absolute path names. FILES
/etc/defaults/periodic.conf The default configuration file. This file contains all default variables and values. /etc/periodic.conf The usual system specific variable override file. /etc/periodic.conf.local An additional override file, useful when /etc/periodic.conf is shared or distributed. SEE ALSO
apropos(1), calendar(1), df(1), diff(1), gzip(1), man(1), msgs(1), netstat(1), nice(1), ac(8), newsyslog(8), periodic(8), sendmail(8) HISTORY
The periodic.conf file appeared in FreeBSD 4.1. AUTHORS
Brian Somers <brian@Awfulhak.org> BSD
May 12, 2007 BSD
All times are GMT -4. The time now is 07:42 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy