Performance issue with 'grep' command for huge file size Post: 302574531

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep matched records from huge file

111111111100000000001111111111 123232323200000010001114545454 232435424200000000001232131212 342354234301000000002323423443 232435424200000000001232131212 2390898994200000000001238908092 This is the record format. From 11th position to 20th position in a record there are 0's occuring,and...

2. Shell Programming and Scripting

performance of shell script ( grep command)

Hi, I have to find out the run time for 40-45 different componets. These components writes in to a genreric log file in a single directory. eg. directory is LOG and the log file name format is generic_log_<process_id>_<date YY_MM_DD_HH_MM_SS>.log i am taking the run time using the time...

3. Shell Programming and Scripting

Implement in one line sed or awk having no delimiter and file size is huge

I have file which contains around 5000 lines. The lines are fixed legth but having no delimiter.Each line line contains nearly 3000 characters. I want to delete the lines a> if it starts with 1 and if 576th postion is a digit i,e 0-9 or b> if it starts with 0 or 9(i,e header and footer) ...

4. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Background ------------- The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files. File-1 ------ Contains 50,000 rows with 2 fields in each row, separated by pipe. Row structure is like Object_Id|Object_Name, as following: 111|XXX 222|YYY 333|ZZZ ...

5. Shell Programming and Scripting

FTP a huge Size file

Dear All, Good Evening!! I have a requirement to ftp a 220GB backup file to a remote backup server. I wrote a script for this purpose. But it takes more than 8 hours to transfer this file. Is there any other method to do it in less time??? Thanks in Advance!!! ---------- Post updated...

6. Shell Programming and Scripting

Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB).

Hi Experts, I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised...

7. Shell Programming and Scripting

Performance issue while using find command

Hi, I have created a shell script for Server Log Automation Process. I have used find xargs grep command to search the string. for Example, find -name | xargs grep "816995225" > test.txt . Here my problem is, We have lot of records and we want to grep the string...

8. UNIX for Advanced & Expert Users

Performance problem with removing duplicates in a huge file (50+ GB)

I'm trying to remove duplicate data from an input file with unsorted data which is of size >50GB and write the unique records to a new file. I'm trying and already tried out a variety of options posted in similar threads/forums. But no luck so far.. Any suggestions please ? Thanks !!

9. UNIX for Dummies Questions & Answers

What is the faster way to grep from huge file?

Hi All, I am new to this forum and this is my first post. My requirement is like to optimize the time taken to grep the file with 40000 lines. There are two files FILEA(40000 lines) FILEB(40000 lines). The requirement is like this, both the file will be in the format below...

10. Shell Programming and Scripting

Performance Issue for a file search command

Hi All, This query is regarding performance improvement of a command. I have a list of IDs in a file (say file1 with single ID column) and file2 has the data rows. I need to get the IDs from file1 and search in file2, matching rows from file2 should be written to a file3. For this...

LEARN ABOUT MOJAVE

git-ls-remote

GIT-LS-REMOTE(1)                                                    Git Manual                                                    GIT-LS-REMOTE(1)

NAME

       git-ls-remote - List references in a remote repository

SYNOPSIS

       git ls-remote [--heads] [--tags] [--refs] [--upload-pack=<exec>]
                     [-q | --quiet] [--exit-code] [--get-url]
                     [--symref] [<repository> [<refs>...]]

DESCRIPTION

       Displays references available in a remote repository along with the associated commit IDs.

OPTIONS

       -h, --heads, -t, --tags
           Limit to only refs/heads and refs/tags, respectively. These options are not mutually exclusive; when given both, references stored in
           refs/heads and refs/tags are displayed.

       --refs
           Do not show peeled tags or pseudorefs like HEAD in the output.

       -q, --quiet
           Do not print remote URL to stderr.

       --upload-pack=<exec>
           Specify the full path of git-upload-pack on the remote host. This allows listing references from repositories accessed via SSH and
           where the SSH daemon does not use the PATH configured by the user.

       --exit-code
           Exit with status "2" when no matching refs are found in the remote repository. Usually the command exits with status "0" to indicate it
           successfully talked with the remote repository, whether it found any matching refs.

       --get-url
           Expand the URL of the given remote repository taking into account any "url.<base>.insteadOf" config setting (See git-config(1)) and
           exit without talking to the remote.

       --symref
           In addition to the object pointed by it, show the underlying ref pointed by it when showing a symbolic ref. Currently, upload-pack only
           shows the symref HEAD, so it will be the only one shown by ls-remote.

       <repository>
           The "remote" repository to query. This parameter can be either a URL or the name of a remote (see the GIT URLS and REMOTES sections of
           git-fetch(1)).

       <refs>...
           When unspecified, all references, after filtering done with --heads and --tags, are shown. When <refs>... are specified, only
           references matching the given patterns are displayed.

EXAMPLES

           $ git ls-remote --tags ./.
           d6602ec5194c87b0fc87103ca4d67251c76f233a        refs/tags/v0.99
           f25a265a342aed6041ab0cc484224d9ca54b6f41        refs/tags/v0.99.1
           7ceca275d047c90c0c7d5afb13ab97efdf51bd6e        refs/tags/v0.99.3
           c5db5456ae3b0873fc659c19fafdde22313cc441        refs/tags/v0.99.2
           0918385dbd9656cab0d1d81ba7453d49bbc16250        refs/tags/junio-gpg-pub
           $ git ls-remote http://www.kernel.org/pub/scm/git/git.git master pu rc
           5fe978a5381f1fbad26a80e682ddd2a401966740        refs/heads/master
           c781a84b5204fb294c9ccc79f8b3baceeb32c061        refs/heads/pu
           $ git remote add korg http://www.kernel.org/pub/scm/git/git.git
           $ git ls-remote --tags korg v*
           d6602ec5194c87b0fc87103ca4d67251c76f233a        refs/tags/v0.99
           f25a265a342aed6041ab0cc484224d9ca54b6f41        refs/tags/v0.99.1
           c5db5456ae3b0873fc659c19fafdde22313cc441        refs/tags/v0.99.2
           7ceca275d047c90c0c7d5afb13ab97efdf51bd6e        refs/tags/v0.99.3

GIT

       Part of the git(1) suite

Git 2.17.1                                                          10/05/2018                                                    GIT-LS-REMOTE(1)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep matched records from huge file

Discussion started by: mjkreddy

2. Shell Programming and Scripting

performance of shell script ( grep command)

Discussion started by: vikash_k

3. Shell Programming and Scripting

Implement in one line sed or awk having no delimiter and file size is huge

Discussion started by: millan

4. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Discussion started by: Souvik