Sponsored Content
Top Forums Shell Programming and Scripting Removing Lines if value exist in first file Post 302348776 by Scrutinizer on Saturday 29th of August 2009 04:22:03 PM
Old 08-29-2009
Quote:
Originally Posted by Azhrei
No way would I use a shell for that job! The following Perl script is probably a hundred times faster and more efficient!
While I agree that Perl is usually well suited for this type of application, I do not think this generalization is accurate. The shell scripts above are fine but there is room for some significant speed optimizations. If we use ksh (ksh93s+) instead of bash and a method that resembles the one in your Perl script, I think there would not be a real big difference in speed.

filter.ksh93
Code:
#!/usr/bin/ksh
typeset -A EXCLUDED
EXCLUDE_LIST=$(< $1)
INFILE=$2
for excl in $EXCLUDE_LIST; do
  EXCLUDED[$excl]=1
done
IFS=","
while read a b c id d; do
  if [[ ${EXCLUDED[$id]} -ne 1 ]]; then
    echo "${a},${b},${c},${id},${d}"
  fi
done < $INFILE

Code:
./filter.ksh93 excludes infile > outfile


Last edited by Scrutinizer; 08-29-2009 at 05:32 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing lines from a file

Hello i have 2 files file1 and file2 as shown below file1 110010000000206|567810008161509 110010000000207|567810072227627 110010000000208|567811368851555 110010000000209|567811422513652 110010000000210|567812130217683 110010000000211|567813220211182 110010000000212|567813449322589... (4 Replies)
Discussion started by: PradeepRed
4 Replies

2. Shell Programming and Scripting

Removing lines within a file

Hi There, I've written a script that processes a data file on our system. Basically the script reads a post code from a list file, looks in the data file for the first occurrence (using grep) and reads the line number. It then tails the data file, with the line number just read, and outputs to a... (3 Replies)
Discussion started by: tookers
3 Replies

3. UNIX for Dummies Questions & Answers

Removing lines from a file

I'm trying to find a command which will allow me to remove a range of lines (2-4) from a .dat file from the command line without opening the file. Someone mentioned using the ex command? Does anyone have any ideas? thanks (6 Replies)
Discussion started by: computersaysno
6 Replies

4. Shell Programming and Scripting

Removing the first and last lines in a file

Hi Gurus, I'm a little new to UNIX. How can I do remove the first and last line in a file? Say, supppose I have a file as below: Code: 1DMA 400002BARRIE 401002CALGARY/LETHBRI 402002CARLETON 500001PORTLAND-AUBRN 501001NEW YORK, NY 502001BINGHAMTON, NY ... (2 Replies)
Discussion started by: naveendronavall
2 Replies

5. UNIX for Dummies Questions & Answers

removing several lines from a file

Hi folks, I have a long string of DNA sequences, and I need to remove several lines, as well as the line directly following them. For example, here is a sample of my starting material: >548::GY31UMJ02DLYEH rank=0007170 x=1363.5 y=471.0 length=478... (1 Reply)
Discussion started by: kkohl78
1 Replies

6. UNIX for Dummies Questions & Answers

Removing a user that doesnt exist from a group

Hi there, normally if I want to remove a user tht I have added to a specific group, i would do the following this is what my group2 looks like # grep group2 /etc/group group2:x:7777:user2,user1,user4 user1 has been defined in a few groups # id -nG user1 group1 group2 group3 So... (3 Replies)
Discussion started by: rethink
3 Replies

7. Shell Programming and Scripting

Deleting lines of a file if they exist in another file

I have a reference file that needs to remain static and another file that may or may not have duplicate rows that match the reference file. I need help with a command that will delete any duplicate rows from the second file while leaving reference file intact For example reference file would... (4 Replies)
Discussion started by: bjdamon
4 Replies

8. Shell Programming and Scripting

Remove lines from one file that exist in another file

Hello Everyone, I'm currently have a requirement where I've generated a list of files with specific attributes and I need to know what lines are similar between the two files. For example: -File 1- line1 line2 line3 -File 2- line1 line2 line4 line5 -Desires Output- line1 line2... (5 Replies)
Discussion started by: omnivir
5 Replies

9. Shell Programming and Scripting

Removing lines from a file

Hi, I have a linux server that was hacked and I have a bunch of files that sporadically contain the following lines through out the file: <?php eval(base64_decode("Xxxxxxxxxxxxxx/xxxxxxxx")); I did't put the exact lines of the file in this post. The "Xxxx" are random letters/numbers.... (8 Replies)
Discussion started by: nck
8 Replies

10. Shell Programming and Scripting

Removing lines from a file

I have a file `/tmp/wrk` containing filenames with paths. I want to remove filenames from this file, for example remove all filenames containing alja cagr cavt clta cmdo or corl remove all filenames containing data for days in region `d.2016.001` to `d.2016.207` remove all filenames... (10 Replies)
Discussion started by: kristinu
10 Replies
shell_builtins(1)						   User Commands						 shell_builtins(1)

NAME
shell_builtins, case, for, foreach, function, if, repeat, select, switch, until, while - shell command interpreter built-in commands DESCRIPTION
The shell command interpreters csh(1), ksh(1), ksh93(1), and sh(1) have special built-in commands. The commands case, for, foreach, func- tion, if, repeat, select, switch, until, and while are commands in the syntax recognized by the shells. They are described in the Commands section of the manual pages of the respective shells. In ksh93(1), fc, hash, stop, suspend, times, and type are aliases by default. In ksh93, the following built-ins are bound to the /bin pathname by default and are invoked if the pathname search encounters an executable command of that name in the /bin or /usr/bin directory: cat, chown, getconf, head, mkdir, rmdir, tee, uniq, and wc. The remaining commands listed in the following table are built into the shells for reasons such as efficiency or data sharing between com- mand invocations. They are described on their respective manual pages. Command Shell ----------------------------------------------------------- ++**alias csh, ksh, ksh93 bg csh, ksh, ksh93, sh +*break csh, ksh, ksh93, sh builtin ksh93 case csh, ksh, ksh93, sh cat ksh93 cd csh, ksh, ksh93, sh chdir csh, sh chown ksh93 command ksh93 +*continue csh, ksh, ksh93, sh dirs csh disown ksh93 echo csh, ksh, ksh93, sh +*eval csh, ksh, ksh93, sh +*exec csh, ksh, ksh93, sh +*exit csh, ksh, ksh93, sh ++**export ksh, ksh93, sh false ksh, ksh93 fc ksh, ksh93 fg csh, ksh, ksh93, sh for ksh, ksh93, sh foreach csh function ksh, ksh93 getconf ksh93 getopts ksh, ksh93, sh glob csh goto csh hash ksh, ksh93, sh hashstat csh head ksh93 hist ksh93 history csh if csh, ksh, ksh93, sh jobs csh, ksh, ksh93, sh kill csh, ksh, ksh93, sh let ksh, ksh93, limit csh login csh, ksh, ksh93, sh logout csh mkdir ksh93 nice csh +*newgrp ksh, ksh93, sh nohup csh notify csh onintr csh popd csh print ksh, ksh93 printf ksh93 pushd csh pwd ksh, ksh93, sh read ksh, ksh93, sh ++**readonly ksh, ksh93, sh rehash csh repeat csh +*return ksh, ksh93, sh select ksh, ksh93 +set csh, ksh, ksh93, sh setenv csh shift csh, ksh, ksh93, sh sleep ksh93 source csh stop csh, ksh, ksh93, sh suspend csh, ksh, sh switch csh tee ksh93 test ksh, ksh93, sh time csh *times ksh, ksh93, sh *+trap ksh, ksh93, sh true ksh, ksh93 type ksh, ksh93, sh ++**typeset ksh, ksh93 ulimit ksh, ksh93, sh umask csh, ksh, ksh93, sh +unalias csh, ksh, ksh93 unhash csh uniq ksh93 unlimit csh +unset csh, ksh, ksh93, sh unsetenv csh until ksh, ksh93, sh *wait csh, ksh, ksh93, sh whence ksh, ksh93 while csh, ksh, ksh93, sh Bourne Shell, sh, Special Commands Input/output redirection is now permitted for these commands. File descriptor 1 is the default output location. When Job Control is enabled, additional Special Commands are added to the shell's environment. In addition to these built-in reserved command words, sh also uses: : No effect; the command does nothing. A zero exit code is returned. .filename Read and execute commands from filename and return. The search path specified by PATH is used to find the directory contain- ing filename. C shell, csh Built-in commands are executed within the C shell. If a built-in command occurs as any component of a pipeline except the last, it is exe- cuted in a subshell. In addition to these built-in reserved command words, csh also uses: : Null command. This command is interpreted, but performs no action. Korn Shell, ksh, Special Commands Input/Output redirection is permitted. Unless otherwise indicated, the output is written on file descriptor 1 and the exit status, when there is no syntax error, is zero. Commands that are preceded by one or two * (asterisks) are treated specially in the following ways: 1. Variable assignment lists preceding the command remain in effect when the command completes. 2. I/O redirections are processed after variable assignments. 3. Errors cause a script that contains them to abort. 4. Words, following a command preceded by ** that are in the format of a variable assignment, are expanded with the same rules as a variable assignment. This means that tilde substitution is performed after the = sign and word splitting and file name genera- tion are not performed. In addition to these built-in reserved command words, ksh also uses: * : [ arg ... ] The command only expands parameters. * .file [ arg ... ] Read the complete file then execute the commands. The commands are executed in the current shell environment. The search path specified by PATH is used to find the directory containing file. If any arguments arg are specified, they become the positional parameters. Otherwise, the positional parameters are unchanged. The exit status is the exit status of the last command executed. the loop termination test. Korn Shell, ksh93, Special Commands Input/Output redirection is permitted. Unless otherwise indicated, the output is written on file descriptor 1 and the exit status, when there is no syntax error, is zero. Except for :, true, false, echo, newgrp, and login, all built-in commands accept -- to indicate end of options. They also interpret the option --man as a request to display the manual page onto standard error and -? as a help request which prints a usage message on standard error. Commands that are preceded by one or two + are treated specially in the following ways: 1. Variable assignment lists preceding the command remain in effect when the command completes. 2. I/O redirections are processed after variable assignments. 3. Errors cause a script that contains them to abort. 4. They are not valid function names. 5. Words, following a command preceded by ++ that are in the format of a variable assignment, are expanded with the same rules as a variable assignment. This means that tilde substitution is performed after the = sign and field splitting and file name genera- tion are not performed. In addition to these built-in reserved command words, ksh93 also uses: : [ arg ... ] The command only expands parameters. .name [ arg ... ] If name is a function defined with the function name reserved word syntax, the function is executed in the cur- rent environment (as if it had been defined with the name() syntax.) Otherwise if name refers to a file, the file is read in its entirety and the commands are executed in the current shell environment. The search path specified by PATH is used to find the directory containing the file. If any arguments arg are specified, they become the positional parameters while processing the . command and the original positional parameters are restored upon completion. Otherwise the positional parameters are unchanged. The exit status is the exit status of the last command executed. SEE ALSO
Intro(1), alias(1), break(1), builtin(1), cd(1), chmod(1), csh(1), disown(1), echo(1), exec(1), exit(1), find(1), getoptcvt(1), getopts(1), glob(1), hash(1), history(1), jobs(1), kill(1), ksh(1), ksh93(1), let(1), limit(1), login(1), logout(1), newgrp(1), nice(1), nohup(1), print(1), printf(1), pwd(1), read(1), readonly(1), set(1), sh(1), shift(1), sleep(1), suspend(1), test(1)test(1), test(1B), time(1), times(1), trap(1), typeset(1), umask(1), wait(1), chdir(2), chmod(2), creat(2), umask(2), getopt(3C), profile(4), environ(5) SunOS 5.11 20 Nov 2007 shell_builtins(1)
All times are GMT -4. The time now is 05:08 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy