Sponsored Content
Top Forums Shell Programming and Scripting Script to delete rows in a file Post 302875785 by Akshay Hegde on Wednesday 20th of November 2013 11:23:53 AM
Old 11-20-2013
Quote:
Originally Posted by alok2082
Hi All,

I am new to UNIX . Please help me in writing code to delete all records from the file where all columns after cloumn 5 in file is either 0, #MI or NULL.
Initial 5 columns are string

e.g.

"alsod" "1FEV2" "wjwroe" " wsse" "hd3" 1 2 34 #Mi
"malasl" "wses" "trwwwe" " wsse" "hd3" 1 2 0 #Mi
"alsod" "1FEV2" "asd" " wsse" "hd3" #Mi #Mi
"alsod" "1FEV2" "asd" " wsse" "hd3" 0 0 0
"alsod" "1FEV2" "asd" " wsse" "hd3" 1 2 3 4 5


expected output is

"alsod" "1FEV2" "wjwroe" " wsse" "hd3" 1 2 34 #Mi
"malasl" "wses" "trwwwe" " wsse" "hd3" 1 2 0 #Mi
"alsod" "1FEV2" "asd" " wsse" "hd3" 1 2 3 4 5


The file is around 1-2 GB large.
I have written a code but it is taking 45-50 min to execute the script.

grep -EHv ([1-9]/s) file.txt > file2.txt

can some one please suggest alternate code where we are selectively deleting the records containing 0/#Mi/NULL after column 5

Thanks
Try :

Code:
$ cat file
"alsod" "1FEV2" "wjwroe" " wsse" "hd3" 1 2 34 #Mi
"malasl" "wses" "trwwwe" " wsse" "hd3" 1 2 0 #Mi
"alsod" "1FEV2" "asd" " wsse" "hd3" #Mi #Mi
"alsod" "1FEV2" "asd" " wsse" "hd3" 0 0 0
"alsod" "1FEV2" "asd" " wsse" "hd3" 1 2 3 4 5

$ awk '$7~/[0-9]/ && $7 !=0' file
"alsod" "1FEV2" "wjwroe" " wsse" "hd3" 1 2 34 #Mi
"malasl" "wses" "trwwwe" " wsse" "hd3" 1 2 0 #Mi
"alsod" "1FEV2" "asd" " wsse" "hd3" 1 2 3 4 5

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete repeated rows from a file

Hi everybody: Could anybody tell me how I can delete repeated rows from a file?, this is, for exemple I have a file like this: 0.490 958.73 281.85 6.67985 0.002481 0.490 954.833 283.991 8.73019 0.002471 0.590 950.504 286.241 6.61451 0.002461 0.690 939.323 286.112 6.16451 0.00246 0.790... (8 Replies)
Discussion started by: tonet
8 Replies

2. Shell Programming and Scripting

How to delete particular rows from a file

Hi I have a file having 1000 rows. Now I would like to remove 10 rows from it. Plz give me the script. Eg: input file like 4 1 4500.0 1 5 1 1.0 30 6 1 1.0 4500 7 1 4.0 730 7 2 500000.0 730 8 1 785460.0 45 8 7 94255.0 30 9 1 31800.0 30 9 4 36000.0 30 10 1 15000.0 30... (5 Replies)
Discussion started by: suresh3566
5 Replies

3. Shell Programming and Scripting

how to delete duplicate rows in a file

I have a file content like below. "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","","" "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","","" "0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""... (5 Replies)
Discussion started by: vamshikrishnab
5 Replies

4. Shell Programming and Scripting

[HELP] - Delete rows on a CSV file

Hello to all members, I am very new in unix stuff (shell scripting), but a want to learn a lot. I am a ex windows user but now i am absolutely Linux super user... :D So i am tryng to made a function to do this: I have two csv files only with numbers, the first one a have: 1 2 3 4 5... (6 Replies)
Discussion started by: Sadarrab
6 Replies

5. Shell Programming and Scripting

delete rows in a file based on the rows of another file

I need to delete rows based on the number of lines in a different file, I have a piece of code with me working but when I merge with my C application, it doesnt work. sed '1,'\"`wc -l < /tmp/fileyyyy`\"'d' /tmp/fileA > /tmp/filexxxx Can anyone give me an alternate solution for the above (2 Replies)
Discussion started by: Muthuraj K
2 Replies

6. UNIX for Advanced & Expert Users

Delete rows from a file...!!

Say i have a file with X rows and Y columns....i see that in some of the rows,some columns are blank (no value set)...i wish to delete such rows....how can it be done? e.g 181766 100 2009-06-04 184443 2009-06-04 10962 151 2009-06-04 161 2009-06-04... (7 Replies)
Discussion started by: ak835
7 Replies

7. Shell Programming and Scripting

Delete rows in text file

Hi I do have a text file with 1000's of lines with 1 row and column with a specific pattern. 1102 1 1 1 1 1234 1 1 1 1 1009 1 1 1 1 1056 1 (3 Replies)
Discussion started by: Lucky Ali
3 Replies

8. Shell Programming and Scripting

Delete rows from big file

Hi all, I have a big file (about 6 millions rows) and I have to delete same occurrences, stored in a small file (about 9000 rews). I have tried this: while read line do grep -v $line big_file > ok_file.tmp mv ok_file.tmp big_file done < small_file It works, but is very slow. How... (2 Replies)
Discussion started by: Tibbeche
2 Replies

9. Shell Programming and Scripting

Delete unique rows - optimize script

Hi all, I have the following input - the unique row key is 1st column cat file.txt A response C request C response D request C request C response E request The desired output should be C request (7 Replies)
Discussion started by: varu0612
7 Replies

10. UNIX and Linux Applications

Script to delete few rows from a file and then update header

HJKL1Name00014300010800000418828124201 L201207022012070228XAM 00000000031795404 001372339540000000000000000000000 COOLTV KEYA Zx00 xI-50352202553 00000000 00000000 G000000000000 00000000 ... (10 Replies)
Discussion started by: mirwasim
10 Replies
MONITOR(8)						      System Manager's Manual							MONITOR(8)

NAME
monitor, edparams - load and start Minix, modify boot parameters SYNOPSIS
edparams device [command ...] boot.com virdisk DESCRIPTION
This text describes the Boot Monitor, a boot time interactive program designed not only to load and start Minix, its most important task, but to also provide an easy to use interface to configure Minix and to boot other operating systems. The monitor is controlled with an environment that is modeled after the Bourne shell. This environment is filled at startup with default values that depend on the machine the monitor is running on and the environment settings saved into the boot parameters sector (the second sector on a device). When the environment is loaded, the monitor executes the function named main, which by default starts a simple menu. The environment can be manipulated at boot time from the monitor prompt, but may also be edited using edparams on a given device. Edparams simulates the monitor as much as it can, echoing commands it can't execute between brackets. It can also be used in Makefiles and scripts by giving it commands as arguments. The MS-DOS version of the monitor, usually named boot.com under DOS, boots Minix from a "DOS virtual disk". (See below.) COMMANDS
The monitor is best described by the commands you can type to the '>' prompt. This is known as the "monitor mode". You can enter this mode by hitting the Escape key. These are the monitor commands: name = [device] value Set environment variable. Changes the value of name to value. The optional word device marks name as being subject to device translation. (See the section on devices.) These (name, value) pairs are passed to the kernel who uses them to configure itself. These variables are passed by default: rootdev This is the device used as your root device. It is by default set to ram, which means that the device specified by ramim- agedev will be loaded into the RAM disk and used as root. If you change this variable then a physical device will be used as root, and the RAM disk will be uninitialized and have the size specified by ramsize. ramimagedev Describes the device to use to initialize the RAM disk if rootdev is set to ram. It's by default set to bootdev, a special name for the device the monitor booted from. ramsize The size of the RAM disk. If the RAM disk is used for the root file system then the root file system is stretched out to ramsize if possible. processor Set by default to 86, 186, 286, 386, 486, ... depending on the hardware you have. You can set it to a smaller value to test your kernel in a more limited environment. bus The type of system bus, either xt, at or mca. This answers basic questions like: "How many interrupt controllers and how to initialize?" Or: "Does the keyboard have LEDs?" memsize Kilobytes of conventional memory. This is the amount of RAM within the first megabyte. emssize Kilobytes of extended memory. video Describes capabilities of the VDU: mda, cga, ega or vga. chrome Either color or mono. console If set to a hexadecimal value makes the monitor set the BIOS video mode to this value. This allows the use of video modes with more rows or colums than the standard 80x25 mode. The kernel must of course be able to handle a nonstandard mode. More parameters may follow the mode number. Warning: Not all monitors can handle all of the modes, some may generate frequencies that can damage your monitor. Read the manual of card and monitor for details. Two variables are only used by the monitor, even though they are passed to the kernel too: image The name of the file containing the kernel image, by default minix. If it refers to a directory however then the newest file inside the directory is chosen to be the kernel image. The names inside /minix/ are best set to the Minix version you are using, which looks good when the monitor prints its name. Rules for pretty printing image names: A '/' or '_' is changed to a space. The first letter is changed from lowercase to uppercase. An 'r' if followed by a digit changes to " revision ". label If set then only processes marked with this label or without a label are loaded from the image. Installboot -boot will create functions to select images and labels. These functions will set label and image and echo what you selected. The two numbers separated by a colon used as an image name tell the starting sector and sector count of the image on disk. name() { ... } Define function. Functions may be used to bundle a set of commands, so that you can easily boot Minix with a different set of parameters then normal. E.g. ram() { rootdev=ram; boot } will allow you to run Minix with the root device on RAM for a change, if you normally use a real device as root. The only pre-set function is main with default value menu, which is the default command executed by the monitor. You can use newlines after the ')' token, the monitor will then use a '+' prompt and ask for the rest. name(key) { ... } Define kernel selecting function. The menu command uses functions like these to add menu entries to select a different kernel from a boot disk. Installboot -boot produces these functions when the images are labeled. The label AT would give: AT(a) {label=AT;image=42:626;echo AT kernel selected;menu} With the menu option: a Select AT kernel Typing a will then execute the AT function above. name(key,text) { ... } User defined menu option. This variant may be used to make any menu entry you like: dos(d,Boot MS-DOS) { boot hd1 } Text may be anything, even parentheses if they match. name Call function. If name is a user defined function then its value is expanded and executed in place of name. Try a recursive one like 'rec() {rec;xx}' one day. You can see the monitor run out of space with nice messages about using chmem(1) to increase it's heap. boot [-opts] boot device Boot Minix or another O.S. Without an argument, boot will load and execute the Minix image named by the image variable. With options the variable bootopts is first set to -opts before Minix is started, and unset when Minix returns. With a device argument, boot loads the boot sector of device into memory and jumps to it, starting another operating system. You would normally use partitions on the first hard disk for this command (hd[1-4]), using hd0 will also work (choosing the active partition). One can also boot devices on the second hard disk (hd[5-9]) if the bootstrap writer did not hardwire the disk number to disk 0. Some Operating Systems can only be booted from the active partition, if you use a '*', e.g. boot *hd3, then partition 3 is first made active. You'll then need to use installboot -master with a fix key to forcefully boot the Minix partition at startup. delay [msec] Delay (500 msec default). Fast booting speed was one of the objectives when this program was created, so a hard disk boot usually takes only a fraction of a second. If you need some time (to hit Escape, or stare at the numbers) you can use delay to make the monitor pause for a specified amount of time. To specify a delay just before Minix is started, you can set the variable delay to a number of milliseconds. Exam- ple: main() {delay 250; delay=500; boot} Look at this carefully, 'delay 250' means: "wait 1/4 sec now!", while 'delay=500' means: "wait 1/2 sec after loading Minix". If you use delay=swap then the monitor will wait until you have inserted a root diskette and typed RETURN. echo word ... Print these words. Used to tell you that you just selected image X. ls [directory] List contents of a directory. Useful when looking for kernel images. menu Menu driven startup. This command allows you to execute functions defined with a key. If no menu functions have been defined then menu will use this one hidden built-in function: *(=,Start Minix) { boot } Kernel selecting functions only add new options to this set, but if you define a two argument function yourself then the above one is no longer shown, allowing you to customize the menu completely. Your first function definition should therefore be one that starts Minix. Menu entries are shown in the same order as set shows them. If you don't like the order then you have to unset the functions and retype them in the proper order. If you type a key then a scheduled trap is killed and the appropriate menu function is executed. If you need more time to choose then hit the spacebar. A key not on the menu also kills a trap, but does nothing more. save Save environment. This will save all the environment variables and functions with nondefault values to the parameter sector (the second sector on the boot device), so they are automatically set the next time you boot the monitor. set Show environment. Show the current values of the environment variables and functions. Default values are shown between parentheses to distinguish them from values that were explicitly set. trap msec command Schedule command. Schedules a command to be executed after msec milliseconds. Only the monitor mode cannot be interrupted, a scheduled trap is killed when the prompt is printed. Example: main() {trap 10000 boot; menu} This gives you 10 seconds to choose a menu option before Minix is booted. unset name ... Unset environment variables. Removes the named variables and functions from the environment, and sets special variables back to their default values. This is also the only way to remove the "device name translation" property from a variable. exit Exit the monitor Reboot the machine, exit to Minix or exit to DOS as appropriate. DEVICES
The Minix kernel can't do anything with device names, so they have to be translated to device numbers before they are passed to the kernel. This number is found under the st_rdev field (see stat(2)) of the file on the boot file system. The monitor will look for the device file with the working directory set to '/dev'. If it can't find the device name then it will translate names like 'ram', 'fd1', 'hd6', 'hd3a', and 'sd2' to what it itself thinks the numbers should be. The special name bootdev is translated to the name of the device booted from, like 'fd0', or 'hd3', and then searched for in /dev. Bootdev can't be translated to a device other then the fd or hd devices, so SCSI devices for instance must be named explicitly. EXTENSIONS
A few extensions have been made to this program for kernel hackers. They may be triggered by setting bits in the flags word in the kernel startup code (the mpx file.) The flag bits are: 0x0001 Call kernel in 386 mode. 0x0002 Do not make space for the bss areas of processes other then the kernel. 0x0004 Use the stack size set by chmem(1). 0x0008 Load MM, FS, etc. into extended memory. 0x0010 No need to patch process sizes into the kernel. 0x0020 The kernel can return to the monitor on halt or reboot. MS-DOS MONITOR Minix-vmd has a version of the monitor that runs under MS-DOS to boot a "DOS virtual disk". It is a simple COM program that interprets an MS-DOS file as a disk, loads a Minix kernel from the active partition in the same way as the BIOS based monitor, and executes it to start Minix. All the monitor commands function in the same way, except for the boot command, it can only load Minix. The memory that MS-DOS has in use is copied out of the way when Minix takes control, and is put back in place when Minix exits. This memory shuffling also happens when the BIOS disk driver makes BIOS calls, slowing things to a crawl. It is better to use a Minix driver. The MS-DOS monitor does not work if there is a memory manager active that runs in 386 protected mode, like EMM386. SEE ALSO
chmem(1), stat(2), installboot(8), usage(8), boot(8). BUGS
The delay command will hang forever on the original IBM PC (not the XT!). Not that it matters, as everything takes forever on that box. Reading the first sector to boot a floppy (e.g. boot fd1), is done using whatever floppy parameters boot currently has available. This will probably always work. The two forms of delay are a crock. The word emssize comes from EMS, that has to do with expanded memory, not extended memory. ACKNOWLEDGMENTS
Guy Helmer, for the floppy sensing code that somehow disappeared into the boot block. Earl Chew, for the inspiration his ShoeLace package provided, unless he wants to file a "look and feel" suit against me, then I will say I modeled it after the SunOS ROM boot monitor, which is also true. AUTHOR
Kees J. Bot (kjb@cs.vu.nl) MONITOR(8)
All times are GMT -4. The time now is 12:42 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy