Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Deleting duplicated chunks in a file using awk/sed
Top Forums Shell Programming and Scripting Deleting duplicated chunks in a file using awk/sed Post 302974140 by Don Cragun on Thursday 26th of May 2016 01:07:01 AM
Old 05-26-2016
Maybe something more like:
Code:
awk '
{	for(i = 1; i <= NF; i++) {
		f[NR % 4, i] = $i
	}
}
!(NR % 4) {
	ocnt = 0
	for(i = 1; i <= NF; i++)
		if(!(f[3, i] in of)) {
			of[f[3, i]]
			spot[++ocnt] = i
		}
	for(i = 1; i <= ocnt; i++)
		for(j = 1; j <= 4; j++) {
			ol[j] = ol[j] f[j % 4, spot[i]] ((i == ocnt) ? "" : "\t")
		}
	for(i = 1; i <= 4; i++) {
		print ol[i]
		delete ol[i]
	}
	for(i in of)
		delete of[i]
}' input.txt

would work better for you. This uses a single tab character as the output field separator instead of a seemingly random number of spaces (but you can easily change it to a fixed number of spaces if you want to).

From the sed command you're using, I assume that you're not running this on a Solaris system, but if someone else wants to try the above code on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.
This User Gave Thanks to Don Cragun For This Post:
jypark22
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

using sed to get rid of duplicated columns...

I can not figure out this one, so I turn to unix.com for help, I have a file, in which there are some lines containing continuously duplicate columns, like the following adb abc abc asd adfj 123 123 123 345 234 444 444 444 444 444 23 and the output I want is adb abc asd adfj 123 345... (5 Replies)
Discussion started by: fedora
5 Replies

2. Shell Programming and Scripting

Deleting Doubled Block with sed or awk

hi there, i have a text file like that one: I like to delete the second block with the Start and End Line! Does anyone have a idea? Thanks for your help, Roland (4 Replies)
Discussion started by: rolandh
4 Replies

3. Shell Programming and Scripting

Deleting a line from a file with sed and awk?

cat file.txt fvnuiehuewf ruevhxncvkjrh zxjvurhfuwe jkhvBEGINvfnvf ijrgioe Trying to delete a line that has the pattern "BEGIN" cat sedtest filename=file.txt pattern=BEGIN sed "/^$pattern/d" "$filename" (9 Replies)
Discussion started by: cola
9 Replies

4. Shell Programming and Scripting

Deleting the first column with sed,awk or perl

336 brtr 256 hello Output: brtr hello How can i do this with sed,awk or perl? (5 Replies)
Discussion started by: cola
5 Replies

5. Shell Programming and Scripting

Deleting characters with sed,perl,awk

Input: :: gstreamer :: xine-lib :: xine-lib-extras Output should be: gstreamer xine-lib xine-lib-extras How can it be done with sed or perl? (12 Replies)
Discussion started by: cola
12 Replies

6. Shell Programming and Scripting

Can I use a shell script for deleting chunks from a watch folder?

Hello I have a unique problem of needing to delete large files slowly off of an XSan. I was wondering if there is a script I could use to delete 100gb chunks of files and folders that get placed in to a watch folder, slowly so as not to disrupt the other users. I would like to use Automator in... (0 Replies)
Discussion started by: ajsoto
0 Replies

7. Shell Programming and Scripting

deleting lines between patterns using sed or awk

hi, Here is excerpt from my xml file <!-- The custom module to do the authentication for LDAP --> </login-module> <login-module code="com.nlayers.seneca.security.LdapLogin" flag="sufficient"> <module-option... (1 Reply)
Discussion started by: sunrexstar
1 Replies

8. UNIX for Dummies Questions & Answers

Awk: Print out overlapping chunks of file - rows 0-20,10-30,20-40 etc.

First time poster, but the forum has saved my bacon more times than... Lots. Anyway, I have a text file, and wanted to use Awk (or any other sensible program) to print out overlapping sections, or arbitrary length. To describe by example, for file 1 2 3 4 5 etc... I want the out put... (3 Replies)
Discussion started by: matfald
3 Replies

9. Shell Programming and Scripting

awk for splitting file in constant chunks

Hi gurus, I wanted to split main file in 20 files with 2500 lines in each file. My main file conatins total 2500*20 lines. Following awk I made, but it is breaking with error. awk '{ for (i = 1; i <= 20; i++) { starts=2500*$i-1; ends=2500*$i; NR>=starts && NR<=ends {f=My$i".txt"; print >> f;... (10 Replies)
Discussion started by: mukesh.lalwani
10 Replies

10. Shell Programming and Scripting

Deleting lines containing duplicated strings

Dear all, I always appreciate your help. I would like to delete lines containing duplicated strings in the second column. test.txt 658 invert_d2e_q_reg_0_/Qalu_ecl_zlow_e 0.825692 659 invert_d2e_q_reg_0_/Qalu_byp_rd_data_e 0.825692 660 invert_d2e_q_reg_0_/Qalu_byp_rd_data_e 0.825692... (1 Reply)
Discussion started by: jypark22
1 Replies

LEARN ABOUT DEBIAN

pyp

PYP(1)							      General Commands Manual							    PYP(1)

NAME

       pyp - The Pyed Piper: A Modern Python Alternative to awk, sed and Other Unix Text Manipulation Utilities

SYNOPSIS

       pyp [options] files ...

DESCRIPTION

       pyp,  the  Pyed Piper, is a command line tool for text manipulation. It is similar to awk and sed in functionality, but its subcommands are
       Python based, and thus more familiar to many programmers.

       It can operate both on a per-line base and on the complete input stream.  Different features can be pipelined in a single command by  using
       the pipe character familiar from shell commands.

       pyp  backs  up  its  input  for reruns with modified commands, and can save commands as macros. On the downside, the rerun feature makes it
       unsuitable for continuous pipe operation.

OPTIONS

       These programs follow the usual GNU command line syntax, with long options starting with  two  dashes  (`-').   A  summary  of  options	is
       included below.	For a complete description, use --manual.

       -h, --help
	      Show this help message and exit.

       -m, --manual
	      Prints out extended help.

       -l, --macro_list
	      Lists all available macros.

       -s MACRO_SAVE_NAME, --macro_save=MACRO_SAVE_NAME
	      Saves current command as macro. use "#" for adding
	      comments	EXAMPLE:
	      pyp -s "great_macro # prints first letter" "p[1]".

       -f MACRO_FIND_NAME, --macro_find=MACRO_FIND_NAME
	      Searches for macros with keyword or user name.

       -d MACRO_DELETE_NAME, --macro_delete=MACRO_DELETE_NAME
	      Deletes specified public macro.

       -g, --macro_group
	      Specify group macros for save and delete; default is user.

       -t TEXT_FILE, --text_file=TEXT_FILE
	      Specify text file to load. For advanced users,
	      you should typically cat a file into pyp.

       -x, --execute
	      Execute all commands.

       -c, --turn_off_color
	      Prints raw, uncolored output.

       -u, --unmodified_config
	      Prints out generic PypCustom.py config file.

       -b BLANK_INPUTS, --blank_inputs=BLANK_INPUTS
	      Generate this number of blank input lines; useful for
	      generating numbered lists with variable 'n'.

       -n, --no_input
	      Use with command that generates output with no input;
	      same as --dummy_input 1.

       -k, --keep_false
	      Print blank lines for lines that test as False.
	      default is to filter out False lines from the output.

       -r, --rerun
	      Rerun based on automatically cached data from the last run.
	      Use this after executing "pyp", pasting input into the shell,
	      and hitting CTRL-D.

SEE ALSO

       awk(1), grep(1), sed(1).

AUTHOR

       pyp was written by Toby Rosen <tobyrosen@gmail.com>.

       This manual page was written by Khalid El Fathi <khalid@elfathi.fr>, for the Debian project (and may be used by others).

								  March 19, 2012							    PYP(1)
All times are GMT -4. The time now is 02:41 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy