Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: remove chunks of text from file
Top Forums Shell Programming and Scripting remove chunks of text from file Post 302339139 by staze on Thursday 30th of July 2009 01:59:53 PM
Old 07-30-2009
Wow, great.

So, the first one to remove the line breaks works great. The second one, which removes the chunk, doesn't seem to work. The resulting output is significantly truncated (it seems to be removing a very large portion of the file).

Before the second command, the file is 181095 lines. After the sed command, it's 2652. Since there are 6411 users in the ldif file, theoretically, that sed command should only be removing one line per user, so the output should be about 175k lines.

sed should stop it's pattern match at the first "stop" it sees. Could that not be working?

For example...

Here's the output after taking my "input" and running it through the "de-space/de-return sed"
Code:
# username, users, example.com
dn: uid=username,cn=users,dc=example,dc=com
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
objectClass: apple-user
objectClass: extensibleObject
objectClass: organizationalPerson
objectClass: top
objectClass: person
apple-generateduid: 53CA02D7-B116-4461-B220-E3FC0B15964A
apple-mcxflags:: PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPCFET0NUWVBFIHBsaXN0IFBVQkxJQyAiLS8vQXBwbGUgQ29tcHV0ZXIvL0RURCBQTElTVCAxLjAvL0VOIiAiaHR0cDovL3d3dy5hcHBsZS5jb20vRFREcy9Qcm9wZXJ0eUxpc3QtMS4wLmR0ZCI+CjxwbGlzdCB2ZXJzaW9uPSIxLjAiPgo8ZGljdD4KCTxrZXk+c2ltdWx0YW5lb3VzX2xvZ2luX2VuYWJsZWQ8L2tleT4KCTx0cnVlLz4KPC9kaWN0Pgo8L3BsaXN0Pgo=
loginShell: /bin/bash
uidNumber: 20192
authAuthority: ;ApplePasswordServer;0x470bb9eb325f31c3000040ee00002257,1024 35 142348687369980182134507175767473848406728018835938950439244504199810591467084867869429532763785664902803450035110236201552277202539905523086333992178101548673534094938083763850217881171960226316582341046758647121979394968026644558722582733133246430363127883800192071325741645982074225105651514207812440564579 root@example.com:123.456.789.111
authAuthority: ;Kerberosv5;0x470bb9eb325f31c3000040ee00002257;username@EXAMPLE.COM;EXAMPLE.COM;1024 35 142348687369980182134507175767473848406728018835938950439244504199810591467084867869429532763785664902803450035110236201552277202539905523086333992178101548673534094938083763850217881171960226316582341046758647121979394968026644558722582733133246430363127883800192071325741645982074225105651514207812440564579 root@example.com:123.456.789.111
userPassword:: KioqKioqKio=
uid: username
cn: Firstname Lastname
gidNumber: 1029
givenName: Firstname
sn: Lastname
apple-user-homeurl:: PGhvbWVfZGlyPjx1cmw+YWZwOi8vamRhdGExLnVvcmVnb24uZWR1L1VzZXJzPC91cmw+PHBhdGg+c3R1cmNvPC9wYXRoPjwvaG9tZV9kaXI+
homeDirectory: /Network/Servers/example.com/Users/username
apple-user-homequota: 4294967296
mail: username@example.com

Now here's what comes out after the sed to remove the kerberos block (`sed '/authAuthority: ;Kerberosv5/,/ root@example.com/d' test2.ldif`):
Code:
# username, users, example.com
dn: uid=username,cn=users,dc=example,dc=com
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: shadowAccount
objectClass: apple-user
objectClass: extensibleObject
objectClass: organizationalPerson
objectClass: top
objectClass: person
apple-generateduid: 53CA02D7-B116-4461-B220-E3FC0B15964A
apple-mcxflags:: PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPCFET0NUWVBFIHBsaXN0IFBVQkxJQyAiLS8vQXBwbGUgQ29tcHV0ZXIvL0RURCBQTElTVCAxLjAvL0VOIiAiaHR0cDovL3d3dy5hcHBsZS5jb20vRFREcy9Qcm9wZXJ0eUxpc3QtMS4wLmR0ZCI+CjxwbGlzdCB2ZXJzaW9uPSIxLjAiPgo8ZGljdD4KCTxrZXk+c2ltdWx0YW5lb3VzX2xvZ2luX2VuYWJsZWQ8L2tleT4KCTx0cnVlLz4KPC9kaWN0Pgo8L3BsaXN0Pgo=
loginShell: /bin/bash
uidNumber: 20192
authAuthority: ;ApplePasswordServer;0x470bb9eb325f31c3000040ee00002257,1024 35 142348687369980182134507175767473848406728018835938950439244504199810591467084867869429532763785664902803450035110236201552277202539905523086333992178101548673534094938083763850217881171960226316582341046758647121979394968026644558722582733133246430363127883800192071325741645982074225105651514207812440564579 root@example.com:123.456.789.111

So obviously, something is not quite right...

Thanks!

---------- Post updated 07-30-09 at 10:59 AM ---------- Previous update was 07-29-09 at 07:43 PM ----------

ah ha... okay, so after removing line breaks, the command should be:

Code:
sed '/authAuthority: ;Kerberosv5/d' filename

Obviously (now) because it's all one line, and sed stops at the first line break after the match.
Last edited by staze; 07-29-2009 at 11:53 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove specified text from file

I am trying to write a script that kills old sessions, I've posted here over the past few days and the script is just about perfect except I want to be given the option to exclude specified PIDs from being killed. this is the entire script: if then rm /tmp/idlepids fi if then rm... (2 Replies)
Discussion started by: raidzero
2 Replies

2. Shell Programming and Scripting

Remove filename is text file

Hello, I got files full path in a text file like that /main/k/kdelibs/kdelibs4c2a_3.5.10.dfsg.1-2ubuntu7_i386.deb /main/k/kdelibs-experimental/libknotificationitem-dev_4.3.2-0ubuntu1_i386.deb /main/k/kdemultimedia/dragonplayer_4.3.2-0ubuntu1_i386.deb... (13 Replies)
Discussion started by: davidkhan
13 Replies

3. Shell Programming and Scripting

Split file into chunks of low & high byte

Hi guys, i have a question about spliting a binary file into 2 chunks. First chunk with all high bytes and the second one with all low bytes. What unix tools can i use? And how can this be performed? I looked in manpages of split and dd but this does not help. Thanks (2 Replies)
Discussion started by: basta
2 Replies

4. Shell Programming and Scripting

Parsing chunks of text and finding data

Hi, I need a script that parses and greps data out of a textfile. I have a text file that has this structure: File1 host1.localdomain text random text Found errors this text is random (41123) --- random random at.5165 ---- random random at.5165 ---- random random at.5165 ----... (2 Replies)
Discussion started by: erick_tuk
2 Replies

5. UNIX for Dummies Questions & Answers

Awk: Print out overlapping chunks of file - rows 0-20,10-30,20-40 etc.

First time poster, but the forum has saved my bacon more times than... Lots. Anyway, I have a text file, and wanted to use Awk (or any other sensible program) to print out overlapping sections, or arbitrary length. To describe by example, for file 1 2 3 4 5 etc... I want the out put... (3 Replies)
Discussion started by: matfald
3 Replies

6. Shell Programming and Scripting

awk for splitting file in constant chunks

Hi gurus, I wanted to split main file in 20 files with 2500 lines in each file. My main file conatins total 2500*20 lines. Following awk I made, but it is breaking with error. awk '{ for (i = 1; i <= 20; i++) { starts=2500*$i-1; ends=2500*$i; NR>=starts && NR<=ends {f=My$i".txt"; print >> f;... (10 Replies)
Discussion started by: mukesh.lalwani
10 Replies

7. Shell Programming and Scripting

Reverse sort on delimited chunks within a file

Hello, I have a large file in which data of names is sorted according to their homographs. The database has the following structure:Each set of homographs with their corresponding equivalents in Devanagari is separated out from the next set by a hard return. An example will make this... (12 Replies)
Discussion started by: gimley
12 Replies

8. Shell Programming and Scripting

Splitting a file into chunks of 1TB

Hi I have a file with different filesystems with there sizes. I need to split them in chucks of 1TB. The file looks like vf_MTLHQNASF07_Wkgp2 187428400 10601AW1 vf_MTLHQNASF07_Wkgp2 479504596 10604AW1 vf_MTLHQNASF07_Wkgp2 19940 10605AID vf_MTLHQNASF07_Wkgp2 1242622044... (4 Replies)
Discussion started by: bombcan
4 Replies

9. Shell Programming and Scripting

Deleting duplicated chunks in a file using awk/sed

Hi all, I'd always appreciate all helps from this site. I would like to delete duplicated chunks of strings on the same row(?). One chunk is comprised of four lines such as: path name starting point ending point voltage number I would like to delete duplicated chunks on the same... (5 Replies)
Discussion started by: jypark22
5 Replies

10. Shell Programming and Scripting

How to remove the text between all curly brackets from text file?

Hello experts, I have a text file with lot of curly brackets (both opening { & closing } ). I need to delete them alongwith the text between opening & closing brackets' pair. For ex: Input:- 59. Rh1 Qe4 {(Qf5-e4 Qd8-g8+ Kg6-f5 Qg8-h7+ Kf5-e5 Qh7-e7+ Ke5-f5 Qe7-d7+ Qe4-e6 Qd7-h7+ Qe6-g6... (6 Replies)
Discussion started by: prvnrk
6 Replies
All times are GMT -4. The time now is 01:22 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy