Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Deleting Repeating lines from a txt file via script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Deleting Repeating lines from a txt file via script
# 8  
Old 05-31-2014
Try something like this:
Code:
awk -F, '
  NF==4 && $1~/^[0-9]+$/ {                          # If there are 4 fields and the first field is numerical, then found a header
    is_header=1                                      
  }
  is_header {
    current=current $0 ORS                          # Add lines to current header
    if(/From=/){                                    # Last line of header
      if(current!=previous) printf "%s", current
      previous=current
      current=""
      is_header=0
    } 
    next
  }
  1
' file

# 9  
Old 05-31-2014
Hi Scrutinizer,

I'm able to catch your logic but still its not working, displays the entire file as it is.

I'm also trying will share once it works fine.
# 10  
Old 05-31-2014
Does it work with your own sample? What is your OS and version?
# 11  
Old 05-31-2014
Could you share the output. I'm using online Linux Emulator. Executing it in a script
# 12  
Old 05-31-2014
When I run it with your sample I get:
Code:
20140522121432,0,12,ram
Loc=India
From=ram@xxx.com, To=ravi@yyy.com,,
1
2
3
4
.
.
30
20140523121432,0,12,roger
Loc=redmont
From=roger@xxx.com, To=david@yyy.com,,
1
2
3
.
.
30
......
....
......
...

# 13  
Old 05-31-2014
oh is it.

I don't know where its going wrong. Even i'm using the same sample as input.
# 14  
Old 05-31-2014
What is your OS and version?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Creating repeating record in between file through script

apprecieate your help to resove this. My source file looke like 1001 000 HEADER 1001 001 RAJESH 1001 002 100 1001 002 200 1001 002 500 1001 006 FOOTER 1002 000 HEADER 1002 001 RAMESH 1002 002 100 1002 002 200 1002 002 500 1002 006 FOOTER my... (8 Replies)
Discussion started by: Ganesh L
8 Replies

2. Shell Programming and Scripting

Getting lines from .txt file

Hi I have a file with contents: NAMES John carrey williams How can I get all the names and store them in seperate variables(or arrays) please keep in mind that the no. of such names is not known.Three here is a bogus value ~thanks (4 Replies)
Discussion started by: leghorn
4 Replies

3. UNIX for Dummies Questions & Answers

redirecting arguments in a script to multiple lines in a .txt file

Ok hope my vocab is right here, i'm trying to write multiple sets of arguments to another file for example: I have a script that accepts four arguments and sends them to a another file $write.sh it then out in so the file receiver.txt would contain this: it then out in what... (2 Replies)
Discussion started by: austing5
2 Replies

4. UNIX for Dummies Questions & Answers

find lines in file1.txt not found in file2.txt memory problem

I have a diff command that does what I want but when comparing large text/log files, it uses up all the memory I have (sometimes over 8gig of memory) diff file1.txt file2.txt | grep '^<'| awk '{$1="";print $0}' | sed 's/^ *//' Is there a better more efficient way to find the lines in one file... (5 Replies)
Discussion started by: raptor25
5 Replies

5. Shell Programming and Scripting

merging two .txt files by alternating x lines from file 1 and y lines from file2

Hi everyone, I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines). Does anybody have an idea how to do that using perl, awk or sed?... (6 Replies)
Discussion started by: ink_LE
6 Replies

6. Shell Programming and Scripting

sed to cp lines x->y from 1.txt into lines a->b in file2.txt

I have one base file, and multiple target files-- each have uniform line structure so no need to use grep to find things-- can just define sections by line number. My question is quite simple-- can I use sed to copy a defined block of lines (say lines 5-10) from filename1.txt to overwrite an... (3 Replies)
Discussion started by: czar21
3 Replies

7. Shell Programming and Scripting

script to delete lines from a txt file if pattern matches

File 6 dbnawldb010-b office Memphis_Corp_SQL_Diff Memphis-Corp-SQL-Inc-Application-Backup 03/09/11 03:24:04 42 luigi-b IPNRemitDB Memphis_Corp_SQL_Diff Memphis-Corp-SQL-Inc-Application-Backup 03/10/11 00:41:36 6 ebs-sqldev1-b IPNTracking Memphis_Corp_SQL_Diff... (4 Replies)
Discussion started by: ajiwww
4 Replies

8. UNIX for Dummies Questions & Answers

Deleting lines in .txt with nonspecific value

Hello, i am new to the forum and know nothing about programing, Linux or Unix :( hope somebody can help me out. I have a .txt file that i need to delete certain lines from. After searching the forum i noticed that using "sed" was the way to go, so i installed gnuwin32 (i use windows xp... (4 Replies)
Discussion started by: luis3141
4 Replies

9. Shell Programming and Scripting

Deleting lines that contain spaces in a txt file

I need some help deleting lines in a file that contain spaces. Im sure awk or sed will work but i dont know much about those commands. Any help is appreciated :D (7 Replies)
Discussion started by: r04dw4rri0r
7 Replies

10. Shell Programming and Scripting

print all even lines of a txt file

In other news, I have a colors text file with hundreds of lines, and I want to print only the even numbered lines. for example I have this file looks something like this: ALLCOLORS.TXT red red green red blue red red red green red red blue green green green blue blue blue red blue blue blue... (1 Reply)
Discussion started by: ajp7701
1 Replies
Login or Register to Ask a Question