Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Awk script to create new file based on previous line

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Awk script to create new file based on previous line
# 1  
Old 05-12-2010
Awk script to create new file based on previous line
Need help creating a script that does the following:


Code:
  •  Sort a file
    • 

  •  Compare the previous line "last field" with current line "last field"
    • 

  •  If they are the same, print output to a file
    • 

  •  If they are different print output to a new file
    • 

  •  The script should keep creating new files if the previous line "last field" does not match the current lines last field
    • 

i.e.

Input file has the following:

Code:
001901070|S|DOE|JOHN|A|19630219|11|10|1|20021119|99991231|UNK|UNK||1005190567|IMM_20090228
001901070|S|DOE|JOHN|A|19630219|11|15|0|20021119|20031001|PMC|U0983AA||1005190567|IMM_20090131
001901070|S|DOE|JOHN|A|19630219|11|15|1|20031125|20090801|AVB|U1269AA||1005190567|IMM_20090131
001901070|S|DOE|JOHN|A|19630219|11|15|1|20061030|20070801|PMC|UNK||1005190567|IMM_20090131
001901070|S|DOE|JOHN|A|19630219|11|29|1|20021213|99991231|UNK|FAV065||1005190567|IMM_20090228
001901070|S|DOE|JOHN|A|19630219|11|29|2|20030122|20030205|MIP|FAV065||1005190567|IMM_20090330
001901070|S|DOE|JOHN|A|19630219|11|29|3|20030211|20030711|MIP|FAV065||1005190567|IMM_20090630
001901070|S|DOE|JOHN|A|19630219|11|29|5|20070215|20070819|MIP|FAV103||1005190567|IMM_20090630
001901070|S|DOE|JOHN|A|19630219|11|29|9|20090325|20090921|MIP|UNK||1005190567|IMM_20090630
001901070|S|DOE|JOHN|A|19630219|11|9|1|20021119|20121116|UNK|UNK||1005190567|IMM_20090330

The field separator is a "|"
# 2  
Old 05-12-2010
what have you tried till now??
# 3  
Old 05-12-2010
I am able to print the line if the the previous variable is different, but I am not able to:
  • Create a new file if the content differs
  • write to the existing file if they are the same

I need to find a way to keep creating files as well as append to current file if they are the same. This is the only thing I have so far:
Code:
awk -F '|' '$16 != prev {print; prev=$16}' Test_File.dat

If I change the compare the $16 = prev, I get no output at all. I have already sorted the file, so there should be some entries that are the same.
# 4  
Old 05-13-2010
gawk fun
That wasn't much fun. Took me a while to clue into the fact that arrays are associative and not numeric. Sorting was a pain until I made the element name a combo of the the last field and a record number to guarantee uniqueness.

Without GAWK the function asorti() is unavailable so I'm hoping you have GAWK. I'm a little unsure what you describe about where the records end up but I think you mean.. All the records that match the last field go into unique files.

Code:
cat input13.txt | gawk -F '|' '
BEGIN {
  i = 1;
}
{
  idx = $16 i;
  l[idx] = $0;
  ++i;
}
END {
  n = asorti(l, ni);
  prev = "";
  for (j = 1; j <= n; ++j) {
    outline = l[ni[j]];
    x = split(outline, al, "|");
    
    file = "out_" al[x] ".txt";
    if (al[x] != prev) {
      print "Unique " al[x];
      print outline > file;
      prev = al[x];
    } else {
      print "Same   " al[x];
      print outline >> file;
    }
  }
}'

Your data generates the following files...

Code:
out_IMM_20090131.txt
out_IMM_20090228.txt
out_IMM_20090330.txt
out_IMM_20090630.txt

Each file contains records that match the last field.

There's an easier way to do this and you can skip the sort. Here's the pseudo code...

Code:
# Remove all out_ files.
# Foreach record of input
#    File is record "out_" field[last] ".txt"
#    Append record to file

...which turns into...

Code:
rm out_*.txt > /dev/null 2>&1
cat input13.txt | gawk -F '|' '
{
  file = "out_" $16 ".txt";
  print $0 >> file
}'

This User Gave Thanks to m1xram For This Post:
Muga801
# 5  
Old 05-13-2010
Hi,
Try this.
Code:
sort -t "|" -k 16 Test_File.dat | awk -F "|" '{print > $NF}'

Thanks
Guru.
This User Gave Thanks to guruprasadpr For This Post:
Muga801
# 6  
Old 05-13-2010
Thanks to everyone who replied. I was able to use the 1 liner posted by Guru, it worked perfectly. I appreciate all the time and effort everyone put into this.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Returning multiple outputs of a single line based on previous repeated lines

Hello, I am trying to return a time multiple times from a file that has varying output just before the time instance, i.e. cat jumped cat jumped cat jumped time = 1.1 cat jumped cat jumped time = 1.2 cat jumped cat jumped time = 1.3 In this case i would like to output a time.txt... (6 Replies)
Discussion started by: ryddner
6 Replies

2. Shell Programming and Scripting

awk script -print line when $2 > $2 of previous line

Hi all, From a while loop I am reading a sorted file where I want to print only the lines that have $1 match and $2 only when the difference from $2 from the previous line is > 30. Input would be like ... AN237 010 193019 0502 1 CSU Amoxycillin AN237 080 ... (2 Replies)
Discussion started by: gafoleyo73
2 Replies

3. Shell Programming and Scripting

HELP: Shell Script to read a Log file line by line and extract Info based on KEYWORDS matching

I have a LOG file which looks like this Import started at: Mon Jul 23 02:13:01 EDT 2012 Initialization completed in 2.146 seconds. -------------------------------------------------------------------------------- -- Import summary for Import item: PolicyInformation... (8 Replies)
Discussion started by: biztank
8 Replies

4. Shell Programming and Scripting

Perl: Conditional replace based on previous and current value in a line

I need to read the contents of a file. Then I need to grep for a keyword and replace part of the grepped line based on the condition of previous and present line. Example input file: V { port1 = P; port2 = 0; shift_port = P0; /* if next shift_port is P0 I need... (9 Replies)
Discussion started by: naveen@
9 Replies

5. Programming

create a spool file based on values passed from korn shell to sql script

this is my issue. 4 parameters are passed from korn shell to sql script. parameter_1= varchar2 datatype or no value entered my user. parameter_2= number datatype or no value entered my user. parameter_3= number datatype or no value entered my user. parameter_4= number datatype or no... (5 Replies)
Discussion started by: megha2525
5 Replies

6. Shell Programming and Scripting

Improve script made to calculate value based on present and previous line

Hi all, I have made at small script to make a simple calculation on a file which is formatted in this way: I want to create a new file in which the value of particular line minus the previous line is printed. So my wanted output is: I have made the following program to do the job... (5 Replies)
Discussion started by: s052866
5 Replies

7. Shell Programming and Scripting

awk get matched line's previous line

hi everyone, a b in c d e f in g output is: a e so awk search for "in", then print out the matched line's previuos line. Please advice. (11 Replies)
Discussion started by: jimmy_y
11 Replies

8. Shell Programming and Scripting

Printing previous line based on pattern using sed

Hi, I have a written a shell script to get the previous line based on the pattern. For example if a file has below lines: ---------------------------------------------- #UNBLOCK_As _per #As per 205.162.42.92 #BLOCK_As_per #----------------------- #input checks abc.com... (5 Replies)
Discussion started by: Anjan1
5 Replies

9. Shell Programming and Scripting

Append specific lines to a previous line based on sequential search criteria

I'll try explain this as best I can. Let me know if it is not clear. I have large text files that contain data as such: 143593502 09-08-20 09:02:13 xxxxxxxxxxx xxxxxxxxxxx 09-08-20 09:02:11 N line 1 test line 2 test line 3 test 143593503 09-08-20 09:02:13... (3 Replies)
Discussion started by: jesse
3 Replies

10. Shell Programming and Scripting

awk;sed appending line to previous line....

I know this has been asked before but I just can't parse the syntax as explained. I have a set of files that has user information spread out over two lines that I wish to merge into one: User1NameLast User1NameFirst User1Address E-Mail:User1email User2NameLast User2NameFirst User2Address... (11 Replies)
Discussion started by: walkerwheeler
11 Replies
Login or Register to Ask a Question