10-08-2007
Pivot variable record length file and change delimiter
Hi experts.
I got a file (500mb max) and need to pivot it (loading into ORCL) and change BLANK delimiter to PIPE |.
Sometimes there are multipel BLANKS (as a particular value may be BLANK, or simply two BLANKS instead of one BLANK).
thanks for your input!
Cheers,
Layout
<id>BLANK<record_type>BLANK<field_num>BLANK<field_value>BLANK<field_num> BLANK<field_value>BLANK ........\n
example: (005, 004 are record_type, one leading for each line, same with <id>)
1234 005 001 23 198 5 098 45 033 2.........090 12\n
1234 006 008 66 002 345 129 2 345 2 897 45 091 56788 123 56321 443 45.........100 33\n
I want to use awk to "massage" this file to this output:
<record_type> <field_num> <field_value>
<record_type> <field_num> <field_value>
Example:
1234|005|001|23\n
1234|005|198|5\n
1234|005|098|45\n
1234|005|033|2\n
...
1234|005|090|12\n
1234|006|008|66\n
1234|006|002|345\n
1234|006|129|2\n
1234|006|345|2\n
1234|006|897|45\n
1234|006|091|56788\n
1234|006|443|45\n
....
1234|006|100|33\n
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
I need to pick a field out of a variable record - the field is always found 4 fields after a certain text string, but it can be on any line of the record and in any position across the record on a line. I have had no luck through any of the Unix editors being able to cut a field that isn't always... (17 Replies)
Discussion started by: Barb
17 Replies
2. UNIX for Dummies Questions & Answers
I want to find out the record length of a fixed length file? I forgot the command. Any body know? (9 Replies)
Discussion started by: tranq01
9 Replies
3. UNIX for Dummies Questions & Answers
Hi, all.
I need to convert a file tab delimited/variable length file in AIX to a fixed lenght file delimited by spaces. This is the input file:
10200002<tab>US$ COM<tab>16/12/2008<tab>2,3775<tab>2,3783
19300978<tab>EURO<tab>16/12/2008<tab>3,28523<tab>3,28657
And this is the expected... (2 Replies)
Discussion started by: Everton_Silveir
2 Replies
4. Shell Programming and Scripting
Very, very new to unix scripting and have a unique situation. I have a file of records that contain 3 records types:
(H)eader Records
(D)etail Records
(T)railer Records
The Detail records are 82 bytes in length which is perfect. The Header and Trailer records sometimes are 82 bytes in... (3 Replies)
Discussion started by: jclanc8
3 Replies
5. Shell Programming and Scripting
Hi,
I'm new to unix, i have a variable length file like below,
01|Test|Test1|Sample|
02|AA|BB|CC|DD|
03|AAA|BBB|CCC|DDD|EEE|RRR|TTT|SSS|YYY|
I need to make this as a fixed length file. Assume that i have 10 columns in the DAT file.
for ex: the first 01 record is having 4cols -... (8 Replies)
Discussion started by: Mohankumar Venu
8 Replies
6. Shell Programming and Scripting
Hi,
I have a No Delimiter variable length text file with following schema -
Column Name Data length
Firstname 5
Lastname 5
age 3
phoneno1 10
phoneno2 10
phoneno3 10
sample data - ... (16 Replies)
Discussion started by: Gaurav Martha
16 Replies
7. Shell Programming and Scripting
Hello,
I have a file with a date field with various lengths. For example:
m/d/yyyy hh:mm or h:mm
mm/dd/yyyy hh:mm or h:mm
Is there a way using sed or awk to change the field to m/d/y ? I don't need the hours and minutes in that field, just the date in the proper format.
Thanks in... (6 Replies)
Discussion started by: sonnyo916
6 Replies
8. Shell Programming and Scripting
Hi,
Extremely new to Perl scripting, but need a quick fix without using TEXT::CSV
I need to read in a file, pass any delimiter as an argument, and convert it to bar delimited on the output. In addition, enclose fields within double quotes in case of any embedded delimiters.
Any help would... (2 Replies)
Discussion started by: JPB1977
2 Replies
9. Shell Programming and Scripting
My requirment is for every record of a particular file I've to check for a record delimeter (e.g. "\n") and if any row doesn't have "\n" then report it in error file .
Please suggest me to go through this. (4 Replies)
Discussion started by: manab86
4 Replies
10. Shell Programming and Scripting
Hi Team,
I have an issue to split the file which is having special chracter(German Char) using awk command.
I have a different length records in a file. I am separating the files based on the length using awk command.
The command is working fine if the record is not having any... (7 Replies)
Discussion started by: Anthuvan
7 Replies
LEARN ABOUT NETBSD
getline
GETDELIM(3) BSD Library Functions Manual GETDELIM(3)
NAME
getdelim, getline -- read a delimited record from a stream
LIBRARY
Standard C Library (libc, -lc)
SYNOPSIS
#include <stdio.h>
ssize_t
getdelim(char ** restrict lineptr, size_t * restrict n, int delimiter, FILE * restrict stream);
ssize_t
getline(char ** restrict lineptr, size_t * restrict n, FILE * restrict stream);
DESCRIPTION
The getdelim() function reads from the stream until it encounters a character matching delimiter, storing the input in *lineptr. The buffer
is NUL-terminated and includes the delimiter. The delimiter character must be representable as an unsigned char.
If *n is non-zero, then *lineptr must be pre-allocated to at least *n bytes. The buffer should be allocated dynamically; it must be possible
to free(3) *lineptr. getdelim() ensures that *lineptr is large enough to hold the input, updating *n to reflect the new size.
The getline() function is equivalent to getdelim() with delimiter set to the newline character.
RETURN VALUES
The getdelim() and getline() functions return the number of characters read, including the delimiter. If no characters were read and the
stream is at end-of-file, the functions return -1. If an error occurs, the functions return -1 and the global variable errno is set to indi-
cate the error.
The functions do not distinguish between end-of-file and error, and callers must use feof(3) and ferror(3) to determine which occurred.
EXAMPLES
The following code fragment reads lines from a file and writes them to standard output.
char *line = NULL;
size_t linesize = 0;
ssize_t linelen;
while ((linelen = getline(&line, &linesize, fp)) != -1)
fwrite(line, linelen, 1, stdout);
if (ferror(fp))
perror("getline");
ERRORS
[EINVAL] lineptr or n is a NULL pointer.
[EOVERFLOW] More than SSIZE_MAX characters were read without encountering the delimiter.
The getdelim() and getline() functions may also fail and set errno for any of the errors specified in the routines fflush(3), malloc(3),
read(2), stat(2), or realloc(3).
SEE ALSO
ferror(3), fgets(3), fopen(3)
STANDARDS
The getdelim() and getline() functions conform to IEEE Std 1003.1-2008 (``POSIX.1'').
BSD
June 30, 2010 BSD