Sponsored Content
Top Forums Shell Programming and Scripting Pivot variable record length file and change delimiter Post 302139520 by thomasr on Sunday 7th of October 2007 11:39:46 PM
Old 10-08-2007
Pivot variable record length file and change delimiter

Hi experts.

I got a file (500mb max) and need to pivot it (loading into ORCL) and change BLANK delimiter to PIPE |.
Sometimes there are multipel BLANKS (as a particular value may be BLANK, or simply two BLANKS instead of one BLANK).

thanks for your input!

Cheers,


Layout
<id>BLANK<record_type>BLANK<field_num>BLANK<field_value>BLANK<field_num> BLANK<field_value>BLANK ........\n

example: (005, 004 are record_type, one leading for each line, same with <id>)

1234 005 001 23 198 5 098 45 033 2.........090 12\n
1234 006 008 66 002 345 129 2 345 2 897 45 091 56788 123 56321 443 45.........100 33\n

I want to use awk to "massage" this file to this output:
<record_type> <field_num> <field_value>
<record_type> <field_num> <field_value>

Example:
1234|005|001|23\n
1234|005|198|5\n
1234|005|098|45\n
1234|005|033|2\n
...
1234|005|090|12\n
1234|006|008|66\n
1234|006|002|345\n
1234|006|129|2\n
1234|006|345|2\n
1234|006|897|45\n
1234|006|091|56788\n
1234|006|443|45\n
....
1234|006|100|33\n
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Parsing a variable length record

I need to pick a field out of a variable record - the field is always found 4 fields after a certain text string, but it can be on any line of the record and in any position across the record on a line. I have had no luck through any of the Unix editors being able to cut a field that isn't always... (17 Replies)
Discussion started by: Barb
17 Replies

2. UNIX for Dummies Questions & Answers

What the command to find out the record length of a fixed length file?

I want to find out the record length of a fixed length file? I forgot the command. Any body know? (9 Replies)
Discussion started by: tranq01
9 Replies

3. UNIX for Dummies Questions & Answers

Convert a tab delimited/variable length file to fixed length file

Hi, all. I need to convert a file tab delimited/variable length file in AIX to a fixed lenght file delimited by spaces. This is the input file: 10200002<tab>US$ COM<tab>16/12/2008<tab>2,3775<tab>2,3783 19300978<tab>EURO<tab>16/12/2008<tab>3,28523<tab>3,28657 And this is the expected... (2 Replies)
Discussion started by: Everton_Silveir
2 Replies

4. Shell Programming and Scripting

Make variable length record a fixed length

Very, very new to unix scripting and have a unique situation. I have a file of records that contain 3 records types: (H)eader Records (D)etail Records (T)railer Records The Detail records are 82 bytes in length which is perfect. The Header and Trailer records sometimes are 82 bytes in... (3 Replies)
Discussion started by: jclanc8
3 Replies

5. Shell Programming and Scripting

Adding a delimiter to a variable length file

Hi, I'm new to unix, i have a variable length file like below, 01|Test|Test1|Sample| 02|AA|BB|CC|DD| 03|AAA|BBB|CCC|DDD|EEE|RRR|TTT|SSS|YYY| I need to make this as a fixed length file. Assume that i have 10 columns in the DAT file. for ex: the first 01 record is having 4cols -... (8 Replies)
Discussion started by: Mohankumar Venu
8 Replies

6. Shell Programming and Scripting

Shell script to put delimiter for a no delimiter variable length text file

Hi, I have a No Delimiter variable length text file with following schema - Column Name Data length Firstname 5 Lastname 5 age 3 phoneno1 10 phoneno2 10 phoneno3 10 sample data - ... (16 Replies)
Discussion started by: Gaurav Martha
16 Replies

7. Shell Programming and Scripting

How to change variable length field?

Hello, I have a file with a date field with various lengths. For example: m/d/yyyy hh:mm or h:mm mm/dd/yyyy hh:mm or h:mm Is there a way using sed or awk to change the field to m/d/y ? I don't need the hours and minutes in that field, just the date in the proper format. Thanks in... (6 Replies)
Discussion started by: sonnyo916
6 Replies

8. Shell Programming and Scripting

Perl Code to change file delimiter (passed as argument) to bar delimiter

Hi, Extremely new to Perl scripting, but need a quick fix without using TEXT::CSV I need to read in a file, pass any delimiter as an argument, and convert it to bar delimited on the output. In addition, enclose fields within double quotes in case of any embedded delimiters. Any help would... (2 Replies)
Discussion started by: JPB1977
2 Replies

9. Shell Programming and Scripting

How to check record delimiter of a file ?

My requirment is for every record of a particular file I've to check for a record delimeter (e.g. "\n") and if any row doesn't have "\n" then report it in error file . Please suggest me to go through this. (4 Replies)
Discussion started by: manab86
4 Replies

10. Shell Programming and Scripting

Convert variable length record to fixed length

Hi Team, I have an issue to split the file which is having special chracter(German Char) using awk command. I have a different length records in a file. I am separating the files based on the length using awk command. The command is working fine if the record is not having any... (7 Replies)
Discussion started by: Anthuvan
7 Replies
GETDELIM(3)						   BSD Library Functions Manual 					       GETDELIM(3)

NAME
getdelim, getline -- read a delimited record from a stream LIBRARY
Standard C Library (libc, -lc) SYNOPSIS
#include <stdio.h> ssize_t getdelim(char ** restrict lineptr, size_t * restrict n, int delimiter, FILE * restrict stream); ssize_t getline(char ** restrict lineptr, size_t * restrict n, FILE * restrict stream); DESCRIPTION
The getdelim() function reads from the stream until it encounters a character matching delimiter, storing the input in *lineptr. The buffer is NUL-terminated and includes the delimiter. The delimiter character must be representable as an unsigned char. If *n is non-zero, then *lineptr must be pre-allocated to at least *n bytes. The buffer should be allocated dynamically; it must be possible to free(3) *lineptr. getdelim() ensures that *lineptr is large enough to hold the input, updating *n to reflect the new size. The getline() function is equivalent to getdelim() with delimiter set to the newline character. RETURN VALUES
The getdelim() and getline() functions return the number of characters read, including the delimiter. If no characters were read and the stream is at end-of-file, the functions return -1. If an error occurs, the functions return -1 and the global variable errno is set to indi- cate the error. The functions do not distinguish between end-of-file and error, and callers must use feof(3) and ferror(3) to determine which occurred. EXAMPLES
The following code fragment reads lines from a file and writes them to standard output. char *line = NULL; size_t linesize = 0; ssize_t linelen; while ((linelen = getline(&line, &linesize, fp)) != -1) fwrite(line, linelen, 1, stdout); if (ferror(fp)) perror("getline"); ERRORS
[EINVAL] lineptr or n is a NULL pointer. [EOVERFLOW] More than SSIZE_MAX characters were read without encountering the delimiter. The getdelim() and getline() functions may also fail and set errno for any of the errors specified in the routines fflush(3), malloc(3), read(2), stat(2), or realloc(3). SEE ALSO
ferror(3), fgets(3), fopen(3) STANDARDS
The getdelim() and getline() functions conform to IEEE Std 1003.1-2008 (``POSIX.1''). BSD
June 30, 2010 BSD
All times are GMT -4. The time now is 09:38 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy