Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Copy header values into records

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Copy header values into records
# 1  
Old 09-17-2013
Copy header values into records
I'm using a shell script to manipulate a data file. I have a large file with two sets of data samples (tracking memory consumption) taken over a long period of time, so I have many samples. The problem is that all the data is in the same file so that each sample contains two sets of data. Furthermore, the header line of each data set helpfully indicates the number of fields I should have, but some of the data labels in each record contain sub-labels that increase number of fields in each record. The main data I'm interested in are the labels and the value under AllocBytes in the first data set, and Bytes under the second data set. I think I can reduce the complexity of this problem of getting my fields sorted out by carrying a value from the header down to each record so I can tell each data set apart. Then I can know where my field of interest is by counting the fields from right to left. I'm hoping I can use the same method to copy the time stamp into each record as well.

The file looks something like this:

Code:
Sep 13 00:00:00
Name,Size,AllocSize,AllocBlocks,AllocBytes,MaxAllocBlocks,MaxAllocBytes
label1,4,8,4727,37816,4727,37816
label2,sublabelA,8,12,9037,108444,9062,108744
label3,sublabelA,sublabelB,8,12,1,12,2,24
label4,sublabelA,sublabelB,sublabelC,2,8,12,0,0,2,24
label5,8,12,2,24,2,24
Name,Allocs,Bytes,MaxAllocs,MaxBytes,FuncCalls
label101,1,8,1,8,1
label102_sublabelA,sublabelB,0,0,1,8,6
label103,sublabelA,sublabelB,sublabelC,0,0,1,8,27
label104,6,2,32,2,32,2
label105,sublabelA,sublabelB,0,0,2,16,24
Sep 13 00:00:10
Name,Size,AllocSize,AllocBlocks,AllocBytes,MaxAllocBlocks,MaxAllocBytes
label1,4,8,4727,37816,4727,37816
label2,sublabelA,8,12,9037,108444,9062,108744
label3,sublabelA,sublabelB,8,12,1,12,2,24
label4,sublabelA,sublabelB,sublabelC,2,8,12,0,0,2,24
label5,8,12,2,24,2,24
Name,Allocs,Bytes,MaxAllocs,MaxBytes,FuncCalls
label101,1,8,1,8,1
label102_sublabelA,sublabelB,0,0,1,8,6
label103,sublabelA,sublabelB,sublabelC,0,0,1,8,27
label104,6,2,32,2,32,2
label105,sublabelA,sublabelB,0,0,2,16,24


Here's what I would like to do:

Code:
Sep 13 00:00:00
Name,Size,AllocSize,AllocBlocks,AllocBytes,MaxAllocBlocks,MaxAllocBytes
Sep 13 00:00:00,label1,4,8,4727,37816,4727,37816,MaxAllocBytes
Sep 13 00:00:00,label2,sublabelA,8,12,9037,108444,9062,108744,MaxAllocBytes
Sep 13 00:00:00,label3,sublabelA,sublabelB,8,12,1,12,2,24,MaxAllocBytes
Sep 13 00:00:00,label4,sublabelA,sublabelB,sublabelC,2,8,12,0,0,2,24,MaxAllocBytes
Sep 13 00:00:00,label5,8,12,2,24,2,24,MaxAllocBytes
Name,Allocs,Bytes,MaxAllocs,MaxBytes,FuncCalls
Sep 13 00:00:00,label101,1,8,1,8,1,FuncCalls
Sep 13 00:00:00,label102_sublabelA,sublabelB,0,0,1,8,6,FuncCalls
Sep 13 00:00:00,label103,sublabelA,sublabelB,sublabelC,0,0,1,8,27,FuncCalls
Sep 13 00:00:00,label104,6,2,32,2,32,2,FuncCalls
Sep 13 00:00:00,label105,sublabelA,sublabelB,0,0,2,16,24,FuncCalls
Sep 13 00:00:10
Name,Size,AllocSize,AllocBlocks,AllocBytes,MaxAllocBlocks,MaxAllocBytes
Sep 13 00:00:10,label1,4,8,4727,37816,4727,37816,MaxAllocBytes
Sep 13 00:00:10,label2,sublabelA,8,12,9037,108444,9062,108744,MaxAllocBytes
Sep 13 00:00:10,label3,sublabelA,sublabelB,8,12,1,12,2,24,MaxAllocBytes
Sep 13 00:00:10,label4,sublabelA,sublabelB,sublabelC,2,8,12,0,0,2,24,MaxAllocBytes
Sep 13 00:00:10,label5,8,12,2,24,2,24,MaxAllocBytes
Name,Allocs,Bytes,MaxAllocs,MaxBytes,FuncCalls
Sep 13 00:00:10,label101,1,8,1,8,1,FuncCalls
Sep 13 00:00:10,label102_sublabelA,sublabelB,0,0,1,8,6,FuncCalls
Sep 13 00:00:10,label103,sublabelA,sublabelB,sublabelC,0,0,1,8,27,FuncCalls
Sep 13 00:00:10,label104,6,2,32,2,32,2,FuncCalls
Sep 13 00:00:10,label105,sublabelA,sublabelB,0,0,2,16,24,FuncCalls


Thank you
# 2  
Old 09-17-2013
Code:
awk -F"," 'NR==1 { D=$0 ; print ; next }
$1 == "Name" { print ; next }
{ $0=D"$0" ; print }' inputfile

# 3  
Old 09-17-2013
Two minor corrections to Corona688's proposal:
Code:
awk -F"," 'NF==1        { D=$0 ; print ; next }
           $1 == "Name" { print ; next }
                        { $0=D","$0 ; print }
        ' file

This User Gave Thanks to RudiC For This Post:
Corona688
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Help in printing records where there is a 'header' in the first record ???

Hi, I have a backup report that unfortunately has some kind of hanging indent thing where the first line contains one column more than the others I managed to get the output that I wanted using awk, but just wanting to know if there is short way of doing it using the same awk Below is what... (2 Replies)
Discussion started by: newbie_01
2 Replies

2. Shell Programming and Scripting

Prefix a variable in the first column of all the records of the files with and without header

In a bash shell, I have to prefix a variable to two .CSV files File1.CSV and File2.CSV. One of the files has a header and the other one is with no header in the below format: "value11","value12","value13","value14","value15","value16" "value21","value22","value23","value24","value25","value26"... (7 Replies)
Discussion started by: dhruuv369
7 Replies

3. UNIX for Dummies Questions & Answers

Dynamically merging 2 files on header values

Hi All, I have 2 files which i need to merge together based on the column names provided in the file. The first line in both files are header records. The first file has fixed columns but second file can have subset of the columns from file 1 File 1: ... (6 Replies)
Discussion started by: kushagra
6 Replies

4. Shell Programming and Scripting

Specific Header after every 30 records

Hi All, I have got a requirement. I have a source file, EMPFULL.txt and I need to split the data for every 30 records and place a Typical Header as below with system and page number too. 2012.01.03 Employee Dept Report 1... (6 Replies)
Discussion started by: srk409
6 Replies

5. Shell Programming and Scripting

Adding Header and Trailer records to a appended file

How can we a shell script and pass date parameters .I have 3 files comming from Datastage with |" delimited I need append 3 files as above: File1: P0000|"47416954|"AU|"000|"INS|"0000|"|"20060601|"99991231|"|"|"|"|"01 File 2:... (2 Replies)
Discussion started by: e1994264
2 Replies

6. UNIX for Dummies Questions & Answers

pull date from header and append to all records

I did some searches, but couldn't really find what I'm looking for. I have a file formatted as below: BOF ABC CO - XYZ COMM DATA OF 07/05/2011 EBA00000001 sdfa rtyus uyml EBB00000001 54682 984w3 EBA00000002 mkiyuasdf 98234 I want to pull the date from the header record and add it... (4 Replies)
Discussion started by: keeferb
4 Replies

7. Shell Programming and Scripting

improve performance - replace $\| with $#@ and remove header and trailer records

Hi All, In my file i need to remove header and trailer records which comes in 1st line and last line respectively. After that i need to replace '$\|' with '$#@'. I am using sed command for this and its taking lot of time. Is there any other command which can be used to improve performance? ... (1 Reply)
Discussion started by: HemaV
1 Replies

8. Shell Programming and Scripting

Merge text files while combining the multiple header/trailer records into one each.

Situation: Our system currently executes a job (COBOL Program) that generates an interface file to be sent to one of our vendors. Because this system processes information for over 100,000 employees/retirees (and growing), we'd like to multi-thread the job into processing-groups in order to... (4 Replies)
Discussion started by: oordonez
4 Replies

9. Shell Programming and Scripting

Count No of Records in File without counting Header and Trailer Records

I have a flat file and need to count no of records in the file less the header and the trailer record. I would appreciate any and all asistance Thanks Hadi Lalani (2 Replies)
Discussion started by: guiguy
2 Replies

10. UNIX for Dummies Questions & Answers

How to extract duplicate records with associated header record

All, I have a task to search through several hundred files and extract duplicate detail records and keep them grouped with their header record. If no duplicate detail record exists, don't pull the header. For example, an input file could look like this: input.txt HA D1 D2 D2 D3 D4 D4... (17 Replies)
Discussion started by: run_eim
17 Replies
Login or Register to Ask a Question