Shell script for .Txt to .csv conversion with data processing


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell script for .Txt to .csv conversion with data processing
# 15  
Old 01-04-2015
Guys
I've been trying to decypher post#3 to come up with an output example to work from. The OP listed 4 columns, but maybe I'm missing the point. The way it reads to me is; column 1 and 2 get info from Parameter.txt; columns 3 and 4 from CNAI_DUMP_RAW.txt after a lookup for a match in the header row, and then values from the other rows in those matching columns.
That's how I came up with 'Ranges' (Parameters.txt col 2).

P.S. I sympathize with the OP in not being able to use his native tongue. I would hate to have to explain all this in French; being that English is all I know ( and I could use some lessons with that even ). Smilie

Last edited by ongoto; 01-04-2015 at 04:29 PM..
# 16  
Old 01-04-2015
Ongoto: Yes I am looking for that output only. Instead of Ranges, Lets keep 100 as a value.

Derekludwig: Ranges actually refers to [100-103], Here to make our work somewhats easier, I have selected 100.
# 17  
Old 01-05-2015
A program which extracts the values you want to see shouldn't be hard to do, but you will have to edit the parameters.txt file yourself and provide the values you want to see, and omit what you dont want to see. The script can only extract the values you offer. It can't do the editing. In order for scripts to work, the data types and format have to be consistent.

You need to use single words with no spaces, or use numeric values, in the values column:
HSN AsperPlan # would be good (camel case)

Code:
# The following types can be a problem for Linux:
CS 1 for non MBC
CHAP 6 for MBC 
CHAP 11 for BCCH on OL
etc.
# Better would be CS_1_for_non_MBC, etc. (or camel case)

If you want to use 100 in place of Ranges, then you will have to make those changes yourself, and give us (or yourself) a modified parameters.txt file to work from.

The field names in parameters.txt that are not found in CNAI_DUMP_RAW1.txt should be omited from parameters.txt.

---------- Post updated 01-05-15 at 07:42 AM ---------- Previous update was 01-04-15 at 11:17 PM ----------

First attempt in bash. Awk would be much faster I'm sure.
Code:
#!/bin/bash
###############################################################################
# Use these commands to strip carriage returns from input files
# sed -i 's|\r$||g' CNAI_DUMP_RAW1.txt
# sed -i 's|\r$||g' parameters.txt
###############################################################################
dumpfil="./CNAI_DUMP_RAW1.txt"
parmsfil="./parameters.txt"
outfil="./csv.out"
> $outfil

hdr=($(head -n 1 $dumpfil))
lnum=0
while read raw
do
    lnum=$(( $lnum + 1 ))
    [ $lnum -lt 3 ] && continue     # skip header and row separators
    while read parm1 parm2
    do
        fnum=0  # lookup actual values
        for fld in ${hdr[*]};
        do
            fnum=$(( $fnum + 1 ))
            if [[ $fld == ${parm1,,} ]]; then   # field headers are lower case
                break
            fi
        done
        cell=$(echo $raw | cut -d ' ' -f 4 2>&1)
        act_val=$(echo $raw | cut -d ' ' -f $fnum 2>&1)
        printf "%s,%s,%s,%s\n" "$parm1" "$parm2" "$cell" "$act_val" | tee -a $outfil
    done < $parmsfil
done < $dumpfil

# output
# ------
# CCMIN,RANGES,RBR201A,100
# CCHPWR,33,RBR201A,33
# CCHPWR,30,RBR201A,33
# ATT,YES,RBR201A,YES
# CB,NO,RBR201A,NO
# CBQ,HIGH,RBR201A,HIGH
# ...
# ...
# etc.

# real    2m8.955s
# user    1m59.455s
# sys   0m12.274s


Last edited by ongoto; 01-05-2015 at 11:39 AM..
This User Gave Thanks to ongoto For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to get the shell script to read the .txt file as an input/data?

i have written my shell script in notepad however i am struggling to pass the data file to be read to the script the data file is of .txt format. My target is to run the shell script from the terminal and pass 3 arguments e.g. polg@DESKTOP-BVPDC5C:~/CS1420/coursework$ bash valsplit.sh input.txt... (11 Replies)
Discussion started by: Gurdza32
11 Replies

2. Shell Programming and Scripting

Help with Shell Scrip in Masking particular columns in .csv file or .txt file using shell script

Hello Unix Shell Script Experts, I have a script that would mask the columns in .csv file or .txt file. First the script will untar the .zip files from Archive folder and processes into work folder and finally pushes the masked .csv files into Feed folder. Two parameters are passed ... (5 Replies)
Discussion started by: Mahesh G
5 Replies

3. Shell Programming and Scripting

Copy data to CSV file from txt output

Hi Everyone , Below is output from a TL1 , I want just the NE Name: and beside that the Temperature and the voltages in a csv file , Is this possible? > act-user:AB1S2a:ArshadFO:493::**********; AB1S2a 2016-02-07 10:13:24 M 493 COMPLD "ArshadFO:2016-02-07 10-04-55,0" ;... (11 Replies)
Discussion started by: adgjmpt
11 Replies

4. Shell Programming and Scripting

Conversion of xhtml data into csv format using dump utility

Hi Unix Gurus, I tried to convert the attached xhtml table content into csv file using unix shell script (lynx -dump filename) and got the below results: Title ID Owner Priority Estimate Project Change Date Changed By Complexity Create Date Created By Detail Estimate Total De tail... (6 Replies)
Discussion started by: bi.infa
6 Replies

5. Shell Programming and Scripting

Conversion of below Tabs Tex file into CSV format file : shell script needed

Request if some one could provide me shell script that converts the below "input file" to "CSV format file" given Name Domain Contact Phone Email Location ----------------------- ------------------------------------------------ ------- ----- ---------------------------------... (7 Replies)
Discussion started by: sreenath1037
7 Replies

6. Shell Programming and Scripting

Shell script for CSV conversion

thanks for allowing me join your forum i have an output of linux command "who" which provides following details..... CURRENT USER/ACCT INFO 17:31:36 up 4:49, 4 users, load average: 0.03, 0.04, 0.00 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT root :0 - 12:59 ?xdm? 4:54 0.02s /bin/sh /usr/bi... (1 Reply)
Discussion started by: ayyappancheta
1 Replies

7. UNIX for Advanced & Expert Users

shell script to format .CSV data

Hi all, I have written a shell script to search a specified directory (e.g. /home/user) for a list of specific words (shown as ${TMPDIR}/wordlist below). The script works well enough, but I was wondering if there was a way to display the line number that the word is found on? Thanks! cat... (1 Reply)
Discussion started by: tmcmurtr
1 Replies

8. Shell Programming and Scripting

shell script to insert data from gps.txt to mysql database

Hi, I have gps receiver, by using gpsd data i can read gps log data to my database(my sql). Steps: 1. telenet localhost 2947 > gps.txt (press enter) 2. r (press enter) //then i will get the data like below in gps.txt file Trying 127.0.0.1... Connected to localhost.... (1 Reply)
Discussion started by: gudivada213
1 Replies

9. Shell Programming and Scripting

Shell script to format a .CSV data

Hi There I needed to write a Unix shell script which will pick up the data from a .CSV file and reformat it as per the requirement and write it to another .CSV file. Currently I am in the proess of Data Import to "Remedy System" (A one kind of incident mangement Application) and this... (8 Replies)
Discussion started by: Uday1982
8 Replies
Login or Register to Ask a Question