Sponsored Content
Full Discussion: Read .csv file
Top Forums Shell Programming and Scripting Read .csv file Post 302873491 by FabianDe on Tuesday 12th of November 2013 08:35:14 AM
Old 11-12-2013
Read .csv file

Hello,

this is my very first comment on this forum and i hope i don't mess it up. If i do, please forgive me (also for any language mistakes you may wanna know that i'm not native speaking).

As i just started doing a bigger paper for my studies i got a bunch of data in seperate csv-files which i'll have to analyze. Unfortunately the data logger messed up the naming for most of the files, so i have to fix that first in order to have a consistent naming. So i figured i'll write a little bash program for that. Here's roughly what i got so far:

Code:
#!/bin/bash
filelist="SP*"
echo "filenames" > filenames.dat
for file in $filelist
  do
	temp=$(head -1 $file)
#	echo "$temp" >> filenames.dat
	cp $file $temp
done

So what i'm doing is finding all files that have the string "SP" in it and add them to the list "filelist". Then i consecutively go through that list, open the file and save the first line in the variable "temp" (the correct name of the files is saved in that first line). Then i copy the files using the variable "temp" as new name.

Theoretically the first line consists of two tokens seperated by a semicolon (so two columns since csv-file), whereas i only need the first. But that's just an extra i don't necessarily need to work. The first line would be okay, too.

So my .csv-files look kinda like this:

Code:
SPY.SPYNODE.SIDE_R.IOM1.LaneA.SYSAP_IOM.SYSAP_IOM_IMU_READER.W_APPLI_IOM_IMU_READER_MEAS03_SigInsReceiveImuOutputToInsMeas03VelincY#VALUE;Integer
Timestamp;Value
1366346707344760064;0
1366346707360758016;0
1366346707376759040;0
1366346707392758016;0

As i thought it should work but did not ("Argument too long") i took a closer look to what actually happens. Apparently the head command doesn't work properly, so what i really does is saving the whole file (so including all the numbers also) in the variable "temp". If i then use that variable to create/copy my file the command is too long for the shell (actually the file consists of about 500 000 lines :-) ). So apparently the "head -1" doesn't work, but i don't know why. "head -n 1" didn't work either.

I have no idea what's going on and i don't know what i could change.

Can anybody help me out here and give me a piece of advice (i'd appreciate it if the advice was actually concerning this topic of course :-) ).

Thank you very much in advance.

Fa
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Read Csv file

Hi All, I need to check if a csv file is empty, leaving the first line as the first line is header pls help Thanks (3 Replies)
Discussion started by: gwrm
3 Replies

2. Shell Programming and Scripting

Read Two lines in a CSV File and Compare

Hi , I have a CSV file ( file.csv) with some data as below: A,1,abc,x,y,z,,xyz,20100101,99991231 A,1,abc,x,y,z,234,xyz,20100101,99991231 I have to delete the duplicate line based on unique identifiers which are values in the fields- 2,3,4,8.These coulmns in both the rows have same... (6 Replies)
Discussion started by: Sheel
6 Replies

3. Shell Programming and Scripting

How can I read values from a CSV file using Shell?

SHELL SCRIPT Hi, I have a file in which contents are as follows: 9999,abdc,123 9988,aba_12,2323 and so on I want to read the contents of this file such that i can do echo "This is $a followed by $b an then $c" I tried the following but id did not work cat test | cut -d ',' -f1|... (7 Replies)
Discussion started by: mayanksargoch
7 Replies

4. Shell Programming and Scripting

read .csv file

Need UNIX script read below .csv file and print the line only records where type = TRN. Srno,Type,InputFileName,NewColumnData 1,TRN,File1.dat,11 2,TRN,File2.dat,12 3,TRN,File3.dat,13 4,REF,File4.dat, 5,REF,File5.dat, regards, santosh (4 Replies)
Discussion started by: santosh2k2
4 Replies

5. Shell Programming and Scripting

to read a CSV file and generate SQL output

Friends, This is what I need: I will pass a CSV file as an input, and I want my shell to be reading that CSV file, and based on the parameters it should generate SQLs and write those SQL in a different file in the same location. I'm new to Shell scripting. I'm currently working on a... (1 Reply)
Discussion started by: Ram.Math
1 Replies

6. UNIX for Dummies Questions & Answers

skip first line when doing a read of csv file

Folks, how do i skip the first line in a csv, while doing the read of a csv file in to a variable line by line. eg : do echo $line done < $rpt where rpt is path to csv file The initial 1st line is a garbage that i want to avoid, and start reading from 2nd line ... (2 Replies)
Discussion started by: venu
2 Replies

7. UNIX for Dummies Questions & Answers

When reading a csv file, counter to read 20 lines and wait for minute then read next 20 till end

Hello All, i am a newbie and need some help when reading a csv file in a bourne shell script. I want to read 10 lines, then wait for a minute and then do a reading of another 10 lines and so on in the same way. I want to do this till the end of file. Any inputs are appreciated ... (3 Replies)
Discussion started by: victor.s
3 Replies

8. Shell Programming and Scripting

Read csv file in bash

how to I use IFS to read 2 files (csv) and run the followiung script ./naviseccli -h 1.2.3.4 storagegroup -addhlu -gname $hostname -hlu $hlu_num -alu $alu_num the csv file for $hostname is host1 host2 . . . for hlu and alu its alu,hlu (2 Replies)
Discussion started by: tdubb123
2 Replies

9. Shell Programming and Scripting

Not able to read .csv file until open in vi editor

Hi, I am facing a problem regarding .csv file, my script does not read .csv file and if i open this file in vi editor and perform :wq option then only my script reads the .csv file. Thanks (5 Replies)
Discussion started by: ranabhavish
5 Replies

10. Shell Programming and Scripting

Read csv file, convert the data and make one text file in UNIX shell scripting

I have input data looks like this which is a part of a csv file 7,1265,76548,"0102:04" 8,1266,76545,"0112:04" I need to make the output data should look like this and the output data will be part of text file: 7|1265000 |7654899 |A| 8|12660000 |76545999 |B| The logic behind the... (6 Replies)
Discussion started by: RJG
6 Replies
GIT-CHECKOUT-INDEX(1)                                               Git Manual                                               GIT-CHECKOUT-INDEX(1)

NAME
git-checkout-index - Copy files from the index to the working tree SYNOPSIS
git checkout-index [-u] [-q] [-a] [-f] [-n] [--prefix=<string>] [--stage=<number>|all] [--temp] [-z] [--stdin] [--] [<file>...] DESCRIPTION
Will copy all files listed from the index to the working directory (not overwriting existing files). OPTIONS
-u, --index update stat information for the checked out entries in the index file. -q, --quiet be quiet if files exist or are not in the index -f, --force forces overwrite of existing files -a, --all checks out all files in the index. Cannot be used together with explicit filenames. -n, --no-create Don't checkout new files, only refresh files already checked out. --prefix=<string> When creating files, prepend <string> (usually a directory including a trailing /) --stage=<number>|all Instead of checking out unmerged entries, copy out the files from named stage. <number> must be between 1 and 3. Note: --stage=all automatically implies --temp. --temp Instead of copying the files to the working directory write the content to temporary files. The temporary name associations will be written to stdout. --stdin Instead of taking list of paths from the command line, read list of paths from the standard input. Paths are separated by LF (i.e. one path per line) by default. -z Only meaningful with --stdin; paths are separated with NUL character instead of LF. -- Do not interpret any more arguments as options. The order of the flags used to matter, but not anymore. Just doing git checkout-index does nothing. You probably meant git checkout-index -a. And if you want to force it, you want git checkout-index -f -a. Intuitiveness is not the goal here. Repeatability is. The reason for the "no arguments means no work" behavior is that from scripts you are supposed to be able to do: $ find . -name '*.h' -print0 | xargs -0 git checkout-index -f -- which will force all existing *.h files to be replaced with their cached copies. If an empty command line implied "all", then this would force-refresh everything in the index, which was not the point. But since git checkout-index accepts --stdin it would be faster to use: $ find . -name '*.h' -print0 | git checkout-index -f -z --stdin The -- is just a good idea when you know the rest will be filenames; it will prevent problems with a filename of, for example, -a. Using -- is probably a good policy in scripts. USING --TEMP OR --STAGE=ALL When --temp is used (or implied by --stage=all) git checkout-index will create a temporary file for each index entry being checked out. The index will not be updated with stat information. These options can be useful if the caller needs all stages of all unmerged entries so that the unmerged files can be processed by an external merge tool. A listing will be written to stdout providing the association of temporary file names to tracked path names. The listing format has two variations: 1. tempname TAB path RS The first format is what gets used when --stage is omitted or is not --stage=all. The field tempname is the temporary file name holding the file content and path is the tracked path name in the index. Only the requested entries are output. 2. stage1temp SP stage2temp SP stage3tmp TAB path RS The second format is what gets used when --stage=all. The three stage temporary fields (stage1temp, stage2temp, stage3temp) list the name of the temporary file if there is a stage entry in the index or . if there is no stage entry. Paths which only have a stage 0 entry will always be omitted from the output. In both formats RS (the record separator) is newline by default but will be the null byte if -z was passed on the command line. The temporary file names are always safe strings; they will never contain directory separators or whitespace characters. The path field is always relative to the current directory and the temporary file names are always relative to the top level directory. If the object being copied out to a temporary file is a symbolic link the content of the link will be written to a normal file. It is up to the end-user or the Porcelain to make use of this information. EXAMPLES
To update and refresh only the files already checked out $ git checkout-index -n -f -a && git update-index --ignore-missing --refresh Using git checkout-index to "export an entire tree" The prefix ability basically makes it trivial to use git checkout-index as an "export as tree" function. Just read the desired tree into the index, and do: $ git checkout-index --prefix=git-export-dir/ -a git checkout-index will "export" the index into the specified directory. The final "/" is important. The exported name is literally just prefixed with the specified string. Contrast this with the following example. Export files with a prefix $ git checkout-index --prefix=.merged- Makefile This will check out the currently cached copy of Makefile into the file .merged-Makefile. GIT
Part of the git(1) suite Git 2.17.1 10/05/2018 GIT-CHECKOUT-INDEX(1)
All times are GMT -4. The time now is 07:23 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy