Sponsored Content
Top Forums Shell Programming and Scripting Processing data that contains space and quote delimiters Post 302432657 by RickS on Friday 25th of June 2010 05:01:10 PM
Old 06-25-2010
Processing data that contains space and quote delimiters

I need to write a Bash script to process a data file that is in this format:

1 A B C D E
2 F G "H H" I J

As you can see, the data is delimited by a space, but there are also some fields that contain spaces and are surrounded by double-quotes. An example of that is "H H".

I wrote this test script to display the 4th parameter:

Code:
#!/bin/bash
while read line
do
        echo "line=$line"
        param4=$(echo $line | cut -d" " -f4)
        echo "param4=$param4"
done

Here's what it displays:

line=1 A B C D E
param4=C
line=2 F G "H H" I J
param4="H

For the second line of data, I wanted the fourth parameter to be "H H" (without the quotes) instead of one double quote and one H. It is using the other H and the trailing double quote as parameter 5. That is not what I wanted.

How can I process this data?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Capturing Data between first quote and next quote

I have input file like RDBMS FALIURE UTY8703 'USER_WORK.TEST' .HIghest return code '12' I want to parse data which comed between first quote till next quote USER_WORK.TEST can you please suggest how to do that (4 Replies)
Discussion started by: scorp_rahul23
4 Replies

2. Shell Programming and Scripting

Regex in grep to match all lines ending with a double quote (") OR a single quote (')

Hi, I've been trying to write a regex to use in egrep (in a shell script) that'll fetch the names of all the files that match a particular pattern. I expect to match the following line in a file: Name = "abc" The regex I'm using to match the same is: egrep -l '(^) *= *" ** *"$' /PATH_TO_SEARCH... (6 Replies)
Discussion started by: NanJ
6 Replies

3. UNIX for Dummies Questions & Answers

Remove two delimiters, space and double quotes

I would like to know how to replace a space delimiter with a ^_ (\037) delimiter and a double quote delimiter while maintaining the spaces inside the double quotes. The double quote delimiter is only used on text fields. I'd prefer a one-liner, but could handle a function or script that accepts... (4 Replies)
Discussion started by: SteveDWin
4 Replies

4. Shell Programming and Scripting

Insert data between comma delimiters-large file

Having a huge file in the following format. 2,3,1,,,4 1,2,3,,,,,5, 8,7,3,4,,,, Output needed is: 2,3,1,0.0,0.0,4 1,2,3,0.0,0.0,0.0,0.0,5, 8,7,3,4,0.0,0.0,0.0, I have tried reading the file each line, using AWK to parse to find out ",," and then insert 0.0 . It works but very slow. Need... (8 Replies)
Discussion started by: wincrazy
8 Replies

5. Shell Programming and Scripting

Replacing trailing space with single quote

Platform : RHEL 5.8 I want to end each line of this file with a single quote. $ cat hello.txt blueskies minnie mickey gravity snoopyAt VI editor's command mode, I have used the following command to replace the last character with a single quote. ~ ~ ~ :%s/$/'/gNow, the lines in the... (10 Replies)
Discussion started by: John K
10 Replies

6. UNIX for Dummies Questions & Answers

Replace Delimiters with Space

Hi All, How to Replace the delimiter for a particular field. I have used awk to replace the field values based on the position, but I tried to remove/replace delimiters with space on particular positions. I tried tr command with combination of awk not sure if this is the correct way, but I am... (3 Replies)
Discussion started by: mora
3 Replies

7. Shell Programming and Scripting

Skip the delimiter with in double quotes and count the number of delimiters during data extract

Hi All, I'm stuck-up in finding a way to skip the delimiter which come within double quotes using awk or any other better option. can someone please help me out. Below are the details: Delimited: | Sample data: 742433154|"SYN|THESIS MED CHEM PTY.... (2 Replies)
Discussion started by: BrahmaNaiduA
2 Replies

8. Shell Programming and Scripting

Replacing all but the first and last double quote in a line with a single quote with awk

From: 1,2,3,4,5,This is a test 6,7,8,9,0,"This, is a test" 1,9,2,8,3,"This is a ""test""" 4,7,3,1,8,"""" To: 1,2,3,4,5,This is a test 6,7,8,9,0,"This; is a test" 1,9,2,8,3,"This is a ''test''" 4,7,3,1,8,"''"Is there an easy syntax I'm overlooking? There will always be an odd number... (5 Replies)
Discussion started by: Michael Stora
5 Replies

9. Shell Programming and Scripting

Eval command with space, quote and bracket characters

Hi, I am Pradnya Gandhe trying to use in shell script. I want to use a bit complicated command using eval command in a shell script. <path to>\wsadmin.sh -lang jython -conntype NONE -c "AdminApp.install('war file path', '] -MapWebModToVH ] -MapRolesToUsers ] ]')" Works correctly as expected... (2 Replies)
Discussion started by: Pradnya Gandhe
2 Replies

10. Shell Programming and Scripting

Help/Advise please for converting space delimited string variable to comma delimited with quote

Hi, I am wanting to create a script that will construct a SQL statement based on a a space delimited string that it read from a config file. Example of the SQL will be For example, it will read a string like "AAA BBB CCC" and assign to a variable named IN_STRING. I then concatenate... (2 Replies)
Discussion started by: newbie_01
2 Replies
ENV(1)							    BSD General Commands Manual 						    ENV(1)

NAME
env -- set environment and execute command, or print environment SYNOPSIS
env [-iv] [-P altpath] [-S string] [-u name] [name=value ...] [utility [argument ...]] DESCRIPTION
The env utility executes another utility after modifying the environment as specified on the command line. Each name=value option specifies the setting of an environment variable, name, with a value of value. All such environment variables are set before the utility is executed. The options are as follows: -i Execute the utility with only those environment variables specified by name=value options. The environment inherited by env is ignored completely. -P altpath Search the set of directories as specified by altpath to locate the specified utility program, instead of using the value of the PATH environment variable. -S string Split apart the given string into multiple strings, and process each of the resulting strings as separate arguments to the env util- ity. The -S option recognizes some special character escape sequences and also supports environment-variable substitution, as described below. -u name If the environment variable name is in the environment, then remove it before processing the remaining options. This is similar to the unset command in sh(1). The value for name must not include the '=' character. -v Print verbose information for each step of processing done by the env utility. Additional information will be printed if -v is spec- ified multiple times. The above options are only recognized when they are specified before any name=value options. If no utility is specified, env prints out the names and values of the variables in the environment, with one name/value pair per line. Details of -S (split-string) processing The processing of the -S option will split the given string into separate arguments based on any space or <tab> characters found in the string. Each of those new arguments will then be treated as if it had been specified as a separate argument on the original env command. Spaces and tabs may be embedded in one of those new arguments by using single (``''') or double ('"') quotes, or backslashes (''). Single quotes will escape all non-single quote characters, up to the matching single quote. Double quotes will escape all non-double quote charac- ters, up to the matching double quote. It is an error if the end of the string is reached before the matching quote character. If -S would create a new argument that starts with the '#' character, then that argument and the remainder of the string will be ignored. The '#' sequence can be used when you want a new argument to start with a '#' character, without causing the remainder of the string to be skipped. While processing the string value, -S processing will treat certain character combinations as escape sequences which represent some action to take. The character escape sequences are in backslash notation. The characters and their meanings are as follows: c Ignore the remaining characters in the string. This must not appear inside a double-quoted string. f Replace with a <form-feed> character. Replace with a <new-line> character. Replace with a <carriage return> character. Replace with a <tab> character. v Replace with a <vertical tab> character. # Replace with a '#' character. This would be useful when you need a '#' as the first character in one of the arguments created by splitting apart the given string. $ Replace with a '$' character. \_ If this is found inside of a double-quoted string, then replace it with a single blank. If this is found outside of a quoted string, then treat this as the separator character between new arguments in the original string. " Replace with a <double quote> character. ' Replace with a <single quote> character. \ Replace with a backslash character. The sequences for <single-quote> and backslash are the only sequences which are recognized inside of a single-quoted string. The other sequences have no special meaning inside a single-quoted string. All escape sequences are recognized inside of a double-quoted string. It is an error if a single '' character is followed by a character other than the ones listed above. The processing of -S also supports substitution of values from environment variables. To do this, the name of the environment variable must be inside of '${}', such as: ${SOMEVAR}. The common shell syntax of $SOMEVAR is not supported. All values substituted will be the values of the environment variables as they were when the env utility was originally invoked. Those values will not be checked for any of the escape sequences as described above. And any settings of name=value will not effect the values used for substitution in -S processing. Also, -S processing can not reference the value of the special parameters which are defined by most shells. For instance, -S can not recog- nize special parameters such as: '$*', '$@', '$#', '$?' or '$$' if they appear inside the given string. Use in shell-scripts The env utility is often used as the interpreter on the first line of interpreted scripts, as described in execve(2). Note that the way the kernel parses the '#!' (first line) of an interpreted script has changed as of FreeBSD 6.0. Prior to that, the FreeBSD kernel would split that first line into separate arguments based on any whitespace (space or <tab> characters) found in the line. So, if a script named /usr/local/bin/someport had a first line of: #!/usr/local/bin/php -n -q -dsafe_mode=0 then the /usr/local/bin/php program would have been started with the arguments of: arg[0] = '/usr/local/bin/php' arg[1] = '-n' arg[2] = '-q' arg[3] = '-dsafe_mode=0' arg[4] = '/usr/local/bin/someport' plus any arguments the user specified when executing someport. However, this processing of multiple options on the '#!' line is not the way any other operating system parses the first line of an interpreted script. So after a change which was made for FreeBSD 6.0 release, that script will result in /usr/local/bin/php being started with the arguments of: arg[0] = '/usr/local/bin/php' arg[1] = '-n -q -dsafe_mode=0' arg[2] = '/usr/local/bin/someport' plus any arguments the user specified. This caused a significant change in the behavior of a few scripts. In the case of above script, to have it behave the same way under FreeBSD 6.0 as it did under earlier releases, the first line should be changed to: #!/usr/bin/env -S /usr/local/bin/php -n -q -dsafe_mode=0 The env utility will be started with the entire line as a single argument: arg[1] = '-S /usr/local/bin/php -n -q -dsafe_mode=0' and then -S processing will split that line into separate arguments before executing /usr/local/bin/php. ENVIRONMENT
The env utility uses the PATH environment variable to locate the requested utility if the name contains no '/' characters, unless the -P option has been specified. EXIT STATUS
The env utility exits 0 on success, and >0 if an error occurs. An exit status of 126 indicates that utility was found, but could not be exe- cuted. An exit status of 127 indicates that utility could not be found. EXAMPLES
Since the env utility is often used as part of the first line of an interpreted script, the following examples show a number of ways that the env utility can be useful in scripts. The kernel processing of an interpreted script does not allow a script to directly reference some other script as its own interpreter. As a way around this, the main difference between #!/usr/local/bin/foo and #!/usr/bin/env /usr/local/bin/foo is that the latter works even if /usr/local/bin/foo is itself an interpreted script. Probably the most common use of env is to find the correct interpreter for a script, when the interpreter may be in different directories on different systems. The following example will find the 'perl' interpreter by searching through the directories specified by PATH. #!/usr/bin/env perl One limitation of that example is that it assumes the user's value for PATH is set to a value which will find the interpreter you want to execute. The -P option can be used to make sure a specific list of directories is used in the search for utility. Note that the -S option is also required for this example to work correctly. #!/usr/bin/env -S -P/usr/local/bin:/usr/bin perl The above finds 'perl' only if it is in /usr/local/bin or /usr/bin. That could be combined with the present value of PATH, to provide more flexibility. Note that spaces are not required between the -S and -P options: #!/usr/bin/env -S-P/usr/local/bin:/usr/bin:${PATH} perl COMPATIBILITY
The env utility accepts the - option as a synonym for -i. SEE ALSO
printenv(1), sh(1), execvp(3), environ(7) STANDARDS
The env utility conforms to IEEE Std 1003.1-2001 (``POSIX.1''). The -P, -S, -u and -v options are non-standard extensions supported by FreeBSD, but which may not be available on other operating systems. HISTORY
The env command appeared in 4.4BSD. The -P, -S and -v options were added in FreeBSD 6.0. BUGS
The env utility does not handle values of utility which have an equals sign ('=') in their name, for obvious reasons. The env utility does not take multibyte characters into account when processing the -S option, which may lead to incorrect results in some locales. BSD
April 17, 2008 BSD
All times are GMT -4. The time now is 08:36 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy