Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Normalize text field or sort values
Top Forums UNIX for Beginners Questions & Answers Normalize text field or sort values Post 303045956 by shepsol1 on Sunday 19th of April 2020 06:35:06 PM
Old 04-19-2020
Normalize text field or sort values
Hello so I am trying to "normalize" data so that it will upload correctly into a postgreql db.
The problem I am running into is that I formatted my db to injest the data in the order of which it is normalized.
However in certain data sets several fields of information are not available, so when I normalize those with my script it gets thrown off, and the values are not injested into the correct column in the DB.
So the options I have are
1. when I normalize the information, sort it so the first fields are the ones that are always present or
2. When i normalize the information insert the ,, for the missing fields in order to keep everything in the correct column.
Either way I have no idea how to do this and would appreciate any suggestions.
THANK YOU SO MUCH FOR YOUR TIME
Last edited by vbe; 04-20-2020 at 03:06 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to sort a field in a file having date values

Hi All, I am having a pipe delimited file .In this file the 3rd column is having date values.I need to get the min date and max date from that file. I have used cut -d '|' test.dat -f 3|sort -u But it is not sorting the date .How to sort the date column using unix commands Thanks ... (4 Replies)
Discussion started by: risshanth
4 Replies

2. Shell Programming and Scripting

Find top N values for field X based on field Y's value

I want to find the top N entries for a certain field based on the values of another field. For example if N=3, we want the 3 best values for each entry: Entry1 ||| 100 Entry1 ||| 95 Entry1 ||| 30 Entry1 ||| 80 Entry1 ||| 50 Entry2 ||| 40 Entry2 ||| 20 Entry2 ||| 10 Entry2 ||| 50... (1 Reply)
Discussion started by: FrancoisCN
1 Replies

3. Shell Programming and Scripting

Sort alpha on 1st field, numerical on 2nd field (sci notation)

I want to sort alphabetically on the first field and sort in descending numerical order on the 2nd field. With a normal "sort -r -n" it does this: abc ||| 5e-05 ||| bla abc ||| 3 ||| ble def ||| 1 ||| abc def ||| 0.2 ||| def As you can see it ignores the fact that 5e-05 is actually 0.00005... (1 Reply)
Discussion started by: FrancoisCN
1 Replies

4. Shell Programming and Scripting

adding field values if field matches

hi i have file as below , i want to add duplicate records like bell_bb to one record with valuve as 15 ( addition of both ) any oneline awk script to achive this ? header 0 CAMPAIGN_NAME 1 Bell_BB 14 Bell_MONTHLY 803 SOLO_UNBEATABLE 644 Bell_BB 1 Bell_MONTHLY 25 SOLO_UNBEATABLE... (4 Replies)
Discussion started by: raghavendra.cse
4 Replies

5. Shell Programming and Scripting

awk, comma as field separator and text inside double quotes as a field.

Hi, all I need to get fields in a line that are separated by commas, some of the fields are enclosed with double quotes, and they are supposed to be treated as a single field even if there are commas inside the quotes. sample input: for this line, 5 fields are supposed to be extracted, they... (8 Replies)
Discussion started by: kevintse
8 Replies

6. Shell Programming and Scripting

Awk Search text string in field, not all in field.

Hello, I am using awk to match text in a tab separated field and am able to do so when matching the exact word. My problem is that I would like to match any sequence of text in the tab-separated field without having to match it all. Any help will be appreciated. Please see the code below. awk... (3 Replies)
Discussion started by: rocket_dog
3 Replies

7. Shell Programming and Scripting

Read record from the text file contain multiple separated values & assign those values to variables

I have a file containing multiple values, some of them are pipe separated which are to be read as separate values and some of them are single value all are these need to store in variables. I need to read this file which is an input to my script Config.txt file name, first path, second... (7 Replies)
Discussion started by: ketanraut
7 Replies

8. Shell Programming and Scripting

Sort file by field 1 that has text as well as a number

I am using the below awk that results in the below output: awk '{k=$1 OFS $2; s+=$4; c++} END{for(i in s) print i, s/c}' input.txt > output.txt output.txt chr20:43625799-43625957 STK4:exon.6;STK4:exon.7 310.703 chr20:36770455-36770611 TGM2:exon.6;TGM2:exon.7 614.756... (5 Replies)
Discussion started by: cmccabe
5 Replies

9. Shell Programming and Scripting

awk to parse field and include the text of 1 pipe in field 4

I am trying to parse the input in awk to include the |gc= in $4 but am not able to. The below is close: awk so far: awk '{sub(/\|]+]++/, ""); print }' input.txt Input chr1 955543 955763 AGRN-6|pr=2|gc=75 0 + chr1 957571 957852 AGRN-7|pr=3|gc=61.2 0 + chr1 970621 ... (7 Replies)
Discussion started by: cmccabe
7 Replies

10. UNIX for Beginners Questions & Answers

awk GSUB read field values from multiple text files

My program run without error. The problem I am having. The program isn't outputting field values with the column headers to file.txt. Each of the column headers in file.txt has no data. MEMSIZE SECOND SASFoundation Filename The output results in file.txt should show: ... (1 Reply)
Discussion started by: dellanicholson
1 Replies

LEARN ABOUT OSX

unicode

unicode(n)						       Unicode normalization							unicode(n)

__________________________________________________________________________________________________________________________________________________

NAME

       unicode - Implementation of Unicode normalization

SYNOPSIS

       package require Tcl  8.3

       package require unicode	1.0

       ::unicode::fromstring string

       ::unicode::tostring uclist

       ::unicode::normalize form uclist

       ::unicode::normalizeS form string

_________________________________________________________________

DESCRIPTION

       This is an implementation in Tcl of the Unicode normalization forms.

COMMANDS

       ::unicode::fromstring string
	      Converts string to list of integer Unicode character codes which is used in unicode for internal string representation.

       ::unicode::tostring uclist
	      Converts list of integers uclist back to Tcl string.

       ::unicode::normalize form uclist
	      Normalizes  Unicode  characters  list  ulist according to form and returns the normalized list. Form form takes one of the following
	      values: D (canonical decomposition), C (canonical decomposition, followed by canonical composition),  KD	(compatibility	decomposi-
	      tion), or KC (compatibility decomposition, followed by canonical composition).

       ::unicode::normalizeS form string
	      A  shortcut  to ::unicode::tostring [unicode::normalize $form [::unicode::fromstring $string]].  Normalizes Tcl string and returns
	      normalized string.

EXAMPLES

       % ::unicode::fromstring "u0410u0411u0412u0413"
       1040 1041 1042 1043
       % ::unicode::tostring {49 50 51 52 53}
       12345
       %

       % ::unicode::normalize D {7692 775}
       68 803 775
       % ::unicode::normalizeS KD "u1d2c"
       A
       %

REFERENCES

       [1]    "Unicode Standard Annex #15: Unicode Normalization Forms", (http://unicode.org/reports/tr15/)

AUTHORS

       Sergei Golovan

BUGS, IDEAS, FEEDBACK
       This document, and the package it describes, will undoubtedly contain bugs and other problems.  Please report such in the category  string-
       prep  of  the  Tcllib  SF Trackers [http://sourceforge.net/tracker/?group_id=12883].  Please also report any ideas for enhancements you may
       have for either package and/or documentation.

SEE ALSO

       stringprep(n)

KEYWORDS

       normalization, unicode

COPYRIGHT

       Copyright (c) 2007, Sergei Golovan <sgolovan@nes.ru>

stringprep							       1.0.0								unicode(n)
All times are GMT -4. The time now is 10:54 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy