Sponsored Content
Full Discussion: Transpose Messy Data
Top Forums UNIX for Advanced & Expert Users Transpose Messy Data Post 302943014 by 91674io on Monday 4th of May 2015 03:28:04 PM
Old 05-04-2015
Transpose Messy Data

I have a messy, pipe-delimited ("|") input dataset.

I would like to create a file of ID plus each component of field 4 which is delimited by ";" into a long, skinny shape for easier processing.

A couple of complications are that field 4 may contain both commas and linefeed characters from the source.

Sample data looks like:

Code:
ID1|VAR2|VAR3|VAR4|VAR5
ID2|VAR2|VAR3|PART1;PART2|1;2
ID3|VAR2|VAR3|A, B, C;PART2;BEFORE LF\nAFTER LF|1;2;3
ID4|VAR2|VAR3|1;2;3,;4|1;2;3;4

I would something like data like:

I
Code:
D1|VAR4
ID2|PART1
ID2|PART2
ID3|A, B, C
ID3|PART2
ID3|BEFORE LF  AFTER LF
ID4|1
ID4|2
ID4|3
ID4|4

Is there an elegant way to do this at the command line?

Thanks!

Last edited by Corona688; 05-04-2015 at 04:38 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to transpose data elements in awk

Hi, I have an input data file :- Test4599,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,2,2,Rain Test90,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1,1,0,Not Rain etc.... I wanted to transpose these data to:-... (2 Replies)
Discussion started by: ahjiefreak
2 Replies

2. Shell Programming and Scripting

How to transpose a table of data using awk

Hi. I have this data below:- v1 28 14 1.72414 1.72414 1.72414 1.72414 1.72414 v2 77 7 7.47126 6.89655 6.89655 6.89655 6.89655 v3 156 3 21.2644 21.2644 20.6897 21.2644 20.6897 v4 39 3 1.72414 1.72414 1.72414 1.72414 1.72414 v5 155 1 21.2644 23.5632 24.1379 23.5632 24.1379 v6 62 2 2.87356... (2 Replies)
Discussion started by: ahjiefreak
2 Replies

3. Shell Programming and Scripting

Transpose columns to Rows : Big data

Hi, I did read a few posts on the subjects, tried out a few solutions, but did not solve my problem. https://www.unix.com/302121568-post11.html https://www.unix.com/shell-programming-scripting/137953-large-file-columns-into-rows-etc-4.html Please help. Problem very similar to the second link... (15 Replies)
Discussion started by: genehunter
15 Replies

4. Shell Programming and Scripting

Transpose Daily Data from Column to Row.

Hi I'm looking to transpose Linux data from a daily report that logs every 10mins like below. After the first "comma" I need the daily total for Col2 and Col3 transposed like below. The new transposed format below will then be exported to Microsoft Excel for Reporting. Any help would be... (9 Replies)
Discussion started by: ravzter
9 Replies

5. Shell Programming and Scripting

Transpose Data from Columns to rows

Hello. very new to shell scripting and would like to know if anyone could help me. I have data thats being pulled into a txt file and currently have to manually transpose the data which is taking a long time to do. here is what the data looks like. Server1 -- Date -- Other -- value... (7 Replies)
Discussion started by: Mikes88
7 Replies

6. Shell Programming and Scripting

Transpose Column of Data to Rows

I can no longer find my commands, but I use to be able to transpose data with common fields from a single column to rows using a command line. My data is separated as follows: NAME=BOB ADDRESS=COLORADO PET=CAT NAME=SUSAN ADDRESS=TEXAS PET=BIRD NAME=TOM ADDRESS=UTAH PET=DOG I would... (7 Replies)
Discussion started by: docdave78
7 Replies

7. Shell Programming and Scripting

Transpose data as rows using awk

Hi I have below requirement, need help One file contains the meta data information and other file would have the data, match the column from file1 and with file2 and extract corresponding column value and display in another file File1: CUSTTYPECD COSTCENTER FNAME LNAME SERVICELVL ... (1 Reply)
Discussion started by: ravlapo
1 Replies

8. Shell Programming and Scripting

Help with transpose data content

Hi, Below is my input file: c116_g1_i1 -,-,-,+ c118_g2_i1 +,+ c118_g3_i1 + c120_g1_i1 +,+,+,+ . . Desired Output File c116_g1_i1 - c116_g1_i1 - c116_g1_i1 - c116_g1_i1 + c118_g2_i1 + c118_g2_i1 + (3 Replies)
Discussion started by: perl_beginner
3 Replies

9. UNIX for Beginners Questions & Answers

Transpose the data

Hi All, I have sort of a case to transpose data from rows to column input data Afghanistan|10000|1 Albania|25000|4 Algeria|25000|7 Andorra|10000|4 Angola|25000|47 Antigua and Barbuda|25000|23 Argentina|5000|3 Armenia|100000|12 Aruba|20000|2 Australia|50000|2 I need to transpose... (3 Replies)
Discussion started by: radius
3 Replies

10. UNIX for Beginners Questions & Answers

Transpose large data in UNIX

Hi I have the following sample of data: my full data dimention is 900,000* 1119 rs987435 C G 1 1 1 0 2 rs345783 C G 0 0 1 0 0 rs955894 G T 1 1 2 2 1 rs6088791 ... (7 Replies)
Discussion started by: marwah
7 Replies
SYSTEMD-SYSTEM.CONF(5)						systemd-system.conf					    SYSTEMD-SYSTEM.CONF(5)

NAME
systemd-system.conf, systemd-user.conf - System and session service manager configuration file SYNOPSIS
/etc/systemd/system.conf /etc/systemd/user.conf DESCRIPTION
When run as system instance systemd reads the configuration file system.conf, otherwise user.conf. These configuration files contain a few settings controlling basic manager operations. OPTIONS
All options are configured in the "[Manager]" section: LogLevel=, LogTarget=, LogColor=, LogLocation=, DumpCore=yes, CrashShell=no, ShowStatus=yes, CrashChVT=1, DefaultStandardOutput=journal, DefaultStandardError=inherit Configures various parameters of basic manager operation. These options may be overridden by the respective command line arguments. See systemd(1) for details about these command line arguments. DefaultTimeoutStartSec=, DefaultTimeoutStopSec=, DefaultRestartSec= Configures the default time-outs for starting and stopping of units, as well as the default time to sleep between automatic restarts of a units, as configured per-unit in TimeoutStartSec=, TimeoutStopSec= and RestartSec= (for service units see systemd.service(5) for details on the per-unit settings). For non-service units DefaultTimeoutStartSec= sets the default TimeoutSec= value. DefaultStartLimitInterval=, DefaultStartLimitBurst= Configure the default start rate limiting, as configured per-service by StartLimitInterval= and StartLimitBurst=. See systemd.service(5) for details on the per-service settings). CPUAffinity= Configures the initial CPU affinity for the init process. Takes a space-separated list of CPU indexes. JoinControllers=cpu,cpuacct net_cls,netprio Configures controllers that shall be mounted in a single hierarchy. By default, systemd will mount all controllers which are enabled in the kernel in individual hierarchies, with the exception of those listed in this setting. Takes a space-separated list of comma-separated controller names, in order to allow multiple joined hierarchies. Defaults to 'cpu,cpuacct'. Pass an empty string to ensure that systemd mounts all controllers in separate hierarchies. Note that this option is only applied once, at very early boot. If you use an initial RAM disk (initrd) that uses systemd, it might hence be necessary to rebuild the initrd if this option is changed, and make sure the new configuration file is included in it. Otherwise, the initrd might mount the controller hierarchies in a different configuration than intended, and the main system cannot remount them anymore. RuntimeWatchdogSec=, ShutdownWatchdogSec= Configure the hardware watchdog at runtime and at reboot. Takes a timeout value in seconds (or in other time units if suffixed with "ms", "min", "h", "d", "w"). If RuntimeWatchdogSec= is set to a non-zero value, the watchdog hardware (/dev/watchdog) will be programmed to automatically reboot the system if it is not contacted within the specified timeout interval. The system manager will ensure to contact it at least once in half the specified timeout interval. This feature requires a hardware watchdog device to be present, as it is commonly the case in embedded and server systems. Not all hardware watchdogs allow configuration of the reboot timeout, in which case the closest available timeout is picked. ShutdownWatchdogSec= may be used to configure the hardware watchdog when the system is asked to reboot. It works as a safety net to ensure that the reboot takes place even if a clean reboot attempt times out. By default RuntimeWatchdogSec= defaults to 0 (off), and ShutdownWatchdogSec= to 10min. These settings have no effect if a hardware watchdog is not available. CapabilityBoundingSet= Controls which capabilities to include in the capability bounding set for PID 1 and its children. See capabilities(7) for details. Takes a whitespace-separated list of capability names as read by cap_from_name(3). Capabilities listed will be included in the bounding set, all others are removed. If the list of capabilities is prefixed with ~, all but the listed capabilities will be included, the effect of the assignment inverted. Note that this option also affects the respective capabilities in the effective, permitted and inheritable capability sets. The capability bounding set may also be individually configured for units using the CapabilityBoundingSet= directive for units, but note that capabilities dropped for PID 1 cannot be regained in individual units, they are lost for good. TimerSlackNSec= Sets the timer slack in nanoseconds for PID 1 which is then inherited to all executed processes, unless overridden individually, for example with the TimerSlackNSec= setting in service units (for details see systemd.exec(5)). The timer slack controls the accuracy of wake-ups triggered by timers. See prctl(2) for more information. Note that in contrast to most other time span definitions this parameter takes an integer value in nano-seconds if no unit is specified. The usual time units are understood too. DefaultEnvironment= Sets manager environment variables passed to all executed processes. Takes a space-separated list of variable assignments. See environ(7) for details about environment variables. Example: DefaultEnvironment="VAR1=word1 word2" VAR2=word3 "VAR3=word 5 6" Sets three variables "VAR1", "VAR2", "VAR3". DefaultLimitCPU=, DefaultLimitFSIZE=, DefaultLimitDATA=, DefaultLimitSTACK=, DefaultLimitCORE=, DefaultLimitRSS=, DefaultLimitNOFILE=, DefaultLimitAS=, DefaultLimitNPROC=, DefaultLimitMEMLOCK=, DefaultLimitLOCKS=, DefaultLimitSIGPENDING=, DefaultLimitMSGQUEUE=, DefaultLimitNICE=, DefaultLimitRTPRIO=, DefaultLimitRTTIME= These settings control various default resource limits for units. See setrlimit(2) for details. Use the string infinity to configure no limit on a specific resource. These settings may be overridden in individual units using the corresponding LimitXXX= directives. Note that these resource limits are only defaults for units, they are not applied to PID 1 itself. SEE ALSO
systemd(1), systemd.directives(7), environ(7) systemd 208 SYSTEMD-SYSTEM.CONF(5)
All times are GMT -4. The time now is 09:15 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy