Sponsored Content
Top Forums Shell Programming and Scripting Awk multiple lines with 3rd column onto a single line? Post 302264764 by SoMoney on Thursday 4th of December 2008 05:33:13 PM
Old 12-04-2008
Awk multiple lines with 3rd column onto a single line?

I have a H U G E file with over 1million entries in it.
Looks something like this:

USER0001|DEVICE001|VAR1
USER0001|DEVICE001|VAR2
USER0001|DEVICE001|VAR3
USER0001|DEVICE001|VAR4
USER0001|DEVICE001|VAR5
USER0001|DEVICE001|VAR6
USER0001|DEVICE002|VAR1
USER0001|DEVICE002|VAR2
USER0001|DEVICE002|VAR3
USER0001|DEVICE002|VAR4
USER0001|DEVICE002|VAR5
USER0002|DEVICE003|VAR1
USER0002|DEVICE003|VAR2
USER0002|DEVICE003|VAR3
USER0002|DEVICE003|VAR4
USER0002|DEVICE003|VAR5
USER0002|DEVICE004|VAR1
USER0002|DEVICE004|VAR2
USER0002|DEVICE004|VAR3
USER0002|DEVICE004|VAR4
USER0002|DEVICE004|VAR5
USER0003|DEVICE005|VAR1
USER0003|DEVICE005|VAR2
USER0003|DEVICE005|VAR3
USER0003|DEVICE005|VAR4
USER0003|DEVICE005|VAR5
USER0003|DEVICE005|VAR6
USER0003|DEVICE006|VAR1
USER0003|DEVICE006|VAR2
USER0003|DEVICE006|VAR6


I would like a way to output only the uniq veriables onto a single line like so:

USER0001|DEVICE001|VAR1|VAR2|VAR3|VAR4|VAR5|VAR6
USER0001|DEVICE002|VAR1|VAR2|VAR3|VAR4|VAR5
USER0002|DEVICE003|VAR1|VAR2|VAR3|VAR4|VAR5
USER0002|DEVICE004|VAR1|VAR2|VAR3|VAR4|VAR5
USER0003|DEVICE005|VAR1|VAR2|VAR3|VAR4|VAR5|VAR6
USER0003|DEVICE006|VAR1|VAR2|VAR6


Is this doable with awk or do I need to use perl array?

Last edited by SoMoney; 12-04-2008 at 06:39 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Splitting a single line into multiple lines

I have a case where, I need to look into a file. Go to each line of the file, find the length of the line, if the length of the line is more than 75 chars, I need to split the line into multiple lines of 75chars max. If the length of the line is less than 75, we need not do anything. So at the... (4 Replies)
Discussion started by: thanuman
4 Replies

2. Shell Programming and Scripting

Multiple lines into a single line

Hi, I've some files with the following data and i need to convert the lines between the separator ---, into a single line. I've tried with the paste cmd but my main problem is that the number of lines between the separator is not fix, it can very between 1-4 lines. Input --- 2010-02-22... (4 Replies)
Discussion started by: RickyC9999
4 Replies

3. Shell Programming and Scripting

Combine multiple lines in single line

This is related to one of my previous post but now with a slight difference: I need the "Updated:" to be in one line as well as the "Information:" on one line as well. These are in multiple lines right now as seen below. These can have 2 or more lines that needs to be in one line. System name:... (8 Replies)
Discussion started by: The One
8 Replies

4. Shell Programming and Scripting

Multiple lines in a single column to be merged as a single line for a record

Hi, I have a requirement with, No~Dt~Notes 1~2011/08/1~"aaa bbb ccc ddd eee fff ggg hhh" Single column alone got splitted into multiple lines. I require the output as No~Dt~Notes 1~2011/08/1~"aaa<>bbb<>ccc<>ddd<>eee<>fff<>ggg<>hhh" mean to say those new lines to be... (1 Reply)
Discussion started by: Bhuvaneswari
1 Replies

5. Shell Programming and Scripting

Awk multiple lines with 4th column on to a single line

This is related to one of my previous post.. I have huge file currently I am using loop to read file and checking each line to build this single record, its taking much much time to parse those records.. I thought there should be a way to do this in awk or sed. I found this code in this forum... (7 Replies)
Discussion started by: Vasan
7 Replies

6. Shell Programming and Scripting

Awk match multiple columns in multiple lines in single file

Hi, Input 7488 7389 chr1.fa chr1.fa 3546 9887 chr5.fa chr9.fa 7387 7898 chrX.fa chr3.fa 7488 7389 chr21.fa chr3.fa 7488 7389 chr1.fa chr1.fa 3546 9887 chr9.fa chr5.fa 7898 7387 chrX.fa chr3.fa Desired Output 7488 7389 chr1.fa chr1.fa 2 3546 9887 chr5.fa chr9.fa 2... (2 Replies)
Discussion started by: jacobs.smith
2 Replies

7. Shell Programming and Scripting

Combine multiple lines into single line

Hi All , I have a file with below data # User@Host: xyz @ # Query_time: t1 Lock_time: t2 Rows_sent: n1 Rows_examined: n2 SET timestamp=1396852200; select count(1) from table; # Time: 140406 23:30:01 # User@Host: abc @ # Query_time: t1 Lock_time: t2 Rows_sent: n1 Rows_examined:... (6 Replies)
Discussion started by: rakesh_411
6 Replies

8. Shell Programming and Scripting

Coverting multiple lines to a single line

Hi all, I have a requirement to covert multiple lines in a comma delimited file to a single line through shell scripting. We should compare the data in the first column in each line. If it is same, then the other data should be put in the same line.Below is the sample input and expected output:... (4 Replies)
Discussion started by: Bobby_2000
4 Replies

9. UNIX for Beginners Questions & Answers

Merging multiple lines into single line based on one column

I Want to merge multiple lines based on the 1st field and keep into single record. SRC File: AAA_POC_DB.TAB1 AAA_POC_DB.TAB2 AAA_POC_DB.TAB3 AAA_POC_DB.TAB4 BBB_POC_DB.TAB1 BBB_POC_DB.TAB2 CCC_POC_DB.TAB6 OUTPUT ----------------- 'AAA_POC_DB','TAB1','TAB2','TAB3','TAB4'... (10 Replies)
Discussion started by: raju2016
10 Replies

10. UNIX for Beginners Questions & Answers

Multiple lines to single line

I have code as below # create temporary table `temp4277`(key(waybill_no)) select waybill_no,concat_ws('',card_type,card_series_no) cardinfo from rfid_temp_ticket where waybill_no='4277' group by... (4 Replies)
Discussion started by: kaushik02018
4 Replies
SYSTEMD-SYSTEM.CONF(5)						systemd-system.conf					    SYSTEMD-SYSTEM.CONF(5)

NAME
systemd-system.conf, system.conf.d, systemd-user.conf, user.conf.d - System and session service manager configuration files SYNOPSIS
/etc/systemd/system.conf, /etc/systemd/system.conf.d/*.conf, /run/systemd/system.conf.d/*.conf, /lib/systemd/system.conf.d/*.conf /etc/systemd/user.conf, /etc/systemd/user.conf.d/*.conf, /run/systemd/user.conf.d/*.conf, /usr/lib/systemd/user.conf.d/*.conf DESCRIPTION
When run as a system instance, systemd interprets the configuration file system.conf and the files in system.conf.d directories; when run as a user instance, systemd interprets the configuration file user.conf and the files in user.conf.d directories. These configuration files contain a few settings controlling basic manager operations. CONFIGURATION DIRECTORIES AND PRECEDENCE
The default configuration is defined during compilation, so a configuration file is only needed when it is necessary to deviate from those defaults. By default, the configuration file in /etc/systemd/ contains commented out entries showing the defaults as a guide to the administrator. This file can be edited to create local overrides. When packages need to customize the configuration, they can install configuration snippets in /usr/lib/systemd/*.conf.d/. Files in /etc/ are reserved for the local administrator, who may use this logic to override the configuration files installed by vendor packages. The main configuration file is read before any of the configuration directories, and has the lowest precedence; entries in a file in any configuration directory override entries in the single configuration file. Files in the *.conf.d/ configuration subdirectories are sorted by their filename in lexicographic order, regardless of which of the subdirectories they reside in. When multiple files specify the same option, for options which accept just a single value, the entry in the file with the lexicographically latest name takes precedence. For options which accept a list of values, entries are collected as they occur in files sorted lexicographically. It is recommended to prefix all filenames in those subdirectories with a two-digit number and a dash, to simplify the ordering of the files. To disable a configuration file supplied by the vendor, the recommended way is to place a symlink to /dev/null in the configuration directory in /etc/, with the same filename as the vendor configuration file. OPTIONS
All options are configured in the "[Manager]" section: LogLevel=, LogTarget=, LogColor=, LogLocation=, DumpCore=yes, CrashChangeVT=no, CrashShell=no, CrashReboot=no, ShowStatus=yes, DefaultStandardOutput=journal, DefaultStandardError=inherit Configures various parameters of basic manager operation. These options may be overridden by the respective process and kernel command line arguments. See systemd(1) for details. CtrlAltDelBurstAction= Defines what action will be performed if user presses Ctrl-Alt-Delete more than 7 times in 2s. Can be set to "reboot-force", "poweroff-force", "reboot-immediate", "poweroff-immediate" or disabled with "none". Defaults to "reboot-force". CPUAffinity= Configures the initial CPU affinity for the init process. Takes a list of CPU indices or ranges separated by either whitespace or commas. CPU ranges are specified by the lower and upper CPU indices separated by a dash. JoinControllers=cpu,cpuacct net_cls,netprio Configures controllers that shall be mounted in a single hierarchy. By default, systemd will mount all controllers which are enabled in the kernel in individual hierarchies, with the exception of those listed in this setting. Takes a space-separated list of comma-separated controller names, in order to allow multiple joined hierarchies. Defaults to 'cpu,cpuacct'. Pass an empty string to ensure that systemd mounts all controllers in separate hierarchies. Note that this option is only applied once, at very early boot. If you use an initial RAM disk (initrd) that uses systemd, it might hence be necessary to rebuild the initrd if this option is changed, and make sure the new configuration file is included in it. Otherwise, the initrd might mount the controller hierarchies in a different configuration than intended, and the main system cannot remount them anymore. RuntimeWatchdogSec=, ShutdownWatchdogSec= Configure the hardware watchdog at runtime and at reboot. Takes a timeout value in seconds (or in other time units if suffixed with "ms", "min", "h", "d", "w"). If RuntimeWatchdogSec= is set to a non-zero value, the watchdog hardware (/dev/watchdog or the path specified with WatchdogDevice= or the kernel option systemd.watchdog-device=) will be programmed to automatically reboot the system if it is not contacted within the specified timeout interval. The system manager will ensure to contact it at least once in half the specified timeout interval. This feature requires a hardware watchdog device to be present, as it is commonly the case in embedded and server systems. Not all hardware watchdogs allow configuration of the reboot timeout, in which case the closest available timeout is picked. ShutdownWatchdogSec= may be used to configure the hardware watchdog when the system is asked to reboot. It works as a safety net to ensure that the reboot takes place even if a clean reboot attempt times out. By default RuntimeWatchdogSec= defaults to 0 (off), and ShutdownWatchdogSec= to 10min. These settings have no effect if a hardware watchdog is not available. WatchdogDevice= Configure the hardware watchdog device that the runtime and shutdown watchdog timers will open and use. Defaults to /dev/watchdog. This setting has no effect if a hardware watchdog is not available. CapabilityBoundingSet= Controls which capabilities to include in the capability bounding set for PID 1 and its children. See capabilities(7) for details. Takes a whitespace-separated list of capability names as read by cap_from_name(3). Capabilities listed will be included in the bounding set, all others are removed. If the list of capabilities is prefixed with ~, all but the listed capabilities will be included, the effect of the assignment inverted. Note that this option also affects the respective capabilities in the effective, permitted and inheritable capability sets. The capability bounding set may also be individually configured for units using the CapabilityBoundingSet= directive for units, but note that capabilities dropped for PID 1 cannot be regained in individual units, they are lost for good. SystemCallArchitectures= Takes a space-separated list of architecture identifiers. Selects from which architectures system calls may be invoked on this system. This may be used as an effective way to disable invocation of non-native binaries system-wide, for example to prohibit execution of 32-bit x86 binaries on 64-bit x86-64 systems. This option operates system-wide, and acts similar to the SystemCallArchitectures= setting of unit files, see systemd.exec(5) for details. This setting defaults to the empty list, in which case no filtering of system calls based on architecture is applied. Known architecture identifiers are "x86", "x86-64", "x32", "arm" and the special identifier "native". The latter implicitly maps to the native architecture of the system (or more specifically, the architecture the system manager was compiled for). Set this setting to "native" to prohibit execution of any non-native binaries. When a binary executes a system call of an architecture that is not listed in this setting, it will be immediately terminated with the SIGSYS signal. TimerSlackNSec= Sets the timer slack in nanoseconds for PID 1, which is inherited by all executed processes, unless overridden individually, for example with the TimerSlackNSec= setting in service units (for details see systemd.exec(5)). The timer slack controls the accuracy of wake-ups triggered by system timers. See prctl(2) for more information. Note that in contrast to most other time span definitions this parameter takes an integer value in nano-seconds if no unit is specified. The usual time units are understood too. DefaultTimerAccuracySec= Sets the default accuracy of timer units. This controls the global default for the AccuracySec= setting of timer units, see systemd.timer(5) for details. AccuracySec= set in individual units override the global default for the specific unit. Defaults to 1min. Note that the accuracy of timer units is also affected by the configured timer slack for PID 1, see TimerSlackNSec= above. DefaultTimeoutStartSec=, DefaultTimeoutStopSec=, DefaultRestartSec= Configures the default timeouts for starting and stopping of units, as well as the default time to sleep between automatic restarts of units, as configured per-unit in TimeoutStartSec=, TimeoutStopSec= and RestartSec= (for services, see systemd.service(5) for details on the per-unit settings). For non-service units, DefaultTimeoutStartSec= sets the default TimeoutSec= value. DefaultTimeoutStartSec= and DefaultTimeoutStopSec= default to 90s. DefaultRestartSec= defaults to 100ms. DefaultStartLimitIntervalSec=, DefaultStartLimitBurst= Configure the default unit start rate limiting, as configured per-service by StartLimitIntervalSec= and StartLimitBurst=. See systemd.service(5) for details on the per-service settings. DefaultStartLimitIntervalSec= defaults to 10s. DefaultStartLimitBurst= defaults to 5. DefaultEnvironment= Sets manager environment variables passed to all executed processes. Takes a space-separated list of variable assignments. See environ(7) for details about environment variables. Example: DefaultEnvironment="VAR1=word1 word2" VAR2=word3 "VAR3=word 5 6" Sets three variables "VAR1", "VAR2", "VAR3". DefaultCPUAccounting=, DefaultBlockIOAccounting=, DefaultMemoryAccounting=, DefaultTasksAccounting=, DefaultIPAccounting= Configure the default resource accounting settings, as configured per-unit by CPUAccounting=, BlockIOAccounting=, MemoryAccounting=, TasksAccounting= and IPAccounting=. See systemd.resource-control(5) for details on the per-unit settings. DefaultTasksAccounting= defaults to on, the other four settings to off. DefaultTasksMax= Configure the default value for the per-unit TasksMax= setting. See systemd.resource-control(5) for details. This setting applies to all unit types that support resource control settings, with the exception of slice units. DefaultLimitCPU=, DefaultLimitFSIZE=, DefaultLimitDATA=, DefaultLimitSTACK=, DefaultLimitCORE=, DefaultLimitRSS=, DefaultLimitNOFILE=, DefaultLimitAS=, DefaultLimitNPROC=, DefaultLimitMEMLOCK=, DefaultLimitLOCKS=, DefaultLimitSIGPENDING=, DefaultLimitMSGQUEUE=, DefaultLimitNICE=, DefaultLimitRTPRIO=, DefaultLimitRTTIME= These settings control various default resource limits for units. See setrlimit(2) for details. The resource limit is possible to specify in two formats, value to set soft and hard limits to the same value, or soft:hard to set both limits individually (e.g. DefaultLimitAS=4G:16G). Use the string infinity to configure no limit on a specific resource. The multiplicative suffixes K (=1024), M (=1024*1024) and so on for G, T, P and E may be used for resource limits measured in bytes (e.g. DefaultLimitAS=16G). For the limits referring to time values, the usual time units ms, s, min, h and so on may be used (see systemd.time(7) for details). Note that if no time unit is specified for DefaultLimitCPU= the default unit of seconds is implied, while for DefaultLimitRTTIME= the default unit of microseconds is implied. Also, note that the effective granularity of the limits might influence their enforcement. For example, time limits specified for DefaultLimitCPU= will be rounded up implicitly to multiples of 1s. These settings may be overridden in individual units using the corresponding LimitXXX= directives. Note that these resource limits are only defaults for units, they are not applied to PID 1 itself. SEE ALSO
systemd(1), systemd.directives(7), systemd.exec(5), systemd.service(5), environ(7), capabilities(7) systemd 237 SYSTEMD-SYSTEM.CONF(5)
All times are GMT -4. The time now is 07:46 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy