Sponsored Content
Full Discussion: Work with huge Zipped files
Top Forums Shell Programming and Scripting Work with huge Zipped files Post 302872273 by Homa on Friday 8th of November 2013 04:57:52 AM
Old 11-08-2013
Ubuntu Work with huge Zipped files

Hello dear members,

I have one general and one specific question which I will be very grateful if you could help me with them. Let's start with my general question:

1. I am working on cluster computer shared with other people and I need to manipulate a big zipped text file of 13 GB. There is no possibility of unzipping it, if I could do that, I would have any problem but my problem is that working with zipped files using Awk or Linux commands become very difficult. I would be very grateful if you could give me some directions in this regard.

2. The second question is that being in that 13 GB zipped, I want to select one part of the file to make an example file to write my code to be able to later apply them to the whole file. When I display the file on shell by the command:

Code:
zcat zippedfile | less

and I copy a section just by clicking and I paste it in a text editor, the spacing, number of fields, everything gets destroyed. How could I select a part of the file by command line without unzipping it?

Thank you
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

uploading Zipped files get 553 error

I am trying to upload .zip files to Unix server and get the error 553 qmerev2002.zip: Permission denied, what is my problem?? I am able to load other files and folders fine. (3 Replies)
Discussion started by: CoastGuard1970
3 Replies

2. Shell Programming and Scripting

Greping from zipped files without unzipping

I have more than 500 gzipped files in a directory. I have one lookup file in the same directory with 200 key values. I need to get the name of the gzipped file which have any of these 200 key values. Here my criteria is do not unzip the files due to space constraint. Any suggestion? (3 Replies)
Discussion started by: kanu_kanu
3 Replies

3. Linux

grep thru zipped files

Hi All, I need to check the logs and grep it. The problem is that the previous days are zipped with *.gz. How do grep on the *.gz? Thank you in advance. (1 Reply)
Discussion started by: itik
1 Replies

4. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

5. Shell Programming and Scripting

Difference between 2 zipped text files.

Hi, I have below two zipped files - file1.gz 023384148,1,,,02077301961,R,02077301961,N,0,02077301961,N,0,0,8010,02077300518,U,N,,02077300518,U,20100501011732,20100501011732,0,20100501011815,00000430,16,16,10,N;... (8 Replies)
Discussion started by: ravigupta2u
8 Replies

6. Shell Programming and Scripting

Copy all zipped files from one folder to another

Hi everyone, when I try to copy *.gz files run cp within the correct source folder it works as follow: Source folder = C:/Documents and Settings/user/Recent papers/2771/ Destination folder = C:/Documents and Settings/user/My documents/1532/temp cp *.gz "C:/Documents and Settings/user/My... (2 Replies)
Discussion started by: cgkmal
2 Replies

7. Shell Programming and Scripting

How to parse a huge 600MB zipped file?

I'm new to Unix, trying to parse a huge 600MB zipped file... I need to bzcat this file once and do some calculations (word count) on the lines based on certain criteria (see script) the correct result/output should be: column1=6 column2=4 the problem is that I'm getting column2=0 (see... (16 Replies)
Discussion started by: DeltaComp
16 Replies

8. Shell Programming and Scripting

Renaming all files inside a zipped file

Hi, To all the Unix gurus this should be a simple task, but as a newbie I'm finding it hard to crack this. Any help is highly appreciated... Scenario: Step 1 : Move zip file from FTP folder to WORK folder Step 2: Unzip the file "Sample_YYYYMMDDHHMMSS.tar.gz" which contains many file... (10 Replies)
Discussion started by: asandy1234
10 Replies

9. Shell Programming and Scripting

Grabing the same timestamp from files that are ZIPPED

Hi, I am zipping more than 20 files that has same timestamp in all of them. I need to create the zip file with the same timestamp as in the files that are zipped. So I have files: Dummytest_20140601W110515_file1.txt Dummytest_20140601W110515_file2.txt ....... .......... (5 Replies)
Discussion started by: Saanvi1
5 Replies

10. Shell Programming and Scripting

awk does not work well with huge data?

Dear all , I found that if we work with thousands line of data, awk does not work perfectly. It will cut hundreds line (others are deleted) and works only on the remain data. I used this command : awk '$1==1{$1="Si"}{print>FILENAME}' coba.xyz to change value of first column whose value is 1... (4 Replies)
Discussion started by: ariesto
4 Replies
clu_get_info(8) 					      System Manager's Manual						   clu_get_info(8)

NAME
clu_get_info - Gets information about a cluster and its members SYNOPSIS
/usr/sbin/clu_get_info [-full] [-q] [-m memberid] [-raw] [-state] OPTIONS
The -full option returns detailed information about all members in the cluster, or about one member if the -m memberid option is specified. Suppresses output. Use this option when only the return value is of interest; for example, to determine whether the system is running as a cluster member. When specified in combination with the -m option, the -q option can be used to determine whether a particular member of a cluster is up or down. For example: # clu_get_info -q -m 2 && echo "Member 2 is a member of the cluster" Member 2 is a member of the clus- ter The -m option returns information about the member specified by memberid. The -raw option returns detailed information about the clus- ter and all members in the cluster in raw output, which is intended to be used as input to scripts. The -state option returns the state of all members in the cluster in raw output. The states returned are: UP, DOWN, or NOT_CONF (not configured). DESCRIPTION
When invoked on a member of a cluster, the clu_get_info command displays general information about the cluster and its members. The infor- mation includes: The name of the cluster and the number of members configured in the cluster. The member ID of the member issuing the com- mand. The device name and number of votes of the quorum disk, if configured. For each member, its member ID, host name, cluster intercon- nect IP name, and current state (UP or DOWN). When invoked on a system that is not a cluster member, the clu_get_info command prints an error message and exits. The clu_get_info -full option displays additional information about the cluster. The information includes: Cluster incarnation -- A hexa- decimal number used to uniquely identify the cluster. Cluster expected votes -- Number of votes expected when all configured cluster mem- bers are up and any configured quorum disk is present. Current votes -- The number of votes contributed by current cluster members and the quorum disk as seen by this member. Quorum votes -- The number of votes required for the cluster to be up and running. The clu_get_info -full command also displays the following information for each member: Cluster interconnect IP address -- The IP address of the cluster interconnect. Member base O/S version -- The version of the base operating system. Member cluster version -- The version of the TruCluster Server software. Member running version -- The current running version of the TruCluster software, which returns one of the following: INSTALLED -- the current running version; same as the member cluster version returned. BELOW INSTALLED -- one version below the current running version returned in member cluster version; useful when performing a rolling upgrade. UNKNOWN -- the running version is unknown. Member name -- The member name used by the connection manager (CNX) as a member identifier. Node votes -- The votes assigned to this member (as set in the cluster_node_votes kernel attribute for the clubase subsystem). csid -- The cluster system identifier, which is a unique hexadecimal number that identifies each member in a cluster. The clu_get_info -raw option displays detailed information, in raw output, about all members in the cluster. This output is intended to be used as input to scripts. Data is unlabeled, and data fields are separated by colons. The fields returned are the same as if specifying the -full option only data is in raw format. The clu_get_info -raw option returns a clusterwide information record and member-specific information records. A clusterwide information record begins with the initiator C and returns data in the following order: Cluster name Number of members configured in this cluster Mem- ber ID for this member Cluster incarnation Cluster expected votes Current votes Votes required for quorum Quorum disk Quorum disk votes A member-specific information record begins with the initiator M and returns data in the following order: Member ID Host name Cluster interconnect IP name Cluster interconnect IP address Member state Member base O/S version Member cluster version Member running version Member name Member votes Csid The clu_get_info -state option displays the state of all members in the cluster. Output is intended to be used by a script to determine which cluster members are up, down, or not configured. Data is unlabeled, and data fields are separated by colons. A state information record begins with the initiator S and returns the following output based on the configuration state of all members in the cluster: UP -- The member ID is configured and up. DOWN -- The member ID is configured and down. NOT_CONF -- The member ID is not configured in the cluster. If a member of the cluster is down, clu_get_info displays UNKNOWN in various member fields. If the member is down and its /etc/rc.config file contains illegal values, clu_get_info displays CONFIGURATION_ERROR in the related fields. An example is provided in the EXAMPLES sec- tion. EXIT STATUS
Success. The system is not a cluster member or an error has occurred. EXAMPLES
The following command displays general information about the cluster and its members: # clu_get_info Cluster information for cluster deli Number of members configured in this cluster = 3 Memberid for this member = 1 Quorum disk = dsk4h Quorum disk votes = 1 Information on each cluster member Cluster memberid = 1 Hostname = polishham.zk4.com Cluster interconnect IP name = polishham-mc0 Member state = UP Cluster memberid = 2 Hostname = provolone.zk4.com Cluster interconnect IP name = provolone-mc0 Member state = UP Cluster memberid = 3 Hostname = pepicelli.zk4.com Cluster interconnect IP name = pepicelli-mc0 Member state = UP If one member of the cluster is down and its rc.config file contains illegal values, the following is displayed: Cluster memberid = 2 Hostname = CONFIGURATION_ERROR Cluster interconnect IP name = CONFIGURATION_ERROR Member state = DOWN The following command displays general information about one member: # clu_get_info -m 1 Cluster memberid = 1 Hostname = polishham.zk4.com Cluster interconnect IP name = polishham-mc0 Member state = UP The following command displays more information about all members in the cluster: # clu_get_info -full Cluster information for cluster deli Number of members configured in this cluster = 3 Memberid for this member = 1 Cluster incarnation = 0x7b5c2 Cluster expected votes = 3 Current votes = 3 Votes required for quorum = 2 Quorum disk = dsk4h Quorum disk votes = 1 Information on each cluster member Cluster memberid = 1 Hostname = polishham.zk4.com Cluster interconnect IP name = polishham-mc0 Cluster interconnect IP address = 10.0.0.1 Member state = UP Member base O/S version = Compaq Tru64 UNIX V5.0A Member cluster version = TruCluster Server V5.0A Member running version = INSTALLED Member name: polishham Member votes = 1 csid = 0x10001 Cluster memberid = 2 Hostname = provolone.zk4.com Cluster interconnect IP name = provolone-mc0 Cluster interconnect IP address = 10.0.0.2 Member state = UP Member base O/S version = Compaq Tru64 UNIX V5.0A Member cluster version = TruCluster Server V5.0A Member running version = INSTALLED Member name: provolone Member votes: 1 csid = 0x10002 Cluster memberid = 3 Hostname = pepicelli.zk4.com Cluster interconnect IP name = pepicelli-mc0 Cluster interconnect IP address = 10.0.0.3 Member state = UP Member base O/S version = Compaq Tru64 UNIX V5.0A Member cluster version = TruCluster Server V5.0A Member running version = INSTALLED Member name: pepicelli Member votes: 1 csid = 0x10003 If one member of the cluster is down and its rc.config file contains illegal values, the following is displayed: Cluster memberid = 2 Hostname = CONFIGURATION_ERROR Cluster interconnect IP name = CONFIGURATION_ERROR Cluster interconnect IP address = CONFIGURATION_ERROR Member state = DOWN Member base O/S version = CONFIGURATION_ERROR Member cluster version = CONFIGURATION_ERROR Member running version = UNKNOWN Member name = UNKNOWN Member votes = UNKNOWN csid = UNKNOWN The following command displays more information about a particular member of the cluster: # clu_get_info -full -m 1 Cluster memberid = 1 Hostname = polishham.zk4.com Cluster interconnect IP name = polishham-mc0 Cluster interconnect IP address = 10.0.0.1 Member state = UP Member base O/S version = Compaq Tru64 UNIX V5.0A Member cluster version = TruCluster Server V5.0A Member running version = INSTALLED Member name: polishham Member votes: 1 csid = 0x10001 The following command displays detailed information about all members in the cluster in raw output: # clu_get_info -raw C:deli:3:2:0x7b5c2:3:3:2:(null):0 M:1:polishham.zk4.com:polishham-mc0:10.0.0.1:UP:Compaq Tru64 UNIX V5.0A: TruCluster Server V5.0A:INSTALLED:polishham:1:0x10001 M:2:provolone.zk4.com:provolone-mc0:10.0.0.2:UP:Compaq Tru64 UNIX V5.0A: TruCluster Server V5.0A:INSTALLED:provolone:1:0x10002 M:3:pepicelli.zk4.com:pepicelli-mc0:10.0.0.3:UP:Compaq Tru64 UNIX V5.0A: TruCluster Server V5.0A:INSTALLED:pepicelli:1:0x10003 The following command displays the state of all members in the cluster: # clu_get_info -state S:NOT_CONF:UP:UP:UP:NOT_CONF:NOT_CONF:NOT_CONF:NOT_CONF FILES
Run-time configuration file for each cluster member. SEE ALSO
Functions: clu_get_info(3) Files: rc.config(8) TruCluster Server Administration Guide clu_get_info(8)
All times are GMT -4. The time now is 03:37 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy