Quick problem statement:
How to read/extract data from a big-big file.
Details:
We are having a big big problemo in the work we are working at. We are using solaris plarform E25.
There is a very big file created somewhere around 200 million records anad the lenght of each record is more than 1000 columns. The data in these columns is separated by semicolon.
About file:
the start of the file always starts with a 01 or a " " space. The records are more like header-details. the row starting with 01 - implies a header line. the ones below it are details. Data columns are separated by semicolon. Data Column 5 will remain same between header and detail rows.
What I need to do:
1) extract a bunch of header-detail records. For example I need to every 50000 header-detail kind of data rows into a file.
I tired the SED command
sed -n 100,200002p -f fileabc.txt
However the performance is not upto the mark. the problem with sed is that even if it is instructed to copy only 100 to 200002 rows, it still scans the entire file.
When I tried SED with the entire file it took couple of days to run. Thats too much. I need better options?
Is there a way to make this operation run in parallel?
Is there a shell command which helps to copy only the specific rows and down not scan the entire file?
Hello Guys,
I want to create a file system dedicated for an application installation. But there is no space in volume group to create a new logical volume. There is enough space in other logical volume which is being mounted on /var.
I know we can use that logical volume and create a virtual... (2 Replies)
I have a program that output the ownership and permission on each directory and file on the server to a csv file. I am getting error message
when I run the program. The program is not outputting to the csv file.
Error:
the file access permissions do not allow the specified action
cannot... (2 Replies)
Hi,
I am looking for a ready shell script that can help in loading and validating a high volume (around 4 GB) .Dat file . The data in the file has to be validated at each of its column, like the data constraint on each of the data type on each of its 60 columns and also a few other constraints... (2 Replies)
Hi,
LTO5 Data cartridge has 1.5 TB (1500GB) native capacity but when we are taking our 1.44 TB (1475 GB) filesystem backup using backupby filename on these data cartridges it does not fully finish on one cartridge instead it requires another volume to backup the remaining files. I am unable to... (11 Replies)
Hi,
I am new to unix. I am working on Red Hat Linux and side by side on AIX also. After reading the concepts of Storage, I am now really confused regarding the terminologies
1)Physical Volume
2)Volume Group
3)Logical Volume
4)Physical Partition
Please help me to understand these concepts. (6 Replies)
Background
-------------
The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files.
File-1
------
Contains 50,000 rows with 2 fields in each row, separated by pipe.
Row structure is like Object_Id|Object_Name, as following:
111|XXX
222|YYY
333|ZZZ
... (6 Replies)
Hi all,
I have a requirement to create a Header &Trailer for a file which is having 20 millions of records.
If I use the following method, i think it will take more time.
cat "Header"> file1.txt
cat Data_File>>file1.txt
cat "Trailer">>file1.txt
since second CAT command has to read all... (4 Replies)
hello everyone,
well I have a file which contains data, I want to add the data on hourly basis, like my file contains data for 24 hours, (so a total of 1440 ) lines.
Now i want to add the data on hourly basis to get average values.
like if I use (head) command it is ok for first go, but... (5 Replies)
Hi!
Can anyone help me on how I can do a basic check on the Unix filesystems / physical volumes and logical volumes?
What items should I check, like where do I look at in smit? Or are there commands that I should execute?
I need to do this as I was informed by IBM that there seems to be... (1 Reply)
Hi all,
I have a problem with vxvm volume which is mirror with two disks. when i am try to increase file system, it is throwing an ERROR: can not allocate 5083938 blocks, ERROR: can not able to run vxassist on this volume.
Please find a sutable solutions.
Thanks and Regards
B. Nageswar... (0 Replies)