Sponsored Content
Top Forums Shell Programming and Scripting Get only domain from url file bind Post 302960083 by omuhans123 on Tuesday 10th of November 2015 01:21:01 PM
Old 11-10-2015
Hi RudiC, I have thought about the categorization part of each line however it is taking too much processing power. I will rather do this at the end when I have summarized the files and sorted them. Thank you for the clarification on the uniq -d and sort part, I was not aware of the sort that needs to be there. Also the head that it only prints 10 lines. The last script you provided is working great and I have managed to get the commas in place now.

I however still am faced with a challenge as there a 9 million records and 3 servers, so in total 27 million records to be processed. The optimized script is now doing about 50 thousand lines an hour. As such I must find an alternative method of doing this even though it is working great, I need to be able to process these records as close as possible to real time.

Thank you RudiC, it is truly a pleasure and appreciated your assistance.
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

2. Shell Programming and Scripting

url calling and parameter passing to url in script

Hi all, I need to write a unix script in which need to call a url. Then need to pass parameters to that url. please help. Regards, gander_ss (1 Reply)
Discussion started by: gander_ss
1 Replies

3. UNIX for Dummies Questions & Answers

ReDirecting a URL to another URL - Linux

Hello, I need to redirect an existing URL, how can i do that? There's a current web address to a GUI that I have to redirect to another webaddress. Does anyone know how to do this? This is on Unix boxes Linux. example: https://m45.testing.address.net/host.php make it so the... (3 Replies)
Discussion started by: SkySmart
3 Replies

4. Windows & DOS: Issues & Discussions

How to: Linux BOX in Windows Domain (w/out joining the domain)

Dear Expert, i have linux box that is running in the windows domain, BUT did not being a member of the domain. as I am not the System Administrator so I have no control on the server in the network, such as modify dns entry , add the linux box in AD and domain record and so on that relevant. ... (2 Replies)
Discussion started by: regmaster
2 Replies

5. Web Development

Regex to rewrite URL to another URL based on HTTP_HOST?

I am trying to find a way to test some code, but I need to rewrite a specific URL only from a specific HTTP_HOST The call goes out to http://SUB.DOMAIN.COM/showAssignment/7bde10b45efdd7a97629ef2fe01f7303/jsmodule/Nevow.Athena The ID in the middle is always random due to the cookie. I... (5 Replies)
Discussion started by: EXT3FSCK
5 Replies

6. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Here is what I have so far: find . -name "*php*" -or -name "*htm*" | xargs grep -i iframe | awk -F'"' '/<iframe*/{gsub(/.\*iframe>/,"\"");print $2}' Here is an example content of a PHP or HTM(HTML) file: <iframe src="http://ADDRESS_1/?click=5BBB08\" width=1 height=1... (18 Replies)
Discussion started by: striker4o
18 Replies

7. Shell Programming and Scripting

Hit multiple URL from a text file and store result in other test file

Hi, I have a problem where i have to hit multiple URL that are stored in a text file (input.txt) and save their output in different text file (output.txt) somewhat like : cat input.txt http://192.168.21.20:8080/PPUPS/international?NUmber=917875446856... (3 Replies)
Discussion started by: mukulverma2408
3 Replies

8. Shell Programming and Scripting

Reading URL using Mechanize and dump all the contents of the URL to a file

Hello, Am very new to perl , please help me here !! I need help in reading a URL from command line using PERL:: Mechanize and needs all the contents from the URL to get into a file. below is the script which i have written so far , #!/usr/bin/perl use LWP::UserAgent; use... (2 Replies)
Discussion started by: scott_cog
2 Replies

9. UNIX for Dummies Questions & Answers

Putting the colon infront of the URL domain

I have a file like this: http://hello.com www.examplecom computer Company I wanted to keep dot (.) infront of com. to make the file like this http://hello.com www.example.com computer Company I applied this expression sed -r 's/com/.com/g'but what I get is: http://hello.com ... (4 Replies)
Discussion started by: csim_mohan
4 Replies

10. UNIX for Dummies Questions & Answers

Extracting URL with domain

I have a file like this: http://article.wn.com/view/2010/11/26/IV_drug_policy_feels_HIV_patients_Red_Cross/ http://aidsjournal.com/,www.cfpa.org.cn/page1/page2 , www.youtube.com http://seattletimes.nwsource.com/html/jerrybrewer/2013517803_brewer25.html... (1 Reply)
Discussion started by: csim_mohan
1 Replies
sortbib(1)							   User Commands							sortbib(1)

NAME
sortbib - sort a bibliographic database SYNOPSIS
sortbib [-s KEYS] database... DESCRIPTION
sortbib sorts files of records containing refer key-letters by user-specified keys. Records may be separated by blank lines, or by `.[' and `.]' delimiters, but the two styles may not be mixed together. This program reads through each database and pulls out key fields, which are sorted separately. The sorted key fields contain the file pointer, byte offset, and length of corresponding records. These records are delivered using disk seeks and reads, so sortbib may not be used in a pipeline to read standard input. The most common key-letters and their meanings are given below. %A Author's name %B Book containing article referenced %C City (place of publication) %D Date of publication %E Editor of book containing article referenced %F Footnote number or label (supplied by refer) %G Government order number %H Header commentary, printed before reference %I Issuer (publisher) %J Journal containing article %K Keywords to use in locating reference %L Label field used by -k option of refer %M Bell Labs Memorandum (undefined) %N Number within volume %O Other commentary, printed at end of reference %P Page number(s) %Q Corporate or Foreign Author (unreversed) %R Report, paper, or thesis (unpublished) %S Series title %T Title of article or book %V Volume number %X Abstract -- used by roffbib, not by refer %Y,Z Ignored by refer By default, sortbib alphabetizes by the first %A and the %D fields, which contain the senior author and date. sortbib sorts on the last word on the %A line, which is assumed to be the author's last name. A word in the final position, such as `jr.' or `ed.', will be ignored if the name beforehand ends with a comma. Authors with two-word last names or unusual constructions can be sorted correctly by using the nroff convention `' in place of a blank. A %Q field is considered to be the same as %A, except sorting begins with the first, not the last, word. sortbib sorts on the last word of the %D line, usually the year. It also ignores leading articles (like `A' or `The') when sorting by titles in the %T or %J fields; it will ignore articles of any modern European language. If a sort-significant field is absent from a record, sortbib places that record before other records containing that field. No more than 16 databases may be sorted together at one time. Records longer than 4096 characters will be truncated. OPTIONS
-sKEYS Specify new KEYS. For instance, -sATD will sort by author, title, and date, while -sA+D will sort by all authors, and date. Sort keys past the fourth are not meaningful. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWdoc | +-----------------------------+-----------------------------+ SEE ALSO
addbib(1), indxbib(1), lookbib(1), refer(1), roffbib(1), attributes(5) BUGS
Records with missing author fields should probably be sorted by title. SunOS 5.10 14 Sep 1992 sortbib(1)
All times are GMT -4. The time now is 04:45 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy