Sponsored Content
Full Discussion: Curl Loop help
Homework and Emergencies Homework & Coursework Questions Curl Loop help Post 302984916 by kjcraig77 on Wednesday 2nd of November 2016 03:09:07 PM
Old 11-02-2016
Curl Loop help

1. The problem statement, all variables and given/known data:
Here is my assignment.
Parse the /home/access_log and /home/access_.xz files
Create a sorted list of ip address with no duplicates
Pass each ip address to ipinfo.io with curl
Create a logfile that has the ip address,lat,long
Save the data file and script to the inclass dir
Submit your script to canvas email_first_part_lab8.txt

I am trying to figure out how to create a loop to cat all the access logs then curl all the ips in that log to the site ive been provided and then redirect that ouput to a text file. I am still rather ignorant to shell scripting.

2. The attempts at a solution (include all code and scripts):

Here are my scripts to cat the files

Code:
cat /home/access_log | awk '{ print $1 }' | sort | uniq -c | sort -n

Code:
xzcat /home/access_log-20160917.xz | awk '{ print $1 }' | sort | uniq -c | sort -n

Code:
xzcat /home/access_log-20161005.xz | awk '{ print $1 }' | sort | uniq -c | sort -n

And my sample curl with a single ip
Code:
curl --header "X-Forwarded-For: 137.226.113.7" ipinfoio

(cant post urls yet)



4. Modesto junior college, Manetca (CA), US, dale Phillips, and CSCI-210 (Link to Course):




Note: Without school/professor/course information, you will be banned if you post here! You must complete the entire template (not just parts of it).
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

curl

Aren't there any way to download files as below? For example, I want to download all .html files under the root directory of unix.com/ curl -O https://www.unix.com/*.html This won't work, but please tell me the way to do this. Well, the best way is to get the file list of the directory, but i... (6 Replies)
Discussion started by: Euler04
6 Replies

2. AIX

use of curl

Hi guys , need some help I have to transfer a file from my aix box to another server using ftps protocol, how can i achieve this using curl preferably or any other utility. Thanks Sandeep (0 Replies)
Discussion started by: jayawantsandeep
0 Replies

3. Shell Programming and Scripting

use of curl

I have to transfer a file from my aix box to another server using ftps protocol, how can i achieve this using curl preferably or any other utility. Regards Sandeep (0 Replies)
Discussion started by: jayawantsandeep
0 Replies

4. Shell Programming and Scripting

Curl command

Hello, I try to take out of the command curl the info of the time... time curl --write \"Downloaded %{time total}\" -o ...... but having some problem. (1 Reply)
Discussion started by: protocomm
1 Replies

5. Solaris

Curl Installation

Hi, Has anyone installed curl on solaris? I tried installing curl-7.19.4 without any effect. Here's what i have done so far: Copied the package curl-7.19.4-sol8-sparc-local under a certain dir. Tried to add the path in PATH variable. But when I type curl it says bash: curl: command not found.... (2 Replies)
Discussion started by: King Nothing
2 Replies

6. Shell Programming and Scripting

using curl in a script

Hello everyone I'm currently using a script to upload files (different file names, but same date) to an ftp server (see below) #!/bin/sh # Set the variables HOST=<host> USER=<user> PASSWD=<password> ftp -i -n $HOST <<END_SCRIPT user ${USER} ${PASSWD} mput... (0 Replies)
Discussion started by: soliberus
0 Replies

7. Shell Programming and Scripting

Help with cURL

Hi all; first of all i need to clarify that i am new to apache2 server configuration and for some needs i want to transfer some files using curl to web directory,so please bear with me: following is the command i m running to transfer file to my web directory: curl -T "q"... (4 Replies)
Discussion started by: arien001
4 Replies

8. Shell Programming and Scripting

awk loop using array:wish to store array values from loop for use outside loop

Here's my code: awk -F '' 'NR==FNR { if (/time/ && $5>10) A=$2" "$3":"$4":"($5-01) else if (/time/ && $5<01) A=$2" "$3":"$4-01":"(59-$5) else if (/time/ && $5<=10) A=$2" "$3":"$4":0"($5-01) else if (/close/) { B=0 n1=n2; ... (2 Replies)
Discussion started by: klane
2 Replies

9. Shell Programming and Scripting

Loop to curl multiple ips to ipinfo.io

I am rather new to shell scripting and currently taking a linux course. Im having some troubles writing a loop to curl multiple ips in mutiple access logs to the site ipinfo.io and push the output to a text file for easy viewing and removing duplicates. So far i have these simple lines cat... (1 Reply)
Discussion started by: kjcraig77
1 Replies

10. Shell Programming and Scripting

HELP - loop a curl command with different variables from input file

Hi guys! Kind of new to bash scripting and now I'm stuck. I need to curl with these variables: "{ \"nodename\": \"$1\", \"ipaddress\": \"$2\", \"poolname\": \"$3\", \"port\": \"$4\", \"loadbalancer\" : \"$5\" }" and my input_file.txt contains server001 10.10.10.01 serverpool1 80... (4 Replies)
Discussion started by: yort
4 Replies
qmail-users(5)							File Formats Manual						    qmail-users(5)

NAME
qmail-users - assign mail addresses to users OVERVIEW
The file /var/lib/qmail/users/assign assigns addresses to users. For example, =joe.shmoe:joe:503:78:/home/joe::: says that mail for joe.shmoe should be delivered to user joe, with uid 503 and gid 78, as specified by /home/joe/.qmail. Assignments fed to qmail-newu will be used by qmail-lspawn to control qmail-local's deliveries. See qmail-newu(8). A change to /var/lib/qmail/users/assign will have no effect until qmail-newu is run. STRUCTURE
/var/lib/qmail/users/assign is a series of assignments, one per line. It ends with a line containing a single dot. Lines must not contain NUL. SIMPLE ASSIGNMENTS
A simple assignment is a line of the form =local:user:uid:gid:homedir:dash:ext: Here local is an address; user, uid, and gid are the account name, uid, and gid of the user in charge of local; and messages to local will be controlled by homedir/.qmaildashext. If there are several assignments for the same local address, qmail-lspawn will use the first one. local is interpreted without regard to case. WILDCARD ASSIGNMENTS
A wildcard assignment is a line of the form +loc:user:uid:gid:homedir:dash:pre: This assignment applies to any address beginning with loc, including loc itself. It means the same as =locext:user:uid:gid:homedir:dash:preext: for every string ext. A more specific wildcard assignment overrides a less specific assignment, and a simple assignment overrides any wildcard assignment. For example: +:alias:7790:2108:/var/lib/qmail/alias:-:: +joe-:joe:507:100:/home/joe:-:: =joe:joe:507:100:/home/joe::: The address joe is handled by the third line; the address joe-direct is handled by the second line; the address bill is handled by the first line. SEE ALSO
qmail-pw2u(8), qmail-newu(8), qmail-lspawn(8) qmail-users(5)
All times are GMT -4. The time now is 01:19 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy