Sponsored Content
Full Discussion: Curl Loop help
Homework and Emergencies Homework & Coursework Questions Curl Loop help Post 302984916 by kjcraig77 on Wednesday 2nd of November 2016 03:09:07 PM
Old 11-02-2016
Curl Loop help

1. The problem statement, all variables and given/known data:
Here is my assignment.
Parse the /home/access_log and /home/access_.xz files
Create a sorted list of ip address with no duplicates
Pass each ip address to ipinfo.io with curl
Create a logfile that has the ip address,lat,long
Save the data file and script to the inclass dir
Submit your script to canvas email_first_part_lab8.txt

I am trying to figure out how to create a loop to cat all the access logs then curl all the ips in that log to the site ive been provided and then redirect that ouput to a text file. I am still rather ignorant to shell scripting.

2. The attempts at a solution (include all code and scripts):

Here are my scripts to cat the files

Code:
cat /home/access_log | awk '{ print $1 }' | sort | uniq -c | sort -n

Code:
xzcat /home/access_log-20160917.xz | awk '{ print $1 }' | sort | uniq -c | sort -n

Code:
xzcat /home/access_log-20161005.xz | awk '{ print $1 }' | sort | uniq -c | sort -n

And my sample curl with a single ip
Code:
curl --header "X-Forwarded-For: 137.226.113.7" ipinfoio

(cant post urls yet)



4. Modesto junior college, Manetca (CA), US, dale Phillips, and CSCI-210 (Link to Course):




Note: Without school/professor/course information, you will be banned if you post here! You must complete the entire template (not just parts of it).
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

curl

Aren't there any way to download files as below? For example, I want to download all .html files under the root directory of unix.com/ curl -O https://www.unix.com/*.html This won't work, but please tell me the way to do this. Well, the best way is to get the file list of the directory, but i... (6 Replies)
Discussion started by: Euler04
6 Replies

2. AIX

use of curl

Hi guys , need some help I have to transfer a file from my aix box to another server using ftps protocol, how can i achieve this using curl preferably or any other utility. Thanks Sandeep (0 Replies)
Discussion started by: jayawantsandeep
0 Replies

3. Shell Programming and Scripting

use of curl

I have to transfer a file from my aix box to another server using ftps protocol, how can i achieve this using curl preferably or any other utility. Regards Sandeep (0 Replies)
Discussion started by: jayawantsandeep
0 Replies

4. Shell Programming and Scripting

Curl command

Hello, I try to take out of the command curl the info of the time... time curl --write \"Downloaded %{time total}\" -o ...... but having some problem. (1 Reply)
Discussion started by: protocomm
1 Replies

5. Solaris

Curl Installation

Hi, Has anyone installed curl on solaris? I tried installing curl-7.19.4 without any effect. Here's what i have done so far: Copied the package curl-7.19.4-sol8-sparc-local under a certain dir. Tried to add the path in PATH variable. But when I type curl it says bash: curl: command not found.... (2 Replies)
Discussion started by: King Nothing
2 Replies

6. Shell Programming and Scripting

using curl in a script

Hello everyone I'm currently using a script to upload files (different file names, but same date) to an ftp server (see below) #!/bin/sh # Set the variables HOST=<host> USER=<user> PASSWD=<password> ftp -i -n $HOST <<END_SCRIPT user ${USER} ${PASSWD} mput... (0 Replies)
Discussion started by: soliberus
0 Replies

7. Shell Programming and Scripting

Help with cURL

Hi all; first of all i need to clarify that i am new to apache2 server configuration and for some needs i want to transfer some files using curl to web directory,so please bear with me: following is the command i m running to transfer file to my web directory: curl -T "q"... (4 Replies)
Discussion started by: arien001
4 Replies

8. Shell Programming and Scripting

awk loop using array:wish to store array values from loop for use outside loop

Here's my code: awk -F '' 'NR==FNR { if (/time/ && $5>10) A=$2" "$3":"$4":"($5-01) else if (/time/ && $5<01) A=$2" "$3":"$4-01":"(59-$5) else if (/time/ && $5<=10) A=$2" "$3":"$4":0"($5-01) else if (/close/) { B=0 n1=n2; ... (2 Replies)
Discussion started by: klane
2 Replies

9. Shell Programming and Scripting

Loop to curl multiple ips to ipinfo.io

I am rather new to shell scripting and currently taking a linux course. Im having some troubles writing a loop to curl multiple ips in mutiple access logs to the site ipinfo.io and push the output to a text file for easy viewing and removing duplicates. So far i have these simple lines cat... (1 Reply)
Discussion started by: kjcraig77
1 Replies

10. Shell Programming and Scripting

HELP - loop a curl command with different variables from input file

Hi guys! Kind of new to bash scripting and now I'm stuck. I need to curl with these variables: "{ \"nodename\": \"$1\", \"ipaddress\": \"$2\", \"poolname\": \"$3\", \"port\": \"$4\", \"loadbalancer\" : \"$5\" }" and my input_file.txt contains server001 10.10.10.01 serverpool1 80... (4 Replies)
Discussion started by: yort
4 Replies
db_upgrade(1)						    BSD General Commands Manual 					     db_upgrade(1)

NAME
db_upgrade SYNOPSIS
db_upgrade [-NsV] [-h home] [-P password] file ... DESCRIPTION
The db_upgrade utility upgrades the Berkeley DB version of one or more files and the databases they contain to the current release version. The options are as follows: -h Specify a home directory for the database environment; by default, the current working directory is used. -N Do not acquire shared region mutexes while running. Other problems, such as potentially fatal errors in Berkeley DB, will be ignored as well. This option is intended only for debugging errors, and should not be used under any other circumstances. -P Specify an environment password. Although Berkeley DB utilities overwrite password strings as soon as possible, be aware there may be a window of vulnerability on systems where unprivileged users can see command-line arguments or where utilities are not able to overwrite the memory containing the command-line arguments. -s This flag is only meaningful when upgrading databases from releases before the Berkeley DB 3.1 release. As part of the upgrade from the Berkeley DB 3.0 release to the 3.1 release, the on-disk format of duplicate data items changed. To cor- rectly upgrade the format requires that applications specify whether duplicate data items in the database are sorted or not. Specifying the -s flag means that the duplicates are sorted; otherwise, they are assumed to be unsorted. Incorrectly specifying the value of this flag may lead to database corruption. Because the db_upgrade utility upgrades a physical file (including all the databases it contains), it is not possible to use db_upgrade to upgrade files where some of the databases it includes have sorted duplicate data items, and some of the databases it includes have unsorted duplicate data items. If the file does not have more than a single database, if the databases do not support duplicate data items, or if all the databases that support duplicate data items support the same style of duplicates (either sorted or unsorted), db_upgrade will work correctly as long as the -s flag is correctly specified. Otherwise, the file cannot be upgraded using db_upgrade, and must be upgraded manually using the db_dump and db_load utilities. -V Write the library version number to the standard output, and exit. It is important to realize that Berkeley DB database upgrades are done in place, and so are potentially destructive. This means that if the system crashes during the upgrade procedure, or if the upgrade procedure runs out of disk space, the databases may be left in an inconsistent and unrecoverable state. See Upgrading databases for more information. The db_upgrade utility may be used with a Berkeley DB environment (as described for the -h option, the environment variable DB_HOME, or because the utility was run in a directory containing a Berkeley DB environment). In order to avoid environment corruption when using a Berkeley DB environment, db_upgrade should always be given the chance to detach from the environment and exit gracefully. To cause db_upgrade to release all environment resources and exit cleanly, send it an interrupt signal (SIGINT). The db_upgrade utility exits 0 on success, and >0 if an error occurs. ENVIRONMENT
DB_HOME If the -h option is not specified and the environment variable DB_HOME is set, it is used as the path of the database home, as described in DB_ENV->open. SEE ALSO
db_archive(1), db_checkpoint(1), db_deadlock(1), db_dump(1), db_load(1), db_printlog(1), db_recover(1), db_stat(1), db_verify(1) Darwin December 3, 2003 Darwin
All times are GMT -4. The time now is 11:39 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy