MIDTERM PROGRAMMING ASSIGNMENT PREP

This section will cover assignments that will be similar to those in the midterm. 

PROGRAM 1: Zip code lookup

Covers topics:

• user inputs
• fetching web pages
• processing JSON
• lists
• dictionaries
• loops and breaking out of loops
• string functions/regex
• functions
• modules

extra credit:

• writing to files
• decorator function

not covered here but may be on the exam:

• reading from files
• traversing directories
• recursive functions
• sets
• command line arguments

​
Use following API to prompt user for a zip code number and provide back the address associated with the zip code
https://api.zippopotam.us/us/06484
program should run continuously until user enters either q or Q instead of zip code number 

the address outputted should be in the following format:
<zip code>:<place name>, <state abbreviation>
ex: 
06484:Selton, CT
(see more examples below)

additional requirements:

1. Assume only US zip codes (prefix will always be https://api.zippopotam.us/us/) and only 5 digit numeric zip codes.  The program should not break if user entered non numeric string
2. The program should not break if user entered non existing zip code, use 00000 to test https://api.zippopotam.us/us/00000. The program should return "place not found" message back to the user
3. If the HTTP API is not functional, the program should display back an error message but not fail, use following URL to test: https://api.zippopotam.usa/us/06484
4. Write unit tests to validate the test for 2 valid zip codes and the 3 requirements above
5. The zip code lookup must be in its own function
6. If you use regular expressions to validate user input, you will receive 5% extra credit
7. Assume only one place is returned by the API

Grading:

• Program should execute and do what it is asked to do ~50% of the grade, there may be some additional info I may require which you will need to add in the comments
• Edge cases coverage ~20% of the grade (these are requirements 1-3 above)
• Code cleanness ~10% (this includes well organized code/files and meaningful, appropriate comments, format function used to format output, no warnings on running the pytest)
• Decision making ~20% in the program below, this would cover for the current assignment:
  • looping logic (medium penalty -2%)
  • what was removed into the function (medium penalty -2%)
  • the api endpoint removed into own variable (light penalty -1%)
  • the API output is processed as a JSON and then as a data structure - not as a string (heavy penalty -5%)

Submission: either via git or via file upload

• Git submission as a separate folder for only midterm files +5% (practice before midterm if you have not done this)

Example output 1, valid inputs:

$ python zip_lookup.py 
Enter zip code(q to quit):06484
06484:Shelton, CT

$ python zip_lookup.py 
Enter zip code(q to quit):08754
08754:Toms River, NJ

Example output 2, invalid zipcode (ERROR message is not neccesary):

$ python zip_lookup.py 
Enter zip code(q to quit):00000
ERROR:invalid API endpoint
place not found

Example output 3, invalid user input:

Enter zip code(q to quit):ejewlrke
Invalid input ejewlrke

Example output 4, looping with the q exit option:

$ python zip_lookup.py 
Enter zip code(q to quit):08754
08754:Toms River, NJ
Enter zip code(q to quit):06318
ERROR:invalid API endpoint
place not found
Enter zip code(q to quit):06484
06484:Shelton, CT
Enter zip code(q to quit):ooooo
Invalid input ooooo
Enter zip code(q to quit):q
Exiting

Working through the solution:

​I will start with the user input and validation

user_input = input("Enter zip code(q to quit):")
print(user_input)

$ python zip_lookup.py 
Enter zip code(q to quit):06484
06484

Validate the input, I am going for the extra 5% credit and will use regex to validate the user input

import re

user_input = input("Enter zip code(q to quit):")
#beginning of the string and end of the string pattern is important
#so is adding meaningful comments
regexp = re.compile("^\d{5}$")
if not regexp.search(user_input):
	print("Invalid input {}".format(user_input))

$ python zip_lookup.py 
Enter zip code(q to quit):06484
$ python zip_lookup.py 
Enter zip code(q to quit):06484h
Invalid input 06484h

adding looping logic:

import re

#takes a zipcode string as a parameter
#returns location string
def zipcode_lookup(zipcode):


while(True):
	user_input = input("Enter zip code(q to quit):")
	if(user_input == 'q' or user_input == 'Q'):
		print("Exiting")
		break
	#beginning of the string and end of the string pattern is important
	#so is adding meaningful comments
	regexp = re.compile("^\d{5}$")
	if not regexp.search(user_input):
		print("Invalid input {}".format(user_input))

$ python zip_lookup.py 
Enter zip code(q to quit):06484
Enter zip code(q to quit):06473
Enter zip code(q to quit):r
Invalid input r
Enter zip code(q to quit):64586
Enter zip code(q to quit):q
Exiting

$ python zip_lookup.py 
Enter zip code(q to quit):Q
Exiting

Adding zip code lookup function, I am going to start with a dummy call just to make sure the function call works first

import re

#takes a zipcode string as a parameter
#returns location string
def zipcode_lookup(zipcode):
	return "echo:"+zipcode

while(True):
	user_input = input("Enter zip code(q to quit):")
	if(user_input == 'q' or user_input == 'Q'):
		print("Exiting")
		break
	#beginning of the string and end of the string pattern is important
	#so is adding meaningful comments
	regexp = re.compile("^\d{5}$")
	if not regexp.search(user_input):
		print("Invalid input {}".format(user_input))
	else:
		print(zipcode_lookup(user_input))

$ python zip_lookup.py 
Enter zip code(q to quit):06484
echo:06484
Enter zip code(q to quit):we
Invalid input we
Enter zip code(q to quit):q
Exiting

ok, that works, let's start adding a real zip code lookup
​I will add the page validation at this point to cover for the requirement #3

import re
import urllib.request

api_endpoint = "https://api.zippopotam.us/us/"

def page_exists(page):
	try:
		urllib.request.urlopen(page)
		return True
	except:
		return False

#takes a zipcode string as a parameter
#returns location string
def zipcode_lookup(zipcode):
	if(page_exists(api_endpoint+zipcode)):
		page = urllib.request.urlopen(api_endpoint+zipcode)
		print(page.read())
	else:
		print("ERROR:invalid API endpoint")
	return "echo:"+zipcode

while(True):
	user_input = input("Enter zip code(q to quit):")
	if(user_input == 'q' or user_input == 'Q'):
		print("Exiting")
		break
	#beginning of the string and end of the string pattern is important
	#so is adding meaningful comments
	regexp = re.compile("^\d{5}$")
	if not regexp.search(user_input):
		print("Invalid input {}".format(user_input))
	else:
		print(zipcode_lookup(user_input))

​with good URL:

$ python zip_lookup.py 
Enter zip code(q to quit):06484
b'{"post code": "06484", "country": "United States", "country abbreviation": "US", "places": [{"place name": "Shelton", "longitude": "-73.1294", "state": "Connecticut", "state abbreviation": "CT", "latitude": "41.3047"}]}'
echo:06484
Enter zip code(q to quit):q
Exiting

and then if I change the URL in the code to
api_endpoint = "https://api.zippopotam.usa/us/"

the program doesn't break:

$ python zip_lookup.py 
Enter zip code(q to quit):06484
ERROR:invalid API endpoint
echo:06484
Enter zip code(q to quit):q
Exiting

Now we can parse out the 
<place name>, <state abbreviation>
(Note: we covered JSON processing in lecture 6): ​https://www.nurmatova.com/lecture-6---cli-files-regex-pickle-shelve.html
​here is the sample structure for 06484 zip code

{
"post code": "06484", 
"country": "United States", 
"country abbreviation": "US", "places":
 [
  {"place name": "Shelton", 
   "longitude": "-73.1294", 
   "state": "Connecticut", 
   "state abbreviation": "CT", 
   "latitude": "41.3047"}
 ]
}

Given the requirement #7, we need the only first element from the "places"

json uses two word keys, so just to make sure I did not miss anything, I will print the output first within function:

import re
import urllib.request
import json

api_endpoint = "https://api.zippopotam.us/us/"

def page_exists(page):
	try:
		urllib.request.urlopen(page)
		return True
	except:
		return False

#takes a zipcode string as a parameter
#returns location string
def zipcode_lookup(zipcode):
	if(page_exists(api_endpoint+zipcode)):
		page = urllib.request.urlopen(api_endpoint+zipcode)
		content = page.read().decode("utf-8") #keep in mind the byte string needs to be decoded
		data = json.loads(content)
		place = data["places"][0] #[0] from requirement 7
		print("{}:{}, {}".format(zipcode, place["place name"], place["state abbreviation"]))
	else:
		print("ERROR:invalid API endpoint")
	return "echo:"+zipcode

while(True):
	user_input = input("Enter zip code(q to quit):")
	if(user_input == 'q' or user_input == 'Q'):
		print("Exiting")
		break
	#beginning of the string and end of the string pattern is important
	#so is adding meaningful comments
	regexp = re.compile("^\d{5}$")
	if not regexp.search(user_input):
		print("Invalid input {}".format(user_input))
	else:
		print(zipcode_lookup(user_input))

$ python zip_lookup.py 
Enter zip code(q to quit):06484
06484:Shelton, CT
echo:06484

And now, return it from the function:
​

import re
import urllib.request
import json

api_endpoint = "https://api.zippopotam.us/us/"

def page_exists(page):
	try:
		urllib.request.urlopen(page)
		return True
	except:
		return False

#takes a zipcode string as a parameter
#returns location string
def zipcode_lookup(zipcode):
	if(page_exists(api_endpoint+zipcode)):
		page = urllib.request.urlopen(api_endpoint+zipcode)
		content = page.read().decode("utf-8") #keep in mind the byte string needs to be decoded
		data = json.loads(content)
		place = data["places"][0] #[0] from requirement 7
	else:
		print("ERROR:invalid API endpoint")
	return "{}:{}, {}".format(zipcode, place["place name"], place["state abbreviation"])

while(True):
	user_input = input("Enter zip code(q to quit):")
	if(user_input == 'q' or user_input == 'Q'):
		print("Exiting")
		break
	#beginning of the string and end of the string pattern is important
	#so is adding meaningful comments
	regexp = re.compile("^\d{5}$")
	if not regexp.search(user_input):
		print("Invalid input {}".format(user_input))
	else:
		print(zipcode_lookup(user_input))

$ python zip_lookup.py 
Enter zip code(q to quit):06484
06484:Shelton, CT

​Now, what if we put 00000 (requirement #2)?

​we get an error

$ python zip_lookup.py 
Enter zip code(q to quit):00000
ERROR:invalid API endpoint
Traceback (most recent call last):
  File "zip_lookup.py", line 37, in <module>
    print(zipcode_lookup(user_input))
  File "zip_lookup.py", line 24, in zipcode_lookup
    return "{}:{}, {}".format(zipcode, place["place name"], place["state abbreviation"])
UnboundLocalError: local variable 'place' referenced before assignment

​let's handle that, the error is happening because even if page is invalid we are still trying to return the place (else condition)
in addition to handling that, I threw in few more handlers in case the data format is not as expected by the program

import re
import urllib.request
import json

api_endpoint = "https://api.zippopotam.us/us/"

def page_exists(page):
	try:
		urllib.request.urlopen(page)
		return True
	except:
		return False

#takes a zipcode string as a parameter
#returns location string
def zipcode_lookup(zipcode):
	if(page_exists(api_endpoint+zipcode)):
		page = urllib.request.urlopen(api_endpoint+zipcode)
		content = page.read().decode("utf-8") #keep in mind the byte string needs to be decoded
		data = json.loads(content)
		if not data:
			return "place not found"
		place = data["places"][0] #[0] from requirement 7
		if not place:
			return "place not found"
	else:
		print("ERROR:invalid API endpoint")
		return "place not found"
	return "{}:{}, {}".format(zipcode, place["place name"], place["state abbreviation"])

while(True):
	user_input = input("Enter zip code(q to quit):")
	if(user_input == 'q' or user_input == 'Q'):
		print("Exiting")
		break
	#beginning of the string and end of the string pattern is important
	#so is adding meaningful comments
	regexp = re.compile("^\d{5}$")
	if not regexp.search(user_input):
		print("Invalid input {}".format(user_input))
	else:
		print(zipcode_lookup(user_input))

$ python zip_lookup.py 
Enter zip code(q to quit):00000
ERROR:invalid API endpoint
place not found
Enter zip code(q to quit):06484
06484:Shelton, CT
Enter zip code(q to quit):08754
08754:Toms River, NJ
Enter zip code(q to quit):09657
ERROR:invalid API endpoint
place not found
Enter zip code(q to quit):01234
ERROR:invalid API endpoint
place not found
Enter zip code(q to quit):ejewlrke
Invalid input ejewlrke
Enter zip code(q to quit):q
Exiting

There is a slight problem with user input validation, it should logically be inside zipcode lookup function in the form of argument validation, so let's do that now:

import re
import urllib.request
import json

api_endpoint = "https://api.zippopotam.us/us/"

def page_exists(page):
	try:
		urllib.request.urlopen(page)
		return True
	except:
		return False

#takes a zipcode string as a parameter
#returns location string
def zipcode_lookup(zipcode):
	#beginning of the string and end of the string pattern is important
	#so is adding meaningful comments
	regexp = re.compile("^\d{5}$")
	if not regexp.search(zipcode):
		return "Invalid input {}".format(zipcode)
	
	if(page_exists(api_endpoint+zipcode)):
		page = urllib.request.urlopen(api_endpoint+zipcode)
		content = page.read().decode("utf-8") #keep in mind the byte string needs to be decoded
		data = json.loads(content)
		if not data:
			return "place not found"
		place = data["places"][0] #[0] from requirement 7
		if not place:
			return "place not found"
	else:
		print("ERROR:invalid API endpoint")
		return "place not found"
	return "{}:{}, {}".format(zipcode, place["place name"], place["state abbreviation"])

while(True):
	user_input = input("Enter zip code(q to quit):")
	if(user_input == 'q' or user_input == 'Q'):
		print("Exiting")
		break
	print(zipcode_lookup(user_input))

$ python zip_lookup.py 
Enter zip code(q to quit):06484
06484:Shelton, CT
Enter zip code(q to quit):08754
08754:Toms River, NJ
Enter zip code(q to quit):ooooo
Invalid input ooooo
Enter zip code(q to quit):00000
ERROR:invalid API endpoint
place not found
Enter zip code(q to quit):q
Exiting

Moving onto unit tests, requirement #4, we covered unit tests (pytest) in the lecture 5:
www.nurmatova.com/lecture-5---functions-variable-scope-modules.html

I create a pytest test in the same folder where my code is located:

​

However, I run into some issues:

Issue #1:

====================================== ERRORS =======================================
______________________ ERROR collecting test_zipcode_lookup.py ______________________
test_zipcode_lookup.py:1: in <module>
    from zip_lookup import zipcode_lookup
zip_lookup.py:38: in <module>
    user_input = input("Enter zip code(q to quit):")
../../../../opt/anaconda3/lib/python3.7/site-packages/_pytest/capture.py:700: in read
    "pytest: reading from stdin while output is captured!  Consider using `-s`."
E   OSError: pytest: reading from stdin while output is captured!  Consider using `-s`.
---------------------------------- Captured stdout ----------------------------------
Enter zip code(q to quit):

This happens, because of the import statement, as you remember, when you import things from another module, everything that is not inside a function gets executed. 
Python's way of dealing with that is declaring a main function and checking if the code is being executed in the main execution thread

if __name__ == '__main__':
	main()

​and another problem is a warning 

================================= warnings summary ==================================
zip_lookup.py:19
  /Users/gulanurmatova/Desktop/python/lecture7/midterm_prep/zip_lookup.py:19: DeprecationWarning: invalid escape sequence \d
    regexp = re.compile("^\d{5}$")

-- Docs: https://docs.pytest.org/en/latest/warnings.html
!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!
============================ 1 warning, 1 error in 0.12s ============================

this is because I forgot to use a raw string in my regular expression, so let's fix that, now our code looks like this:

import re
import urllib.request
import json

api_endpoint = "https://api.zippopotam.us/us/"

def page_exists(page):
	try:
		urllib.request.urlopen(page)
		return True
	except:
		return False

#takes a zipcode string as a parameter
#returns location string
def zipcode_lookup(zipcode):
	#beginning of the string and end of the string pattern is important
	#so is adding meaningful comments
	regexp = re.compile(r'^\d{5}$')
	if not regexp.search(zipcode):
		return "Invalid input {}".format(zipcode)
	
	if(page_exists(api_endpoint+zipcode)):
		page = urllib.request.urlopen(api_endpoint+zipcode)
		content = page.read().decode("utf-8") #keep in mind the byte string needs to be decoded
		data = json.loads(content)
		if not data:
			return "place not found"
		place = data["places"][0] #[0] from requirement 7
		if not place:
			return "place not found"
	else:
		print("ERROR:invalid API endpoint")
		return "place not found"
	return "{}:{}, {}".format(zipcode, place["place name"], place["state abbreviation"])

def main():
	while(True):
		user_input = input("Enter zip code(q to quit):")
		if(user_input == 'q' or user_input == 'Q'):
			print("Exiting")
			break
		print(zipcode_lookup(user_input))

if __name__ == '__main__':
	main()

​pytest executes successfully and the program still runs

$ pytest
================================ test session starts ================================
platform darwin -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /Users/gulanurmatova/Desktop/python/lecture7/midterm_prep
plugins: hypothesis-5.5.4, arraydiff-0.3, remotedata-0.3.2, openfiles-0.4.0, doctestplus-0.5.0, astropy-header-0.1.2
collected 1 item                                                                    

test_zipcode_lookup.py .                                                      [100%]

================================= 1 passed in 1.11s =================================
(base) Gulas-MacBook-Pro:midterm_prep gulanurmatova$ python zip_lookup.py 
Enter zip code(q to quit):06484
06484:Shelton, CT
Enter zip code(q to quit):00000
ERROR:invalid API endpoint
place not found
Enter zip code(q to quit):Q
Exiting

The above program, if submitted via git in its own folder along with the pytest, will get you 110% on the programming assignment

There will be an extra credit you can do to get another 20% extra
Here is an example of what extra credit may look like:

Create a decorator function, and decorate the zipcode lookup function
The decorator function should log every function call with input and output into a log.txt file

​an example of solution is:

import re
import urllib.request
import json

api_endpoint = "https://api.zippopotam.us/us/"

def loggable(func):
	def inner(arg):
		s = func(arg)
		with open("log.txt", "a") as log_file:
			log_file.write("Input:{}\n".format(arg))
			log_file.write("Output:{}\n".format(s))
		return s
	return inner #return the decorator function


def page_exists(page):
	try:
		urllib.request.urlopen(page)
		return True
	except:
		return False

#takes a zipcode string as a parameter
#returns location string
@loggable
def zipcode_lookup(zipcode):
	#beginning of the string and end of the string pattern is important
	#so is adding meaningful comments
	regexp = re.compile(r'^\d{5}$')
	if not regexp.search(zipcode):
		return "Invalid input {}".format(zipcode)
	
	if(page_exists(api_endpoint+zipcode)):
		page = urllib.request.urlopen(api_endpoint+zipcode)
		content = page.read().decode("utf-8") #keep in mind the byte string needs to be decoded
		data = json.loads(content)
		if not data:
			return "place not found"
		place = data["places"][0] #[0] from requirement 7
		if not place:
			return "place not found"
	else:
		print("ERROR:invalid API endpoint")
		return "place not found"
	return "{}:{}, {}".format(zipcode, place["place name"], place["state abbreviation"])


def main():
	while(True):
		user_input = input("Enter zip code(q to quit):")
		if(user_input == 'q' or user_input == 'Q'):
			print("Exiting")
			break
		print(zipcode_lookup(user_input))

if __name__ == '__main__':
	main()

That generates following log file in the current directory

Input:06484
Output:06484:Shelton, CT
Input:oooo
Output:Invalid input oooo
Input:00000
Output:place not found
Input:06518
Output:06518:Hamden, CT