Use requests_mock to test requests logic

When Testing http requests in Python, if we run agains real endpoint, sometimes it will be as slow as turtle, and also have a chance to slam it. it's a good idea if we could mock http endpoint and control what we want to return to test our logic, then requests_mock come to my front.

in simple use case, we could just use request Mocker , the Mocker will do all the ground work for us assume we have code below

def get_my_ip()->Dict:
    response = requests.get("http://lumtest.com/myip.json")
    return response.json()

when we want to test it in pytest, we could use Mocker

def test_get_my_ip():
    with requests_mock.Mocker() as m:
        m.get("http://lumtest.com/myip.json", json={"key": "value"})
        res = get_my_ip()
        assert res["key"]== "value"

or using the decorator

@requests_mock.Mocker(kw='mock')
def test_get_my_ip(**kwargs):
    from ipa.crawler import ipa_fetcher
    kwargs['mock'].get("http://lumtest.com/myip.json", json={"key": "value"})
    res = ipa_fetcher.get_my_ip()
    assert res["key"] == "value"

when we want more control on request matching or dynamic result, we need to use use mock adapter, to use mock adapter, we have to change the original method implementation to accept http session as parameter

def get_my_ip(session: Session, url: str) -> Dict:
    response = session.get(url)
    return response.json()

to test this type of code, we could implement as below

def test_get_my_ip():
    from ipa.crawler import ipa_fetcher
    import requests
    session = requests.Session()
    adapter = requests_mock.Adapter()
    session.mount('mock://', adapter)
    adapter.register_uri("GET", "mock://lumtest.com/myip.json", json={"key": "value"})
    res = ipa_fetcher.get_my_ip(session, "mock://lumtest.com/myip.json")
    assert res["key"] == "value"

with the adapter object, we could use regex to match url and provide dynamic response to the request.