trjhtr

Python3's async/await syntax is great, but it does create a divide between libraries which are async-based and those which are not. For example, boto3 (AWS API library) currently doesn't work with async. There is a separate project, aiobotocore, attempts to recreate some of this functionality in an async context.

I have been thinking for a while about how to create HTTP API clients which can be run async and sync contexts. I have come up with a strategy that involves creating a separation between the logic (preparation of requests, interpretation of responses), and the sending. The logic is implemented as a generator function, which yields out an object which represents the request, and receives back an object representing the response. This generator is "run" by a runner function, which does the actual sending, and may be sync or async.

You can imagine that if, for example, boto3 had been written this way, it would be easy to re-use the bulk of the code to make an async version, rather than having to do an async rewrite.

I have included a toy example below. I would welcome comments.

from typing import Iterable, NamedTuple, Dict, Any, Optional

import requests
from aiohttp import ClientSession

URL_TEMPLATE = "https://api.icndb.com/jokes/id/"


class Request(NamedTuple):
 method: str
 url: str
 json: Optional[Dict[str, Any]] = None


class Response(NamedTuple):
 status: int
 json: Optional[Dict[str, Any]] = None


def get_joke(id: int) -> Iterable[Request]:

 response = yield Request("GET", URL_TEMPLATE.format(id=id))

 try:

 if response.status != 200:
 raise JokeApiError("API request failed")

 data = response.json
 if data.get("type") == 'NoSuchQuoteException':
 raise NoSuchJoke(data.get("value", ""))
 if data.get("type") != "success":
 raise JokeApiError("API request failed")

 raise Return(data["value"]["joke"])

 except (JokeApiError, Return, StopIteration):
 raise
 except Exception as e:
 raise JokeApiError() from e


def call_api_sync(it):
 try:
 for req in it:
 response = requests.request(req.method, req.url, json=req.json)
 it.send(
 Response(status=response.status_code, json=response.json())
 )
 except Return as e:
 return e.value


async def call_api_async(it):
 async with ClientSession() as session:
 try:
 for req in it:
 async with session.request(
 method=req.method,
 url=req.url,
 json=req.json) as res:
 json = await res.json()
 response = Response(status=res.status, json=json)
 it.send(response)
 except Return as e:
 return e.value


class JokeApiError(Exception):
 pass


class NoSuchJoke(JokeApiError):
 pass


class Return(Exception):

 def __init__(self, value):
 self.value = value