2018 Tue May 1

Continued learnings into Python, AWS, Serverless.

Python tooling

  • PEP is an acronym for Python Enhancement Proposal

  • PEP8 is a style guide for python code dating back to 2001

  • There are various lint tools that enforce PEP8; pycodestyle, pydocstyle, pyflakes, pylint, pychecker

  • flake8 is an aggregate tool tying pyflakes, mccabe and third party plugins

  • autopep8 is a tool that automatically fixes style errors (anything not conforming to pep8). Another is pep8ify.

  • There is a newer auto-formatting tool called yapf from an engineer at Google. It seems to go further than autopep8. One idea behind it is that PEP8 alone does not ensure nicely formatted code. Uses clang-format under the hood. Says itself is similar in idea to gofmt in that it aims to end "holy wars about formatting"; quote:

    if the whole codebase of a project is simply piped through YAPF whenever modifications are made, the style remains consistent throughout the project and there's no point arguing about style in every code review.

    Demo here.

  • One thing that yapf will not copy from gofmt is import formatting, so another tool exists for that, for years, called isort.

  • Python got type hints in 3.5. It just exposes type info for tools to leverage.

  • The leading tool for that appears to be mypy whose core team is paid by Dropbox.

  • Dropbox released PyAnnotate which runs a program and deduces the static types. Useful for large existing programs.

  • Finally a general intelligence tool (static analysis, autocomplete) for Python that is modern and popular is jedi.

  • Atom IDE plugin for python relies upon Python Language Server which ties all the above tools together.

Dynamodb

  • Dynamodb has the ability to observe table changes in realtime.

  • This is called Dynamodb Streams

  • AWS Lambda can be made to run on every table change.

  • Under the hood AWS Lambda polls dynamodb four times per second. This means worst-case reaction latency of 250ms.

  • Dynamodb has a TTL feature since January 2017. When keys expire dynamodb will delete them but only in an enventually consistent way. Specifically, AWS documentation says that expired entries will be deleted within 48 hours of expiration. Since TTL is based on an entry attribute in ISO standard time format, read queries can work around this limitation by using a filter operation that checks if time now is past entry ttl deadline.

  • The above workaround for reads does not help Dynamodb Streams which presumably will not know about expired entries until the eventual automatic delete by Dynamodb. There doesn't seem to be any AWS documentation suggesting otherwise.

  • relevant links: