Release Notes¶

This page attempts to document all non-trivial user-visible changes in each Bionic release. (“Non-trivial” means, for example, we won’t include fixes to typos in documentation.)

Versioning Scheme¶

The Bionic project follows a modified version of Semantic Versioning (“SemVer”), using a MAJOR.MINOR.PATCH version number. We try to avoid backwards-incompatible changes (even though SemVer doesn’t require this until version 1.0). However, since Bionic is still in a fairly early stage, we’d like to retain the option of making breaking changes to rarely-used features of Bionic. Thus, our policy is not to make any backwards-incompatible change before version 1 unless we’ve discussed it with known Bionic users beforehand. Currently all known Bionic users work at Square – if you’re not at Square but would like to be included in this group, let us know on our issue tracker.

We also tend to release new functionality with PATCH releases. (In SemVer this would happen in MINOR releases). We tend to issue a MINOR release when we’ve accumulated a significant amount of new functionality; when we make a breaking change (as described above); or when we make a Cache Compatibility Change (as described below).

Once we reach 1.0, we plan to follow SemVer exactly.

Types of Changes¶

For each release, we list the following types of change (in this order):

New Bugs: Notable bugs (i.e., failure to behave the way the documentation says it should) introduced by this release.
Breaking Changes: Any other non-backwards-compatible changes in behavior. Includes any difference in any returned value, thrown exception class, or value passed to a user-provided function. Does not include performance changes; bug fixes; changes in log output, error messages, or warnings; changes in visualizations; changes in documentation or docstrings; changes to features marked as experimental; changes when using Bionic in a way that the documentation tells you not to; or gracefully-handled changes to the cache data format (see next entry). We try to avoid breaking changes outside of major version releases.
Cache Compatibility Changes: Any case where this release of Bionic cannot read cached data generated by a previous release but handles the situation gracefully by automatically recomputing (and re-caching) values instead of loading old ones. If Bionic doesn’t handle this gracefully, this is marked as a bug instead.
Deprecated Features: Features which are still supported but may be removed in a future version. (This removal will count as a breaking change.) If a feature is deprecated, it will be noted in the appropriate docstring and Bionic will emit a warning message when the feature is used.
New Features: New capabilities which need to be explicitly activated by the user, generally through new functions or new arguments to existing functions.
Improvements: Any improvement that doesn’t require code changes from the user to activate, such as improved performance or clearer logging.
Documentation Changes: Significant changes to Bionic’s documentation; does not include changes associated with anything appearing in these notes (those are already assumed to be documented) or small corrections.
Development Changes: Significant changes to Bionic’s development process, such as changes to our Pytest configuration or our Continuous Integration (“CI”).

0.8.0 (Jul 6, 2020)¶

Cache Compatibility Changes¶

This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.

New Features¶

Bionic flows can now be run in parallel, taking advantage of multiple CPUs at once.
Memoization can be globally disabled with the core_memoize_by_default entity, which means you can opt-in which entities are memoized instead of opting out.
Bionic now allows entities to have no caching at all. Previously each entity needed to be cached either in memory or on disk; now both of these can be disabled, in which case it will be recomputed each time it’s needed.
GeoPandas DataFrames can now be serialized and cached. Bionic will do this automatically when an entity function returns a value of the appropriate type, but it can also be explicitly controlled with the new @geodataframe protocol.
Bionic now provides an API for listing and deleting cached artifacts.

Documentation Changes¶

This Release Notes page was added!

Development Changes¶

Updated release process to use GitHub releases.
Moved CI and releases from Travis to GitHub Actions.
CI tests are now run using Python 3.8 (in addition to 3.7 and 3.6).
Add flake8-print, flake8-fixme to CI, and configure flake8 to warn on unused imports.

0.7.1 (May 11, 2020)¶

New Features¶

The @picklable decorator now accepts a pickle_protocol_version argument, which can be used to control the data format used by Python’s Pickle module.

Improvements¶

“Unset” entity (entities that are declared but have no value set) are now handled more cleanly. They now appear in the DAG visualization; if an entity value can’t be computed because one of its ancestors is unset, the exception message now describes the exact problem and the name of the problematic ancestor; and the @gather decorator now handles “missing” values more consistently by treating them as an empty set of values.
Bionic now uses version 4 of the Pickle data format by default, so objects larger than 4 GB can be serialized without crashing.

Bug Fixes¶

The Protocol Interface now matches what is documented (it had drifted out of sync).

0.7.0 (Apr 7, 2020)¶

Cache Compatibility Changes¶

This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.

New Features¶

Flow.render_dag can now output the DAG as SVG in addition to existing formats. When SVG is used, entity docstrings appear as tooltips. SVG is the new default format for rendering in Jupyter Notebooks.
The @changes_per_run decorator was added; this tells Bionic that a function is non-deterministic and should be re-run for each instance of a Flow.

Improvements¶

Exceptions originating in user-defined entity functions are now reported more clearly and include the name of the entity.
Cache invalidation now takes into account the hashed of immediate dependencies (not just top-level ancestors), reducing the number of unnecessary invalidations and recomputations.
The exception message for using protocol decorators with positional arguments (such as @bn.protocol.frame('parquet')), which is not allowed, is now clearer.

Bug Fixes¶

The failure to load cached data from older versions of Bionic, introduced in version 0.6.4, was fixed.

Documentation Changes¶

Some missing decorators were added to the API documentation.
The installation instructions were updated to work correctly on Zsh.

Development Changes¶

Adopted the Black code formatter and configured CI to run it automatically.

0.6.4 (Mar 3, 2020)¶

New Bugs¶

This release crashed when attempting to load cached data from older versions of Bionic. This was fixed in version 0.7.0.

Cache Compatibility Changes¶

This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.

New Features¶

The @docs decorator was added, allowing separate docstrings to be associated with multiple entities generated by the same function.
The @path protocol was added, allowing Path values to work cleanly with Bionic’s cache.

Improvements¶

The Bionic cache directory can now be renamed or moved without breaking it.

Deprecated Features¶

Attaching a docstring to a function that generates multiple entities (using @outputs) is deprecated; instead, use the new @docs decorator to specify a seperate docstring for each entity.

Development Changes¶

Travis config was updated to always use the most recent versions of all dependencies.
Regression tests were added to check that Bionic is capable of loading cached data from the previous versions. (This was intended to catch regressions like the one introduced in version 0.5.7, but unfortunately was too late to catch the one introduced in this version.)

0.6.3 (Dec 24, 2019)¶

Bug Fixes¶

Bionic no longer attempts to initialize Matplotlib in a special way on OS X, which was needed for versions of Matplotlib earlier than 3.1, but caused crashes for 3.1 and later.

0.6.2 (Dec 13, 2019)¶

Improvements¶

Entity-specific documentation is now visible in the docstrings for the shorthand get and set methods, as in help(flow.get.my_entity_name).

0.6.1 (Nov 13, 2019)¶

Bug Fixes¶

The failure to load cached data from older versions of Bionic, introduced in version 0.5.7, was fixed.

0.6.0 (Nov 7, 2019)¶

Breaking Changes¶

Bionic no longer supports any Python version below 3.6. In particular, we no longer support Python 2.

0.5.7 (Nov 6, 2019)¶

New Bugs¶

This release crashed when attempting to load cached data from older versions of Bionic. This was fixed in version 0.6.1.

New Features¶

Flow.get now has a mode argument, which can be used to retrieve the cached files for entities rather than their in-memory representations.
Flow.merge can now be called with keep values of "self" or "arg"; these do the same things as "old" and "new", respectively, but were deemed clearer. The older values are still supported but may be deprecated in the future.

Improvements¶

Bionic is now released as a universal wheel, not just a source distribution.

Deprecated Features¶

Flow.get’s fmt argument is deprecated; instead, use collection, which does the same thing. (The name fmt is confusing, since it would seem to do what the new mode argument does rather than what collection does.)
Flow.export is now deprecated; use the new mode argument of Flow.get instead.

0.5.6 (Oct 23, 2019)¶

This release had no user-facing changes.

Development Changes¶

Bionic releases are now generated automatically by CI.

0.5.5 (Oct 23, 2019)¶

This release had no user-facing changes. It was an attempt to start automatically releasing via CI, but that didn’t work.

0.5.4 (Oct 22, 2019)¶

New Features¶

The @yaml protocol was added, allowing entities to be serialized as YAML files.

Improvements¶

Bionic no longer uses certain recent features of PyYAML, allowing it to be used with PyYAML 3.13. This in turn means it can be used with versions of Apache Beam (such as 2.15.0) which require PyYAML to have a version less than 4.0.

0.5.3 (Oct 21, 2019)¶

Bug Fixes¶

Fixed a bug where Bionic would sometimes import parts of the PIL module in an order which could cause a crash.

0.5.2 (Oct 17, 2019)¶

Bug Fixes¶

Fixed the bug introduced in 0.5.0, where None values could not be cached.

0.5.1 (Oct 11, 2019)¶

New Features¶

The @pyplot decorator now accepts a savefigs_kwargs argument, which can be used to customize Matplotlib’s behavior when saving a plot.

Improvements¶

Matplotlib plots are now saved using bbox_inches="tight" by default.

Documentation Changes¶

The introductory documentation and tutorials were substantially reworded.

0.5.0 (Oct 9, 2019)¶

New Bugs¶

This release introduced a bug where Bionic would crash when attempting to cache a None value to disk. The bug was fixed in 0.5.2.

Cache Compatibility Changes¶

This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.

New Features¶

Bionic now supports Automatic Versioning to help reduce the chance of forgetting to update the @version decorator.

Improvements¶

The “Computed …” log message is now logged immediately after an entity value is computed, so that the total compute time can be more accurately estimated from log timestamps.
Reading cache files is faster now that Bionic uses the C-based PyYAML loader.

Development Changes¶

Some Pytest tests are now designated “slow”, and will only be run when the --slow option is passed to pytest.

0.4.5 (Oct 8, 2019)¶

Improvements¶

Chained exceptions are now used (only in Python 3) to report caching errors more clearly.
A clearer error message is reported when attempting to serialize a Pandas DataFrame with duplicate column names with Parquet (which doesn’t work).
Writing cache files is faster now that Bionic uses the C-based PyYAML dumper.

Bug Fixes¶

Fixed a bug causing entities using @gather to be spuriously recomputed (rather than loaded from the cache) in some situations.

0.4.4 (Oct 4, 2019)¶

New Features¶

Dask DataFrames can now be serialized and cached. Bionic will do this automatically when an entity function returns a value of the appropriate type, but it can also be explicitly controlled with the new @dask protocol.

0.4.3 (Sep 13, 2019)¶

Documentation Changes¶

Square’s CLA was added to our Contributing page.

0.4.2 (Sep 9, 2019)¶

Improvements¶

Bionic now uses IncompatibleEntityError to report problems where the user has defined multiple entities in an incompatible way. This is a subclass of ValueError, which is what we used before.

Documentation Changes¶

The Flow Merging API, released in 0.4.0, is now correctly documented.

0.4.1 (Aug 29, 2019)¶

This was our first public release as an open source project. There were no code changes from the previous release.

Earlier Releases¶

Earlier releases of Bionic were internal to Square and never released on PyPI. We might never get around to documenting them here.