Release Notes

This page attempts to document all non-trivial user-visible changes in each Bionic release. (“Non-trivial” means, for example, we won’t include fixes to typos in documentation.)

Versioning Scheme

The Bionic project follows a modified version of Semantic Versioning (“SemVer”), using a MAJOR.MINOR.PATCH version number. We try to avoid backwards-incompatible changes (even though SemVer doesn’t require this until version 1.0). However, since Bionic is still in a fairly early stage, we’d like to retain the option of making breaking changes to rarely-used features of Bionic. Thus, our policy is not to make any backwards-incompatible change before version 1 unless we’ve discussed it with known Bionic users beforehand. Currently all known Bionic users work at Square – if you’re not at Square but would like to be included in this group, let us know on our issue tracker.

We also tend to release new functionality with PATCH releases. (In SemVer this would happen in MINOR releases). We tend to issue a MINOR release when we’ve accumulated a significant amount of new functionality; when we make a breaking change (as described above); or when we make a Cache Compatibility Change (as described below).

Once we reach 1.0, we plan to follow SemVer exactly.

Types of Changes

For each release, we list the following types of change (in this order):

  • New Bugs: Notable bugs (i.e., failure to behave the way the documentation says it should) introduced by this release.

  • Breaking Changes: Any other non-backwards-compatible changes in behavior. Includes any difference in any returned value, thrown exception class, or value passed to a user-provided function. Does not include performance changes; bug fixes; changes in log output, error messages, or warnings; changes in visualizations; changes in documentation or docstrings; changes to features marked as experimental; changes when using Bionic in a way that the documentation tells you not to; or gracefully-handled changes to the cache data format (see next entry). We try to avoid breaking changes outside of major version releases.

  • Cache Compatibility Changes: Any case where this release of Bionic cannot read cached data generated by a previous release but handles the situation gracefully by automatically recomputing (and re-caching) values instead of loading old ones. If Bionic doesn’t handle this gracefully, this is marked as a bug instead.

  • Deprecated Features: Features which are still supported but may be removed in a future version. (This removal will count as a breaking change.) If a feature is deprecated, it will be noted in the appropriate docstring and Bionic will emit a warning message when the feature is used.

  • New Features: New capabilities which need to be explicitly activated by the user, generally through new functions or new arguments to existing functions.

  • Improvements: Any improvements (other than bug fixes) that don’t require code changes from the user to activate, such as improved performance or clearer logging.

  • Bug Fixes: Any changes in Bionic’s behavior that bring it closer to what the existing documentation would lead one to expect.

  • Documentation Changes: Significant changes to Bionic’s documentation; does not include changes associated with anything appearing in these notes (those are already assumed to be documented) or small corrections.

  • Development Changes: Significant changes to Bionic’s development process, such as changes to our Pytest configuration or our Continuous Integration (“CI”).

0.11.1 (Nov 14, 2022)

Bug Fixes

  • Fixed an issue where automatic versioning erroneously treated classes as having changed if the module they were defined in was run as __main__. (This bug was similar to the one fixed in the previous version, but not the same one.)

Cache Compatibility Changes

  • This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.

0.11.0 (Mar 23, 2021)

Bug Fixes

  • Fixed an issue with hashing of strings containing invalid unicode.

  • Fixed an issue where automatic versioning erroneously treated classes as having changed if the module they were defined in was run as __main__.

  • Fixed a bug in Bionic’s automatic versioning where Bionic would warn about dynamic imports even when warning suppression was requested. The warning messages themselves were also improved.

  • Fixed a bug where Bionic would incorrectly suppress versioning warnings for functions that hadn’t been decorated with the @version decorator.

Improvements

  • Added a line to the SVG DAG visualization mouse hover tooltips indicating whether an entity is persisted.

  • Clarified an error message that occurs when users pass a single argument to @bn.outputs instead of using @bn.output. The error message now provides more information about what is expected and, in the case when a user passes a single argument, asks if the user meant to use @bn.output instead.

Cache Compatibility Changes

  • This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.

0.10.0 (Feb 23, 2021)

New Features

Improvements

  • When a function returns multiple entities (using the @outputs decorator), those entities and the function itself are now all visualized with the same color.

  • When “assisted” or “fully automatic” versioning is enabled, Bionic now inspects not just entity functions and values, but also any functions they reference and the classes of entity values.

  • Fixed an issue where sets were serialized non-deterministically, which could lead to downstream values being spuriously recomputed. (Unfortunately this fix doesn’t help with other objects that happen to contain sets.)

Bug Fixes

  • Fixed a bug where Bionic could crash when using parallel execution after manually deleting some persisted artifacts (such as with the Cache API).

0.9.2 (Oct 26, 2020)

Bug Fixes

  • The previous release introduced a bug where files were downloaded from GCS more often than necessary; this is fixed.

Development Changes

  • Our CI now also runs all GCS tests using the real GCS.

0.9.1 (Oct 15, 2020)

New Bugs

  • This release downloaded files from GCS more frequently than necessary (i.e., even when a local copy was already present).

Improvements

  • Bionic no longer requires gsutil for GCS caching, and as a result, GCS caching now works with Python 3.8.

0.9.0 (Oct 8, 2020)

New Features

  • Flow now has a Flow.reload method that updates the flow instance to use reloaded modules. It is similar to Flow.reloading, but modifies the flow instance in place.

Cache Compatibility Changes

  • This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.

Deprecated Features

  • Bionic no longer supports Matplotlib version 3.2.x, since that version can cause crashes on Mac OS when using multiprocessing. Versions 3.1.x and 3.3+ are still supported.

Improvements

  • When a function returns multiple entities (using the @outputs decorator), it now appears in Bionic’s DAG visualization as a separate node, with the individual entity nodes depending on it.

  • Bionic now reuses some artifact hash information instead of recomputing it, allowing persisted files to be loaded slightly more quickly.

Bug Fixes

  • Fixed an issue where non-persistable entities could be spuriously recomputed even when their values weren’t directly needed.

  • Fixed an issue where caching fails when cache directory paths contain whitespaces.

  • Fixed an issue where reloading of flows that merge other flows from other modules does not work correctly.

  • Fixed a bug where assisted versioning fails to detect code changes in non-persistable entities.

Documentation

  • Fixed broken link in the documentation for the FileCopier class.

Development Changes

  • The pytest flag for activating parallel execution tests has been renamed from --all-execution-modes to --parallel.

  • Our CI no longer runs each test on each Python version. Tests are now split into two groups: fast “baseline” tests, which are run once for each version; and slower “non-baseline” tests, which are randomly sharded among the three tested versions.

0.8.3 (Jul 23, 2020)

Deprecated Features

  • The utility function bionic.util.init_basic_logging has been moved to bionic.utils.misc.init_basic_logging. However, the original name is still supported for now. (The only reason we mention this here and provide a deprecation path is because this function is currently mentioned in our documentation. Other internal functions are regularly moved and renamed without comment.)

New Features

  • Persistence can be globally disabled with the core__persist_by_default entity, which means you can opt-in which entities are persisted instead of opting out.

  • JSON-serializable values are now serialized and cached using the JSON format (instead of the Pickle format). Bionic will do this automatically when an entity function returns a JSON-able value, but it can also be explicitly controlled with the new @json protocol.

Improvements

  • Bionic’s cache now makes fewer round-trip calls to the storage system (local disk or GCS) while reading and writing data. This might (or might not) improve performance if your connection to GCS is slow.

Bug Fixes

  • The previous release introduced a bug where Bionic would fail to recognize directory artifacts when they were stored in GCS, and would just recompute the values instead. (Most artifacts are stored as a single file, so this mainly affected the @dask and @path protocols.)

Development Changes

  • When parallel tests are not enabled, the extra tests are ignored, not skipped.

0.8.2 (Jul 10, 2020)

Improvements

  • Bionic now gracefully handles the situations where a cached artifact file is deleted but the corresponding metadata entry is not. (It now deletes the invalid metadata file and computes a new artifact and metadata entry.)

Bug Fixes

  • Fixed a bug in FlowBuilder.add_case: if the ordering of the entity names changed from case to case, some values would sometimes get assigned to the wrong entity.

  • Parallel execution (introduced in 0.8.0) had a bug in logging where log messages were dropped (with a warning) when any argument to the log message was unpickleable.

  • The cache API (introduced in 0.8.0) had a bug where if two cache entries point to the same artifact, deleting one of them could leave the other in a bad state.

0.8.1 (Jul 6, 2020)

Documentation Changes

  • Fixed some typos in the documentation.

0.8.0 (Jul 6, 2020)

Cache Compatibility Changes

  • This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.

New Features

  • Bionic flows can now be run in parallel, taking advantage of multiple CPUs at once.

  • Memoization can be globally disabled with the core__memoize_by_default entity, which means you can opt-in which entities are memoized instead of opting out.

  • Bionic now allows entities to have no caching at all. Previously each entity needed to be cached either in memory or on disk; now both of these can be disabled, in which case it will be recomputed each time it’s needed.

  • GeoPandas DataFrames can now be serialized and cached. Bionic will do this automatically when an entity function returns a value of the appropriate type, but it can also be explicitly controlled with the new @geodataframe protocol.

  • Bionic now provides an API for listing and deleting cached artifacts.

Documentation Changes

  • This Release Notes page was added!

Development Changes

  • Updated release process to use GitHub releases.

  • Moved CI and releases from Travis to GitHub Actions.

  • CI tests are now run using Python 3.8 (in addition to 3.7 and 3.6).

  • Add flake8-print, flake8-fixme to CI, and configure flake8 to warn on unused imports.

0.7.1 (May 11, 2020)

New Features

  • The @picklable decorator now accepts a pickle_protocol_version argument, which can be used to control the data format used by Python’s Pickle module.

Improvements

  • “Unset” entity (entities that are declared but have no value set) are now handled more cleanly. They now appear in the DAG visualization; if an entity value can’t be computed because one of its ancestors is unset, the exception message now describes the exact problem and the name of the problematic ancestor; and the @gather decorator now handles “missing” values more consistently by treating them as an empty set of values.

  • Bionic now uses version 4 of the Pickle data format by default, so objects larger than 4 GB can be serialized without crashing.

Bug Fixes

0.7.0 (Apr 7, 2020)

Cache Compatibility Changes

  • This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.

New Features

  • Flow.render_dag can now output the DAG as SVG in addition to existing formats. When SVG is used, entity docstrings appear as tooltips. SVG is the new default format for rendering in Jupyter Notebooks.

  • The @changes_per_run decorator was added; this tells Bionic that a function is non-deterministic and should be re-run for each instance of a Flow.

Improvements

  • Exceptions originating in user-defined entity functions are now reported more clearly and include the name of the entity.

  • Cache invalidation now takes into account the hashed of immediate dependencies (not just top-level ancestors), reducing the number of unnecessary invalidations and recomputations.

  • The exception message for using protocol decorators with positional arguments (such as @bn.protocol.frame('parquet')), which is not allowed, is now clearer.

Bug Fixes

  • The failure to load cached data from older versions of Bionic, introduced in version 0.6.4, was fixed.

Documentation Changes

  • Some missing decorators were added to the API documentation.

  • The installation instructions were updated to work correctly on Zsh.

Development Changes

  • Adopted the Black code formatter and configured CI to run it automatically.

0.6.4 (Mar 3, 2020)

New Bugs

  • This release crashed when attempting to load cached data from older versions of Bionic. This was fixed in version 0.7.0.

Cache Compatibility Changes

  • This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.

New Features

  • The @docs decorator was added, allowing separate docstrings to be associated with multiple entities generated by the same function.

  • The @path protocol was added, allowing Path values to work cleanly with Bionic’s cache.

Improvements

  • The Bionic cache directory can now be renamed or moved without breaking it.

Deprecated Features

  • Attaching a docstring to a function that generates multiple entities (using @outputs) is deprecated; instead, use the new @docs decorator to specify a seperate docstring for each entity.

Development Changes

  • Travis config was updated to always use the most recent versions of all dependencies.

  • Regression tests were added to check that Bionic is capable of loading cached data from the previous versions. (This was intended to catch regressions like the one introduced in version 0.5.7, but unfortunately was too late to catch the one introduced in this version.)

0.6.3 (Dec 24, 2019)

Bug Fixes

  • Bionic no longer attempts to initialize Matplotlib in a special way on OS X, which was needed for versions of Matplotlib earlier than 3.1, but caused crashes for 3.1 and later.

0.6.2 (Dec 13, 2019)

Improvements

  • Entity-specific documentation is now visible in the docstrings for the shorthand get and set methods, as in help(flow.get.my_entity_name).

0.6.1 (Nov 13, 2019)

Bug Fixes

  • The failure to load cached data from older versions of Bionic, introduced in version 0.5.7, was fixed.

0.6.0 (Nov 7, 2019)

Breaking Changes

  • Bionic no longer supports any Python version below 3.6. In particular, we no longer support Python 2.

0.5.7 (Nov 6, 2019)

New Bugs

  • This release crashed when attempting to load cached data from older versions of Bionic. This was fixed in version 0.6.1.

New Features

  • Flow.get now has a mode argument, which can be used to retrieve the cached files for entities rather than their in-memory representations.

  • Flow.merge can now be called with keep values of "self" or "arg"; these do the same things as "old" and "new", respectively, but were deemed clearer. The older values are still supported but may be deprecated in the future.

Improvements

  • Bionic is now released as a universal wheel, not just a source distribution.

Deprecated Features

  • Flow.get’s fmt argument is deprecated; instead, use collection, which does the same thing. (The name fmt is confusing, since it would seem to do what the new mode argument does rather than what collection does.)

  • Flow.export is now deprecated; use the new mode argument of Flow.get instead.

0.5.6 (Oct 23, 2019)

This release had no user-facing changes.

Development Changes

  • Bionic releases are now generated automatically by CI.

0.5.5 (Oct 23, 2019)

This release had no user-facing changes. It was an attempt to start automatically releasing via CI, but that didn’t work.

0.5.4 (Oct 22, 2019)

New Features

  • The @yaml protocol was added, allowing entities to be serialized as YAML files.

Improvements

  • Bionic no longer uses certain recent features of PyYAML, allowing it to be used with PyYAML 3.13. This in turn means it can be used with versions of Apache Beam (such as 2.15.0) which require PyYAML to have a version less than 4.0.

0.5.3 (Oct 21, 2019)

Bug Fixes

  • Fixed a bug where Bionic would sometimes import parts of the PIL module in an order which could cause a crash.

0.5.2 (Oct 17, 2019)

Bug Fixes

  • Fixed the bug introduced in 0.5.0, where None values could not be cached.

0.5.1 (Oct 11, 2019)

New Features

  • The @pyplot decorator now accepts a savefigs_kwargs argument, which can be used to customize Matplotlib’s behavior when saving a plot.

Improvements

  • Matplotlib plots are now saved using bbox_inches="tight" by default.

Documentation Changes

  • The introductory documentation and tutorials were substantially reworded.

0.5.0 (Oct 9, 2019)

New Bugs

  • This release introduced a bug where Bionic would crash when attempting to cache a None value to disk. The bug was fixed in 0.5.2.

Cache Compatibility Changes

  • This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.

New Features

  • Bionic now supports Automatic Versioning to help reduce the chance of forgetting to update the @version decorator.

Improvements

  • The “Computed …” log message is now logged immediately after an entity value is computed, so that the total compute time can be more accurately estimated from log timestamps.

  • Reading cache files is faster now that Bionic uses the C-based PyYAML loader.

Development Changes

  • Some Pytest tests are now designated “slow”, and will only be run when the --slow option is passed to pytest.

0.4.5 (Oct 8, 2019)

Improvements

  • Chained exceptions are now used (only in Python 3) to report caching errors more clearly.

  • A clearer error message is reported when attempting to serialize a Pandas DataFrame with duplicate column names with Parquet (which doesn’t work).

  • Writing cache files is faster now that Bionic uses the C-based PyYAML dumper.

Bug Fixes

  • Fixed a bug causing entities using @gather to be spuriously recomputed (rather than loaded from the cache) in some situations.

0.4.4 (Oct 4, 2019)

New Features

  • Dask DataFrames can now be serialized and cached. Bionic will do this automatically when an entity function returns a value of the appropriate type, but it can also be explicitly controlled with the new @dask protocol.

0.4.3 (Sep 13, 2019)

Documentation Changes

0.4.2 (Sep 9, 2019)

Improvements

  • Bionic now uses IncompatibleEntityError to report problems where the user has defined multiple entities in an incompatible way. This is a subclass of ValueError, which is what we used before.

Documentation Changes

  • The Flow Merging API, released in 0.4.0, is now correctly documented.

0.4.1 (Aug 29, 2019)

This was our first public release as an open source project. There were no code changes from the previous release.

Earlier Releases

Earlier releases of Bionic were internal to Square and never released on PyPI. We might never get around to documenting them here.