Release Notes
This page attempts to document all non-trivial user-visible changes in each Bionic release. (“Non-trivial” means, for example, we won’t include fixes to typos in documentation.)
Versioning Scheme
The Bionic project follows a modified version of Semantic Versioning (“SemVer”), using a MAJOR.MINOR.PATCH version number. We try to avoid backwards-incompatible changes (even though SemVer doesn’t require this until version 1.0). However, since Bionic is still in a fairly early stage, we’d like to retain the option of making breaking changes to rarely-used features of Bionic. Thus, our policy is not to make any backwards-incompatible change before version 1 unless we’ve discussed it with known Bionic users beforehand. Currently all known Bionic users work at Square – if you’re not at Square but would like to be included in this group, let us know on our issue tracker.
We also tend to release new functionality with PATCH releases. (In SemVer this would happen in MINOR releases). We tend to issue a MINOR release when we’ve accumulated a significant amount of new functionality; when we make a breaking change (as described above); or when we make a Cache Compatibility Change (as described below).
Once we reach 1.0, we plan to follow SemVer exactly.
Types of Changes
For each release, we list the following types of change (in this order):
New Bugs: Notable bugs (i.e., failure to behave the way the documentation says it should) introduced by this release.
Breaking Changes: Any other non-backwards-compatible changes in behavior. Includes any difference in any returned value, thrown exception class, or value passed to a user-provided function. Does not include performance changes; bug fixes; changes in log output, error messages, or warnings; changes in visualizations; changes in documentation or docstrings; changes to features marked as experimental; changes when using Bionic in a way that the documentation tells you not to; or gracefully-handled changes to the cache data format (see next entry). We try to avoid breaking changes outside of major version releases.
Cache Compatibility Changes: Any case where this release of Bionic cannot read cached data generated by a previous release but handles the situation gracefully by automatically recomputing (and re-caching) values instead of loading old ones. If Bionic doesn’t handle this gracefully, this is marked as a bug instead.
Deprecated Features: Features which are still supported but may be removed in a future version. (This removal will count as a breaking change.) If a feature is deprecated, it will be noted in the appropriate docstring and Bionic will emit a warning message when the feature is used.
New Features: New capabilities which need to be explicitly activated by the user, generally through new functions or new arguments to existing functions.
Improvements: Any improvements (other than bug fixes) that don’t require code changes from the user to activate, such as improved performance or clearer logging.
Bug Fixes: Any changes in Bionic’s behavior that bring it closer to what the existing documentation would lead one to expect.
Documentation Changes: Significant changes to Bionic’s documentation; does not include changes associated with anything appearing in these notes (those are already assumed to be documented) or small corrections.
Development Changes: Significant changes to Bionic’s development process, such as changes to our Pytest configuration or our Continuous Integration (“CI”).
0.11.1 (Nov 14, 2022)
Bug Fixes
Fixed an issue where automatic versioning erroneously treated classes as having changed if the module they were defined in was run as
__main__
. (This bug was similar to the one fixed in the previous version, but not the same one.)
Cache Compatibility Changes
This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.
0.11.0 (Mar 23, 2021)
Bug Fixes
Fixed an issue with hashing of strings containing invalid unicode.
Fixed an issue where automatic versioning erroneously treated classes as having changed if the module they were defined in was run as
__main__
.Fixed a bug in Bionic’s automatic versioning where Bionic would warn about dynamic imports even when warning suppression was requested. The warning messages themselves were also improved.
Fixed a bug where Bionic would incorrectly suppress versioning warnings for functions that hadn’t been decorated with the
@version
decorator.
Improvements
Added a line to the SVG DAG visualization mouse hover tooltips indicating whether an entity is persisted.
Clarified an error message that occurs when users pass a single argument to
@bn.outputs
instead of using@bn.output
. The error message now provides more information about what is expected and, in the case when a user passes a single argument, asks if the user meant to use@bn.output
instead.
Cache Compatibility Changes
This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.
0.10.0 (Feb 23, 2021)
New Features
Bionic can now compute entities in Google AI Platform.
Improvements
When a function returns multiple entities (using the
@outputs
decorator), those entities and the function itself are now all visualized with the same color.When “assisted” or “fully automatic” versioning is enabled, Bionic now inspects not just entity functions and values, but also any functions they reference and the classes of entity values.
Fixed an issue where sets were serialized non-deterministically, which could lead to downstream values being spuriously recomputed. (Unfortunately this fix doesn’t help with other objects that happen to contain sets.)
Bug Fixes
Fixed a bug where Bionic could crash when using parallel execution after manually deleting some persisted artifacts (such as with the Cache API).
0.9.2 (Oct 26, 2020)
Bug Fixes
The previous release introduced a bug where files were downloaded from GCS more often than necessary; this is fixed.
Development Changes
Our CI now also runs all GCS tests using the real GCS.
0.9.1 (Oct 15, 2020)
New Bugs
This release downloaded files from GCS more frequently than necessary (i.e., even when a local copy was already present).
Improvements
Bionic no longer requires gsutil for GCS caching, and as a result, GCS caching now works with Python 3.8.
0.9.0 (Oct 8, 2020)
New Features
Flow
now has aFlow.reload
method that updates the flow instance to use reloaded modules. It is similar toFlow.reloading
, but modifies the flow instance in place.
Cache Compatibility Changes
This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.
Deprecated Features
Bionic no longer supports Matplotlib version 3.2.x, since that version can cause crashes on Mac OS when using multiprocessing. Versions 3.1.x and 3.3+ are still supported.
Improvements
When a function returns multiple entities (using the
@outputs
decorator), it now appears in Bionic’s DAG visualization as a separate node, with the individual entity nodes depending on it.Bionic now reuses some artifact hash information instead of recomputing it, allowing persisted files to be loaded slightly more quickly.
Bug Fixes
Fixed an issue where non-persistable entities could be spuriously recomputed even when their values weren’t directly needed.
Fixed an issue where caching fails when cache directory paths contain whitespaces.
Fixed an issue where reloading of flows that merge other flows from other modules does not work correctly.
Fixed a bug where assisted versioning fails to detect code changes in non-persistable entities.
Documentation
Fixed broken link in the documentation for the
FileCopier
class.
Development Changes
The
pytest
flag for activating parallel execution tests has been renamed from--all-execution-modes
to--parallel
.Our CI no longer runs each test on each Python version. Tests are now split into two groups: fast “baseline” tests, which are run once for each version; and slower “non-baseline” tests, which are randomly sharded among the three tested versions.
0.8.3 (Jul 23, 2020)
Deprecated Features
The utility function
bionic.util.init_basic_logging
has been moved tobionic.utils.misc.init_basic_logging
. However, the original name is still supported for now. (The only reason we mention this here and provide a deprecation path is because this function is currently mentioned in our documentation. Other internal functions are regularly moved and renamed without comment.)
New Features
Persistence can be globally disabled with the
core__persist_by_default
entity, which means you can opt-in which entities are persisted instead of opting out.JSON-serializable values are now serialized and cached using the JSON format (instead of the Pickle format). Bionic will do this automatically when an entity function returns a JSON-able value, but it can also be explicitly controlled with the new
@json
protocol.
Improvements
Bionic’s cache now makes fewer round-trip calls to the storage system (local disk or GCS) while reading and writing data. This might (or might not) improve performance if your connection to GCS is slow.
Bug Fixes
Development Changes
When parallel tests are not enabled, the extra tests are ignored, not skipped.
0.8.2 (Jul 10, 2020)
Improvements
Bionic now gracefully handles the situations where a cached artifact file is deleted but the corresponding metadata entry is not. (It now deletes the invalid metadata file and computes a new artifact and metadata entry.)
Bug Fixes
Fixed a bug in
FlowBuilder.add_case
: if the ordering of the entity names changed from case to case, some values would sometimes get assigned to the wrong entity.Parallel execution (introduced in 0.8.0) had a bug in logging where log messages were dropped (with a warning) when any argument to the log message was unpickleable.
The cache API (introduced in 0.8.0) had a bug where if two cache entries point to the same artifact, deleting one of them could leave the other in a bad state.
0.8.1 (Jul 6, 2020)
Documentation Changes
Fixed some typos in the documentation.
0.8.0 (Jul 6, 2020)
Cache Compatibility Changes
This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.
New Features
Bionic flows can now be run in parallel, taking advantage of multiple CPUs at once.
Memoization can be globally disabled with the
core__memoize_by_default
entity, which means you can opt-in which entities are memoized instead of opting out.Bionic now allows entities to have no caching at all. Previously each entity needed to be cached either in memory or on disk; now both of these can be disabled, in which case it will be recomputed each time it’s needed.
GeoPandas DataFrames can now be serialized and cached. Bionic will do this automatically when an entity function returns a value of the appropriate type, but it can also be explicitly controlled with the new
@geodataframe
protocol.Bionic now provides an API for listing and deleting cached artifacts.
Documentation Changes
This Release Notes page was added!
Development Changes
Updated release process to use GitHub releases.
Moved CI and releases from Travis to GitHub Actions.
CI tests are now run using Python 3.8 (in addition to 3.7 and 3.6).
Add flake8-print, flake8-fixme to CI, and configure flake8 to warn on unused imports.
0.7.1 (May 11, 2020)
New Features
The
@picklable
decorator now accepts apickle_protocol_version
argument, which can be used to control the data format used by Python’s Pickle module.
Improvements
“Unset” entity (entities that are declared but have no value set) are now handled more cleanly. They now appear in the DAG visualization; if an entity value can’t be computed because one of its ancestors is unset, the exception message now describes the exact problem and the name of the problematic ancestor; and the
@gather
decorator now handles “missing” values more consistently by treating them as an empty set of values.Bionic now uses version 4 of the Pickle data format by default, so objects larger than 4 GB can be serialized without crashing.
Bug Fixes
The Protocol Interface now matches what is documented (it had drifted out of sync).
0.7.0 (Apr 7, 2020)
Cache Compatibility Changes
This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.
New Features
Flow.render_dag
can now output the DAG as SVG in addition to existing formats. When SVG is used, entity docstrings appear as tooltips. SVG is the new default format for rendering in Jupyter Notebooks.The
@changes_per_run
decorator was added; this tells Bionic that a function is non-deterministic and should be re-run for each instance of a Flow.
Improvements
Exceptions originating in user-defined entity functions are now reported more clearly and include the name of the entity.
Cache invalidation now takes into account the hashed of immediate dependencies (not just top-level ancestors), reducing the number of unnecessary invalidations and recomputations.
The exception message for using protocol decorators with positional arguments (such as
@bn.protocol.frame('parquet')
), which is not allowed, is now clearer.
Bug Fixes
The failure to load cached data from older versions of Bionic, introduced in version 0.6.4, was fixed.
Documentation Changes
Some missing decorators were added to the API documentation.
The installation instructions were updated to work correctly on Zsh.
Development Changes
Adopted the Black code formatter and configured CI to run it automatically.
0.6.4 (Mar 3, 2020)
New Bugs
This release crashed when attempting to load cached data from older versions of Bionic. This was fixed in version 0.7.0.
Cache Compatibility Changes
This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.
New Features
Improvements
The Bionic cache directory can now be renamed or moved without breaking it.
Deprecated Features
Development Changes
Travis config was updated to always use the most recent versions of all dependencies.
Regression tests were added to check that Bionic is capable of loading cached data from the previous versions. (This was intended to catch regressions like the one introduced in version 0.5.7, but unfortunately was too late to catch the one introduced in this version.)
0.6.3 (Dec 24, 2019)
Bug Fixes
Bionic no longer attempts to initialize Matplotlib in a special way on OS X, which was needed for versions of Matplotlib earlier than 3.1, but caused crashes for 3.1 and later.
0.6.2 (Dec 13, 2019)
Improvements
Entity-specific documentation is now visible in the docstrings for the shorthand
get
andset
methods, as inhelp(flow.get.my_entity_name)
.
0.6.1 (Nov 13, 2019)
Bug Fixes
The failure to load cached data from older versions of Bionic, introduced in version 0.5.7, was fixed.
0.6.0 (Nov 7, 2019)
Breaking Changes
Bionic no longer supports any Python version below 3.6. In particular, we no longer support Python 2.
0.5.7 (Nov 6, 2019)
New Bugs
This release crashed when attempting to load cached data from older versions of Bionic. This was fixed in version 0.6.1.
New Features
Flow.get
now has amode
argument, which can be used to retrieve the cached files for entities rather than their in-memory representations.Flow.merge
can now be called withkeep
values of"self"
or"arg"
; these do the same things as"old"
and"new"
, respectively, but were deemed clearer. The older values are still supported but may be deprecated in the future.
Improvements
Bionic is now released as a universal wheel, not just a source distribution.
Deprecated Features
Flow.get
’sfmt
argument is deprecated; instead, usecollection
, which does the same thing. (The namefmt
is confusing, since it would seem to do what the newmode
argument does rather than whatcollection
does.)Flow.export
is now deprecated; use the newmode
argument ofFlow.get
instead.
0.5.6 (Oct 23, 2019)
This release had no user-facing changes.
Development Changes
Bionic releases are now generated automatically by CI.
0.5.5 (Oct 23, 2019)
This release had no user-facing changes. It was an attempt to start automatically releasing via CI, but that didn’t work.
0.5.4 (Oct 22, 2019)
New Features
The
@yaml
protocol was added, allowing entities to be serialized as YAML files.
Improvements
Bionic no longer uses certain recent features of PyYAML, allowing it to be used with PyYAML 3.13. This in turn means it can be used with versions of Apache Beam (such as 2.15.0) which require PyYAML to have a version less than 4.0.
0.5.3 (Oct 21, 2019)
Bug Fixes
Fixed a bug where Bionic would sometimes import parts of the PIL module in an order which could cause a crash.
0.5.2 (Oct 17, 2019)
Bug Fixes
Fixed the bug introduced in 0.5.0, where
None
values could not be cached.
0.5.1 (Oct 11, 2019)
New Features
The
@pyplot
decorator now accepts asavefigs_kwargs
argument, which can be used to customize Matplotlib’s behavior when saving a plot.
Improvements
Matplotlib plots are now saved using
bbox_inches="tight"
by default.
Documentation Changes
The introductory documentation and tutorials were substantially reworded.
0.5.0 (Oct 9, 2019)
New Bugs
This release introduced a bug where Bionic would crash when attempting to cache a
None
value to disk. The bug was fixed in 0.5.2.
Cache Compatibility Changes
This release of Bionic cannot load cached data created by previous releases. Instead of attempting to load old an old cached value, Bionic will recompute (and re-cache) a new value.
New Features
Bionic now supports Automatic Versioning to help reduce the chance of forgetting to update the
@version
decorator.
Improvements
The “Computed …” log message is now logged immediately after an entity value is computed, so that the total compute time can be more accurately estimated from log timestamps.
Reading cache files is faster now that Bionic uses the C-based PyYAML loader.
Development Changes
Some Pytest tests are now designated “slow”, and will only be run when the
--slow
option is passed topytest
.
0.4.5 (Oct 8, 2019)
Improvements
Chained exceptions are now used (only in Python 3) to report caching errors more clearly.
A clearer error message is reported when attempting to serialize a Pandas DataFrame with duplicate column names with Parquet (which doesn’t work).
Writing cache files is faster now that Bionic uses the C-based PyYAML dumper.
Bug Fixes
Fixed a bug causing entities using
@gather
to be spuriously recomputed (rather than loaded from the cache) in some situations.
0.4.4 (Oct 4, 2019)
New Features
0.4.3 (Sep 13, 2019)
Documentation Changes
Square’s CLA was added to our Contributing page.
0.4.2 (Sep 9, 2019)
Improvements
Bionic now uses
IncompatibleEntityError
to report problems where the user has defined multiple entities in an incompatible way. This is a subclass ofValueError
, which is what we used before.
Documentation Changes
The Flow Merging API, released in 0.4.0, is now correctly documented.
0.4.1 (Aug 29, 2019)
This was our first public release as an open source project. There were no code changes from the previous release.
Earlier Releases
Earlier releases of Bionic were internal to Square and never released on PyPI. We might never get around to documenting them here.