#Airflow 2.0 release notes trial#

Fix edge case in empty bundles with Dataset.streaming_split() ( #36039).

Fix DataContext not propagated properly for Dataset.streaming_split() operator.

Support sub-progress bars on AllToAllOperators with optimizer enabled ( #34997).

Cache the computed schema to avoid re-executing ( #37103).

Propagate iter stats for Dataset.streaming_split() ( #36908).

Support partial execution in Dataset.schema() with new execution plan optimizer ( #36740).

Spread map task stages by default for arg size SplitBlocks to ensure requested read parallelism is always met ( #36352).

Remove shutdown logging from StreamingExecutor ( #36408).

Skip writing a file for an empty block in Dataset.write_datasource() ( #36134).

Enable isort for data directory ( #35836).

Enforce strict mode batch format for er_batches() ( #36686).

Add option for parallelizing post-collation data batch operations in er_batches() ( #36842) ( #37260).

This allows streaming data ingestion for model training, and enables per-epoch data preprocessing.

New streaming integration of Ray Data and Ray Train.

Clean up a few checkpoint related things.

Move constants from tune/results.py to air/constants.py ( #35404).

Remove deprecated mlflow and wandb integrations ( #36860, #36899).

Fix batch format in dreambooth example ( #37102).

Remove experimental features page, add github issue instead ( #36950).

Add case for Dict batches in DummyTrainer read bytes calculation ( #36484).

Cancel pg.ready() task for pending trials that end up reusing an actor ( #35748).

Fix bug in execution for actor re-use ( #36951).

Removed actors that haven't been started shouldn't be tracked ( #36020).

Remove temporary checkpoint directories after restore ( #37173).

Fix infinite recursion in log redirection ( #36644).

Pass on KMS-related kwargs for s3fs ( #35938).

Disable Train/Tune syncing to head node ( #37142).

#Airflow 2.0 release notes trial#

Added support for restoring Results from local trial directories.

This is used under PPO algorithm by default.

RLlib: Public alpha release for the new multi-gpu Learner API that is less complex and more powerful compared to our previous solution ( blogpost).

The DatasetPipeline API is also being deprecated in favor of Dataset with streaming execution.

Data: We are introducing a new streaming integration of Ray Data and Ray Train.

Instead, this will raise an error telling you to switch to one of the recommended alternatives: cloud storage or NFS. This means that results written to different worker machines will not be directly synced to the head node. Train and Tune: Users are now expected to provide cloud storage or NFS path for distributed training or tuning jobs instead of a local path.Serve: Better streaming support - In this release, Support for HTTP streaming response and WebSockets is now on by default.