Cloud Framework v2

This is a significant overhaul of the Cloud Framework, modernising all of the Datastore access calls. At a summary:

General framework changes

• Applications are now constructed in Program using CloudFramework.WebApp, CloudFramework.ServiceApp, etc. in a fluent manner. This makes it much easier to get an application set up.
  • Added support for service applications, which register processors with roles via .AddProcessorWithRole. You can then choose which roles to launch the service app with on the command-line when starting the service application. This allows you to keep everything in a single process at small scale, and split things out into separate pods or deployments in Kubernetes as demand increases. The only processors supported at the moment are those that implement IPeriodicProcessor.
  • Added support for interactive CLI applications, which can access all of the framework features, and either connect to the development environment locally or production via kubectl port-forward and the Google Application Credentials on the local machine. This is intended to be used so you can write custom migration scripts and tooling to interact with your production environment.
• All Google services can be individually turned off, so if your application only uses Datastore, you can make sure none of the interfaces that rely on Pub/Sub are available.
• When running in development, the Cloud Framework will automatically start Docker Desktop and set up the required Redis and Google Cloud emulator containers for you. This means when developing an application, you just launch it from Visual Studio and all the runtime dependencies will be automatically started.
  • You can also add additional Docker containers with .UseDevelopmentDockerContainers.
• Added support for Helm-based dependencies via .UseHelm. Currently the framework won't deploy the Helm dependencies for you, but for more complex applications you can use this to point the Datastore emulator and Redis connections inside your Helm deployment.
• We've added a significant set of tests to ensure that the behaviour of the Cloud Framework matches expectations.
• Applications now wait until they're able to successfully connect to Redis before they'll start service traffic.

Datastore

• Datastore now has a much more simple, async-first API. The following functions are available via IGlobalRepository and IRepository:
  • QueryAsync
  • QueryPaginatedAsync
  • LoadAsync
  • LoadAcrossNamespacesAsync (global only)
  • CreateAsync
  • UpdateAsync
  • UpsertAsync
  • DeleteAsync
  • AllocateKeyAsync
  • GetKeyFactoryAsync
  • BeginTransactionAsync
  • CommitAsync
  • RollbackAsync
• All APIs return and work with IAsyncEnumerable and IAsyncDisposable. This means you can chain together things like QueryAsync and UpdateAsync so your application never needs to load all entities into memory to process them.
• The old APIs are available by a compatibility layer, and are all marked as deprecated. You should migrate to the new versions when possible, though we don't anticipate removing them in the short term.
• All queries are now run through QueryAsync, regardless of whether they're normal or geographic queries.
  • Queries now use LINQ expressions like x => x.someString == "value" && x.myInt == 5, rather than stringly-typed field names. This allows the compiler to verify that your queries have correct field names.
  • To perform geographic queries, use .WithinKilometers on a geopoint (LatLng) field in the query expression.
  • To sort on geographic queries, use .Nearest or .Furthest on a geopoint (LatLng) field in the order expression. Ordering of geopoint data is performed client side (since you should have already constrained your data with .WithinKilometers).
• All queries are now cached in Redis, regardless of their complexity. You no longer have to use different API methods to enable caching.
• You can now implement database migrations via IModelMigrator and calling services.AddMigration<TMigrator>(2 /* targetVersion */) in your application startup. These are automatically run by service and web applications on startup, and it makes use of the locking feature to ensure migrations don't run at the same time across different processes or containers.
• Models can now inherit from AttributedModel instead of Model, which allows you to use attributes rather that virtual overrides to return the metadata for Datastore.
  • Use [Kind("name")] on the class to set the value you would have returned from GetKind().
  • Use [SchemaVersion(2)] on the class to set the value you would have returned from GetSchemaVersion(). It defaults to 1, so you only need to set it if you're incrementing the version.
  • Use [Type(FieldType.String)] on properties to declare them as Datastore fields.
  • Use [Indexed] on properties to declare that they should be indexed.
  • Use [Default("blah")] to set the default values for fields if they're null in Datastore. This is intended to be used with non-nullable reference properties and non-nullable fields. When you use this feature, you no longer need to declare your integer fields as e.g. long? and test for null everywhere in your application - you can instead just use long and let the framework pick the default value if necessary.
    • Note that historical entities do not get the new default value in Datastore until they're loaded and saved, which means if you add a new non-nullable field with a default value, those historical entities won't be returned in query results until they've been re-saved. You can use IModelMigrator to force a re-save of entities if you need to make sure queries on the new field are accurate for existing data.
  • Use [Geopoint(2 /* hashKeyLength */)] to provide the additional data required for geopoint fields.

New: Google Cloud Secret Manager

• The framework now supports loading the appsettings.Production.json file from Google Cloud Secret Manager. This is a much more secure option than storing your appsettings as a Kubernetes secret (where it's effectively accessible by anyone that has access to that Kubernetes namespace).
• To use this feature, you'll need to:
  • Grant your application's service account the "Secret Manager Secret Accessor" and "Secret Manager Viewer" roles in IAM.
  • Add a secret called appsettings with the JSON in it inside Security -> Secret Manager in Google Cloud. You must enable notifications and create a Pub/Sub topic called appsettings-notifications. Setting up the service account to be able to push to Pub/Sub is a little complicated, see the Google Cloud documentation on this topic for instructions.
  • Setting up the notifications channel allows you to create new secret versions at any time, and all production instances of your application will automatically load the new production configuration without requiring a restart.

New: File storage

• The new IFileStorage interface provides generic file storage access.
  • In development and staging the implementation defaults to LocalFileStorage which stores files on disk.
  • In production the implementation defaults to B2NetFileStorage which stores files in Backblaze B2.

New: Typed routing

• To help with routing in ASP.NET Core, you can now use this.RedirectToAction<MyController>(nameof(MyController).ActionName). this.Action<T> also exists.

New: Miscellanous features

• AsyncEvent<T> allows you to create event handlers that have asynchronous handlers.
• ClassifyingLinqExtensions provides you with LINQ extensions to collections to classify elements with a classifier, and then run different handlers based on the classification using .AndForClassification. Classification handles are run in parallel for async enumerables, allowing you to process classified elements quickly.

Deprecated and removed interfaces

• Some interfaces have changed namespaces. You'll need to update your using declarations when upgrading.
• IErrorReporting has been entirely removed. The framework will automatically register the [Sentry](https://sentry.io/ error reporting system], which picks up errors from ILogger.LogError or when unhandled exceptions are thrown.
• ICurrentEnvironment is no longer necessary, and the interface has been removed. Everything that this interface provided is now determined via environment variables and IHostEnvironment.
• ICurrentTenantService defaults to a single tenant application, so you no longer need to implement it if you're not building multi-tenant applications.
• ICommonExceptionFactory now has a default implementation, so you no longer need to implement it if you don't want to override the exception types.
• The RedpointPubSub implementation has been removed.
• IMetricService now defaults to NullMetricService if it's not bound to the Google Cloud metrics APIs.

Stale interfaces

We haven't used the IBigQuery and IEventApi interfaces in our own applications in a long time, and while they're not deprecated and we're not removing them, you shouldn't expect to see updates to them unless that changes.