Appengine-counter is a ShardedCounter implementation for use in Google Appengine. It offers strongly consistent increment/decrement functionality while maintaining high-throughput via on-the-fly shard configuration. Appengine-counter uses memcache for fast counter retrieval, all the while being fully backed by the GAE Datastore for incredible durability and availability.
Appengine-counter is patterned off of the following article from developer.google.com, but uses Objectify for improved maintainability.
The rationale for a ShardedCounter is as follows (quoted from the above linked Google article):
When developing an efficient application on Google App Engine, you need to pay attention to how often an entity is updated. While App Engine's datastore scales to support a huge number of entities, it is important to note that you can only expect to update any single entity or entity group about five times a second. That is an estimate and the actual update rate for an entity is dependent on several attributes of the entity, including how many properties it has, how large it is, and how many indexes need updating. While a single entity or entity group has a limit on how quickly it can be updated, App Engine excels at handling many parallel requests distributed across distinct entities, and we can take advantage of this by using sharding."
Thus, when a datastore-backed counter is required (i.e., for counter consistency, redundancy, and availability) we can increment random Counter shards in parallel and achieve a high-throughput counter without sacrificing consistency or availability. For example, if a particular counter needs to support 100 increments per second, then the application supporting this counter could create the counter with approximately 20 shards, and the throughput could be sustained (this is because, per the above quote, any particular entity group in the appengine HRD can support ~5 updates/second).
Durable
Counter values are stored in the Google Appengine HRD Datastore for data durability and redundancy. Once an increment or decrement is recorded by the datastore, it's there for good.
Available
Since counters are backed by the appengine datastore and appengine itself, counter counts are highly available.
Atomic
Counter increment/decrement operations are atomic and will either succeed or fail as a single unit of work.
High Performance
Appengine datastore entity groups are limited to ~5 writes/second, so in order to provide a high-throughput counter implementation, a particular counter's 'count' is distributed amongst various counter-shard entities. Whenever an increment operation is peformed, one of the available shards is chosen to have its count incremented. In this way, counter throughput is limited only by the number of shards configured for each counter.
Smart Caching
Counter values are cached in memcache for high-performance counter reads, and increment/decrement operations update memache so you almost never have to worry about stale counts.
Growable Shards --> Higher Throughput
Increasing the number of shards for a particular counter will increase the number of updates/second that the system can handle. Using appengine-counter, any counter's shard-count can be adjusted in real-time using the appengine datastore viewer. The more shards, the higher the throughput for any particular counter.
Optional Transactionality
By default, counter increment/decrement operations do not happen in an existing datastore transaction. Instead, a new transaction is always created, which allows the counter to be atomically incremented without having to worry about XG-transaction limits (currently 25 entity groups per Transaction). However, sometimes it's necessary to increment a counter inside of an XG transaction, and appengine-counter allows for this.
Async Counter Deletion
Because some counters may have a large number of counter shards, counter deletion is facilitated in an asynchronous manner using a TaskQueue. Counter deletion is eventually consistent, although the counter-status will reflect the fact that a counter is being deleted.
Note: The current release of this library is not compatible with Objectify versions prior to version 5.0.3, and it works best with Objectify version 5.1.x. See the changelog for previous version support.
To get started, please see the instructions and details in the Getting Started page.
To learn more about using appengine-counter, please see the Usage page.
Version 2.0.0
Version 1.2.0
Version 1.1.2
Version 1.1.1
Version 1.1.0
Version 1.0.2
Version 1.0.1
Version 1.0.0
Instacount Inc. David Fuelling
Copyright 2016 Instacount Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.