dcsimg
December 8, 2016
Hot Topics:

Tips for MongoDB WiredTiger Performance Tuning

By Dharshan Rangegowda, founder of ScaleGrid.io.

MongoDB 3.0 introduced the concept of pluggable storage engines. Currently, there are a number of storage engines available for Mongo: MMAPV1, WiredTiger, MongoRocks, TokuSE, and so forth. Each engine has its own strengths and you can select the right engine based on the performance needs and characteristics of your application.

Starting with MongoDB 3.2.x, WiredTiger is the default storage engine. WiredTiger is the most popular storage engine for MongoDB and marks a significant improvement over the existing default MMAPv1 storage engine in the following areas:

  • Performance & Concurrency: For most workloads, WiredTiger offers much better performance than MMapV1. The WiredTiger storage engine is designed for modern multicore architectures and is better able to leverage multiple cores than MMAPv1. MMAPv1 uses Collection level locking, so if you have concurrent operations updating a single collection, your throughput will be limited. WiredTiger uses document-level locking, which results in enhanced concurrency and throughput. Typical applications should see a 5x-10x improvement in throughput by switching to WiredTiger.
  • Compression & encryption: In MMapv1, the in memory and on-disk representation of data needs to be the same. Hence, it does not support encryption at rest or compression. WiredTiger, on the other hand, supports both encryption at rest and compression.
  • Index prefix compression: WiredTiger uses prefix compression to store Indexes—identical prefixes are stored only once. This results in smaller indexes and consequently results in lower RAM usage for indexes.

In the rest of this article, I'll present some of the parameters you can tune to optimize the performance of WiredTiger on your server.

Working with WiredTiger Cache Size

The size of the cache is the single most important knob for WiredTiger. By default, MongoDB 3.x reserves 50% (60% in 3.2) of the available memory for its data cache. Although the default works for most applications, it is worthwhile to try tuning this number to achieve the best possible performance for your application. The size of the cache should be big enough to hold the working set of your application.

Mongo1
Figure 1: The WiredTiger cache size

MongoDB also needs additional memory outside of this cache for aggregations, sorting, connection management, and the like, so it is important to make sure you leave MongoDB with enough memory to do its work. If not, there is a chance MongoDB can get killed by the OS Out of memory (OOM) killer.

The first step is to understand the usage of your cache with the default settings. Use the following command to get your cache usage statistics:

db.serverStatus().wiredTiger.cache

Here is an example of output from calling the WiredTiger cache command:

{
   "tracked dirty bytes in the cache" : 409861,
   "tracked bytes belonging to internal pages in
      the cache" : 738956332,
   "bytes currently in the cache" : 25769360777,
   "tracked bytes belonging to leaf pages in the cache"
      : 31473298388,
   "maximum bytes configured" : 32212254720,
   "tracked bytes belonging to overflow pages
      in the cache" : 0,
   "bytes read into cache" : 29628550664,
   "bytes written from cache" : 34634778285,
   "pages evicted by application threads" : 0,
   "checkpoint blocked page eviction" : 102,
   "unmodified pages evicted" : 333277,
   "page split during eviction deepened the tree" : 0,
   "modified pages evicted" : 437117,
   "pages selected for eviction unable to be evicted"
      : 44825,
   "pages evicted because they exceeded the in-memory
      maximum" : 74,
   "pages evicted because they had chains of deleted
      items" : 33725,
   "failed eviction of pages that exceeded the in-memory
      maximum" : 1518,
   "hazard pointer blocked page eviction" : 34814,
   "internal pages evicted" : 21623,
   "maximum page size at eviction" : 10486876,
   "eviction server candidate queue empty when topping
      up" : 8235,
   "eviction server candidate queue not empty when
      topping up" : 3020,
   "eviction server evicting pages" : 191708,
   "eviction server populating queue, but not evicting
      pages" : 2996,
   "eviction server unable to reach eviction goal" : 0,
   "pages split during eviction" : 8821,
   "pages walked for eviction" : 157970002,
   "eviction worker thread evicting pages" : 563015,
   "in-memory page splits" : 52,
   "percentage overhead" : 8,
   "tracked dirty pages in the cache" : 9,
   "pages currently held in the cache" : 1499798,
   "pages read into cache" : 2260232,
   "pages written from cache" : 3018846
}

The first number to look at is the percentage of the cache that is dirty. If the percentage is high, increasing your cache size might improve your performance. If your application is read heavy, you can also track the "bytes read into cache" parameter. If this parameter remains constantly high, increasing your cache size might improve your read performance.

The cache size can be changed dynamically without restarting the server by using the following command:

db.adminCommand( { "setParameter": 1, "wiredTigerEngineRuntimeConfig":
   "cache_size=xxG"})

If you would like the custom cache size to be persistent across reboots, you also can add the config instruction to the conf file:

wiredTiger:
   engineConfig:
      cacheSizeGB: xx

Controlling WiredTiger Read and Write Tickets

Mongo2
Figure 2: Read and write tickets

WiredTiger uses tickets to control the number of read/write operations simultaneously processed by the storage engine. The default value is 128 and works well for most cases. If the number of tickets falls to 0, all subsequent operations are queued, waiting for tickets. Long-running operations might cause the number of tickets available to decrease, reducing the concurrency of your system. For example, if your read tickets are decreasing, there is a good chance that there are a number of long running unindexed operations. If you would like to find out which operations are slow, there are third-party tools available. You can tune your tickets up/down depending on the needs of your system and determine the performance impact.

You can check the usage of your tickets by using the following command:

db.serverStatus().wiredTiger.concurrentTransactions

Here is a sample output

{
   "write" : {
      "out" : 0,
      "available" : 128,
      "totalTickets" : 128
   },
   "read" : {
      "out" : 3,
      "available" : 128,
      "totalTickets" : 128
   }
}

You can change the number of read & write tickets dynamically without restarting your server by using the following commands:

db.adminCommand( { setParameter: 1,
   wiredTigerConcurrentReadTransactions: xx } )
db.adminCommand( { setParameter: 1,
   wiredTigerConcurrentWriteTransactions: xx } )

Once you've made your changes, monitor the performance of your system to ensure that it has the desired effect.

About The Author

Dharshan Rangegowda is the founder of ScaleGrid.io, where he leads products such as ScaleGrid, a MongoDB hosting and management solution to manage the lifecycle of MongoDB on public and private clouds, and Slow Query Analyzer, a solution for finding slow operations within MongoDB. He can be reached at @dharshanrg.

 

*** This article was contributed ***


Tags: MongoDB, database, Concurrency, cache, encryption, Storage Engine, WiredTiger, MMAPv1, ticket




Comment and Contribute

 


(Maximum characters: 1200). You have characters left.

 

 


Enterprise Development Update

Don't miss an article. Subscribe to our newsletter below.

Sitemap | Contact Us

Thanks for your registration, follow us on our social networks to keep up-to-date
Rocket Fuel