Since our last post here in July, we have released several new versions for both Koko Analytics and Koko Analytics Pro. Most of the work was aimed at making the plugins more reliable on busy sites.
Much of the work sits behind the scenes. We changed how pageviews are handled after they arrive. We also reduced crawler load on public dashboards, tightened password protection in Pro, and fixed a few smaller bugs.
These changes matter most when there is more work than usual. A newsletter might send a burst of traffic, a post may land on the front page of Reddit, or WP-Cron may have a backlog to clear.
How Koko Analytics stores pageviews
When someone visits a page, Koko Analytics adds a short record to a file on your server. We call this file the tracking buffer. Using the buffer avoids a database update for every single pageview.
At regular intervals, a WordPress scheduled task reads the buffer, groups matching records, and stores the totals in the database. This process is called aggregation. Handling the database work in bulk keeps pageview collection lightweight.
Most aggregation runs have a modest buffer to process. The file can grow much larger during a traffic spike or when WP-Cron does not run for a while. Once the scheduled task catches up, it has more data to hold and more database work to do.
Before version 2.5.3, Koko Analytics already read this file one line at a time. It still kept the combined counts for pages, referrers and visitors in memory until it reached the end of the file. A buffer with many different URLs or referrers could therefore result in high memory use and a large final database query.
Processing the buffer in chunks
Starting with version 2.5.3, Koko Analytics saves the totals after every 10,000 records. It also splits database inserts for pages and referrers into batches of no more than 500 rows.
The 10,000 records are a chunk size, not a tracking limit. If the buffer contains 250,000 records, the plugin processes all of them in 25 chunks. After saving one chunk, it clears the temporary page and referrer totals from memory and continues with the next one.
Splitting the database writes into groups of 500 also avoids building one very large SQL statement on sites with many different pages or referrers.
The exact memory savings depend on the traffic in each buffer. A site with thousands of distinct URLs has more data to group than a site where everyone visits the same page. In either case, the plugin now clears its working data at regular points instead of letting it grow until the entire file has been read.
Koko Analytics Pro follows the same approach
Koko Analytics Pro reads its data from the same buffer. Depending on the features in use, it may also collect custom events, countries, browsers, devices, operating systems and UTM campaign data.
Core now tells extensions whenever it finishes a chunk. Pro 2.4.7 uses that signal to save its own totals and clear them from memory. Its database writes are split into batches of 500 rows as well.
The extra reports no longer leave large queries waiting at the end of the aggregation run. As before, all processing happens on your server and no visitor data is sent to an outside service.
Keeping crawlers out of endless dashboard URLs
We also changed how navigation works on public dashboards. The date picker and table pagination used to contain normal links. Those links are useful to people, but they also give crawlers a long list of date ranges and table pages to visit.
A person may check last week, open the next page of referrers, and stop there. A crawler can continue through every link it finds. It may also request the same dashboard view with the query parameters in a different order, creating duplicate entries in a page cache.
Since version 2.5.3, the date and pagination controls are buttons enhanced with JavaScript. Visitors can use them as before, but crawlers no longer see a trail of links to follow. The plugin also puts query parameters in a consistent order and only accepts GET and HEAD requests on the public dashboard.
This cuts down on duplicate dashboard requests and avoids filling a page cache with several versions of the same view.
Koko Analytics Pro 2.4.7 also changes how password-protected dashboards handle access. It stores access in a signed, HTTP-only cookie for the browser session. The dashboard page and its REST API now use the same access check, and password-protected responses tell public caches not to store them.
Other reliability fixes
The recent releases also include smaller fixes that make tracking more reliable:
- Browsers without support for
navigator.sendBeaconnow fall back tofetch()when sending pageviews. - Tracking requests reject control characters that could break a line in the buffer file.
- The aggregation schedule now runs every 65 seconds instead of every 60 seconds, reducing the chance of scheduling conflicts.
- Pro database migrations stop when creating a table fails, instead of marking the migration as complete.
- Turning off every email-report schedule in Pro now correctly disables the reports.
Updating Koko Analytics
There is nothing to configure after updating. Koko Analytics starts processing the buffer in chunks as part of its normal scheduled task.
Koko Analytics 2.5.3 is available on WordPress.org and through the plugin updates screen in WordPress. Customers with an active Koko Analytics Pro license can download version 2.4.7.
Sites using password-protected public dashboards should update both plugins together, because the access-control changes require Core 2.5.3 and Pro 2.4.7 or newer.
Leave a Reply