If someone loads reloads the page displaying your e-commerce tracking, and you have no measures in place to prevent the tracking being displayed after a successful send, the data gets to GA again and you'll inflate your figures.

This is unfortunate. The Google Analytics documentation on e-commerce tracking also fails to warn you of the issue, so I am guessing that many developers make the same mistake. 

How to determine whether your sites are affected 

Some of my sites were affected by this issue because my usually razor-sharp devs had missed this scenario, probably assuming GA had some kind of protection built-in to prevent this.

To determine which sites were affected I made a custom report. You can add it to your account by clicking here.  

It can be used to show whether you're affected by this bug due to your developer failing to prevent the data being sent more than once.

If you set the reporting period to a wide time period and look at the transactions column, they should all be 1. If any are greater than 1 then your ecommerce tracking is incorrectly configured. 

If your Google Analytics implementation is set up to allow you to zero transactions by re-passing the transaction ID with null values, you'll additionally need to check that the number of transactions doesn't include this extra empty transaction. 

How to fix your code

If your sites are affected, your developers will need to modify the e-commerce tracking code on your site to prevent the same data being sent more than once. 

1. Record that the transaction has been sent

I overcame this by using a database lookup and a few code and template checks. First, I check the database to see if the transaction ID has been recorded in the database. If it hasn't, I log the transaction ID, then send the data.

(Obviously the code here is very specific to my particular set-up, which uses the as yet unreleased Measurement Protocol, Data Layers and Universal Analytics, but you should get the idea...)

2. Prevent caching

If you're using a caching system (either page, template or server) then you'll need to make sure you exclude the page which displays your e-commerce tracking.

We inadvertently broke a site's tracking by installing memcached and forgot to exclude the tracking on this page from the cache...

3. Prevent client-side reloading

On your client-side code you'll need a check to make sure a browser reload doesn't cause the code to be re-displayed. I use the PHP Smarty system for templating on the site I look after, so I've handled it there.

Again, this is the code to handle Universal Analytics tracking (not the current version of GA) but the principles are the same.

My e-commerce tracking is contained in a separate template file, but my main analytics template runs a quick check to determine whether it needs to load the e-commerce tracking and a check to determine whether it's been logged/sent already.  

If the data is new and hasn't been sent before, the master template displays the e-commerce tracking. If the data has already been sent, it registers the page view but doesn't reload the e-commerce tracking. 

As a result of this simple rule, most ecommerce teams have targets for their content producers and editors to hit which define how many new product ranges they need to add.

While it's possible to track this in a spreadsheet, it's laborious given that it's something that can be easily automated using Google Analytics. 

Step 1: Capture when products are added

To allow you to monitor which products have been added or edited, and by which editor, you can use event tracking. To do this you'll need to place the Google Analytics tracking code onto the header of your ecommerce platform's back-end templates, allowing you to track activity by staff members on this private part of your site. 

You might need to add a little server-side logic, but the overall aim is to capture the name of the staff member (usually available in their session data) and the name of the product they've added.

You can then simply write this data into the Google Analytics tracking code using your server-side language and add the variables you've captured. 

['_trackEvent', 'Content: Product added', '$product', '$author']

Which will write code like this to your tracking tag when the page is created, storing the data in the event tracking section of the Google Analytics UI. 

['_trackEvent', 'Content: Product added', 'MacBook Pro Retina', 'Matt Clarke']

If you want to track when products are edited or when blog posts are added you can use the same kind of technique.

Now, you'll be able to see the total number of products that have been added over a particular time period to see how well the team are meeting their product additions targets. 

Step 2: Creating a custom dashboard

To help you monitor all of the product additions and edits made, and to see how well individual members of the content team are performing to their targets, you can create a dashboard to help you monitor all these metrics in one place. 

I recorded total product additions by editor as a table, individual metrics for total products added, total products edited, total ranges added, total ranges edited, tables showing the individual products added and edited, and pie charts showing the performance of team members. 

You can get a copy of the dashboard here. The event category and action values in this match those I configured in my tracking code, so you will need to change these depending on the values you enter. Every week, you can get Google Analytics to send this report to you so you can check on progress. 

Step 3: Custom reports

Since the dashboard only displays a summary, I also set up some custom reports to provide additional data. I made one to record all of the products added, one to record all of the products edited, and then individual reports for each member of the content team. 

This allows me to see not only the performance of the entire team, but also how many products individual team members added and what they were. Here's an example report showing all product additions.  

The next step is in measuring the value provided by the new products that have been added, but I'll explain that in another post.  

While the general public using GA don't get to see the beta features of the UI, the Google Analytics Tracking Code functions are there for anyone to see if you grab a copy of the ga.js file and decompress it to make the JavaScript readable. 

I took a quick look at the current GATC to see what functions were in there and which ones were potential new features Google is currently testing for future releases of Google Analytics. These are a few of the things I spotted. I'm sure there are quite a few I've missed! 

_setPageGroup

The _setPageGroup function appears to work a little like a page-level custom variable and is designed to allow users to group pages according to things like their function or their author.

According to the guide in Google's help pages which explains how to Interpret Page Groups, you can categorise pages by type and authorship using a hierarchical system with new functionality due to appear in the reports later on.

This looks like a really nice new feature. Presumably, given the fact that it's currently in the help system and that there's a Page Groups Best Practices Guide visible, it's probably also due to be released fairly soon.

It's been possible to achieve similar results using event tracking and page-level custom variables, as I've been doing on this blog for a couple of years, using code like this: _gaq.push(['_setCustomVar',5,'Author','Matt_Clarke',3]);, however, the new _setPageGroup method appears to add a lot more functionality to the UI through the improved separation of data.

As a quick test, I tried implementing the _setPageGroup code on this site and it appears to work. There's currently no official documentation on how to use the function, but using a similar format to custom variables, using slots, seems to work.

So, I am guessing that creating an authorship page group might work a bit like this: _gaq.push(['_setPageGroup', 3, 'Author: Matt_Clarke']);. The GATC definitely picks it up. The data gets sent to Google using the new utmpg GIF request parameter. Presumably there will also be a few other attributes you can pass to it, though.

Following a Twitter discussion on this yesterday, Yehoshua Coren (@AnalyticsNinja) reckons it won't add an extra page hit either, as it's treated in the same way as a page-level custom variable. Indeed, the Google Analytics Debugger for Chrome currently reports it as this, emphasising the underlying similarities between the two features. 

Update (July 30, 2012): Looks like Content Grouping is definitely on the way. Here's a screenshot from the recently announced Remarketing Lists feature which shows Content Grouping on the Admin control panel.  

_setAutoTrackOutbound

The _setAutoTrackOutbound function has been in the Google Analytics tracking code for ages. I can remember reading about it at least a few years ago when Phil Pearce (@philpearce) discussed it here when we were all still using the old synchronous GATC. 

The idea is that by adding this function to your JavaScript tracking code, Google Analytics will then record any clicks on outgoing links. At the moment, this is possible but you do have to either manually attach events or virtual pageviews to each link, or apply it to all external links using the DOM or jQuery.  

It's possible to send data, but as far as I know, there's no way to view the outbound links from _setAutoTrackOutbound in Google Analytics at the moment. Is the also undocumented _setHrefExamineLimit related?

_clearTrans

Could _clearTrans be a new function to remove ecommerce transactions from your data? This can be an issue when you need to process refunds and want the revenue data removed, or when you've been subject to black hat techniques which have injected data into your account. 

There is, of course, a workaround for this at the moment, but no official function yet. So perhaps this is it.  

Whatever it does, there's nothing currently written about _clearTrans and using it in a simple manner as _gaq.push(['_clearTrans','1234']); appears to add nothing to the GIF request, so presumably it either requires additional parameters or depends on the presence of an additional function.

_setMaxCustomVariables

A year or so ago, the _setMaxCustomVariables function used to work, as I've explained before in this post. By passing an integer, such as 10, to the function in your GATC, you used to be able to get more than the usual five custom variables.

However, just before Google Analytics Premium came out (which provided 50 custom variables instead of five) it stopped working. The data would still be sent, but clicking on them resulted in an error page in the UI. I can only assume that this function, which was never really announced by Google, will now be deprecated.

Caveat

Obviously, this is unreleased, undocumented, dangerous stuff, so don't use it in a production environment!  

While I'm comfortable using the command line for most stuff in MySQL, there are times when it's faster and more productive to use a GUI, and the new MySQL Workbench suite makes this quite easy. It's much like the old MySQL Administrator and MySQL Query Browser it replaces in that it can connect to MySQL over a SSH tunnel, and if anything it's now easier than ever.

While this example shows it on OS X, MySQL Workbench is cross platform and the same procedures should apply to any operating system.

Creating your tunnel

Open MySQL Workbench and click "New Connection". Enter a connection name, such as "Amazon EC2" and select "Standard TCP/IP over SSH" as the connection method.

In the SSH Hostname field you need to enter the Elastic IP for your EC2 instance. Unless you've configured SSH to use a port other than the default 22, leave the port :22 part as it is. For the username field you'll need to enter your SSH username. On Ubuntu Amazon EC2 instances the default username is "ubuntu", so enter that.

Amazon EC2 doesn't provide an SSH username and password, but instead provides a PEM certificate or key which you can store on your local machine to provide access. Click the "..." button on the SSH Key section to select your PEM key.

Again, if you've not done anything fancy with your MySQL installation, you can leave the default port as 3306 and use whatever user you've added, or connect with the root account if you're performing admin tasks. Finally, enter your default schema name and hit OK and you should have a remote connection to your MySQL database tunneled via SSH.

A few of the guides I checked suggested using MacFuse and Macfusion, however, neither of these appear to work on OS X Lion. MacFuse is now unsupported and Macfusion seemed really buggy, so I had to find an alternative method. Thankfully, it was pretty easy once I figured out what package was needed. 

Step 1: Set up SSH

In my fresh install of Mac OS X Lion I didn't have a hidden .ssh directory inside my home directory so the first thing I needed to do was open a terminal and create one: mkdir ~/.ssh

I then opened the Finder on that directory using open ~/.ssh and dragged the PEM key into the directory. It's vital now to change the permissions of the key, because if it's insecure SSH will (cleverly) not let you use it. You can do this by entering chmod 600 ~/.ssh/YourKeyName.pem

Now that the PEM key is in place, you ought to test that it works by entering ssh -i ~/.ssh/YourKeyName.pem ubuntu@1.2.3.4, where 1.2.3.4 is the IP of your EC2 instance that you've added using an Elastic IP.

Once you're sure that the key works as intended, you can store it in SSH so it can run without the need to manually use the -i flag or enter the path to the key itself. You can add the key to SSH by typing: ssh-add -K ~/.ssh/YourKeyName.pem

Now it should be possible to access your Amazon EC2 instance from the command line simply by typing ubuntu@1.2.3.4 and SSH will simply use the certificate you stored to gain access.

Step 2: Mounting the Amazon EC2 filesystem locally

To mount the Amazon EC2 filesystem on your Mac you'll need to download and install the SSHFS package from the OXSFuse project. It's free and open source and installs as a panel in System Preferences.

Now you need to create a local mount point for the Amazon EC2 instance which you can do by entering mkdir ~/AmazonEC2

In its simplest form, you can mount the EC2 instance by entering sshfs ubuntu@1.2.3.4:/var/www ~/AmazonEC2 - where ubuntu is your EC2 username and /var/www/ is the default directory you want to show on the mount point.

There are, of course, a ton of optional parameters, and you can probably configure OS X to mount this automatically at boot time if you want, but this is sufficient for basic access to an EC2 instance in a graphical environment.

Google Analytics provides some really useful campaign tracking parameters for monitoring the performance of email marketing campaigns, so you can measure how many clicks you get from them, how many visits they drive and which parts of your email are most appealing.

However, there's no recommended method for using Google Analytics to monitor your email open rate and it's not really what GA was intended for. 

Most people do it using their email service provider (ESP), which is definitely what I'd recommend. However, quite a few people seem to want to do something similar in GA, so I thought I'd explain briefly how it can be done. 

Tracking email open rate in Google Analytics

Although it's possible to send HTML emails, you can't simply stick the standard JavaScript tracking code in your email and expect it to be tracked.

Email clients (including web mail systems like Windows Live mail and Gmail) strip out JavaScript for security reasons, so it wouldn't get executed and you'd receive no data back. Instead, what you need to do is use old-school pixel tracking. 

The standard client-side Google Analytics tracking code takes data gleaned from your page using JavaScript and cookies and adds it to a GET request string appended to an invisible single pixel transparent GIF image called __utm.gif.

Each time you do something on a site with Google Analytics installed, the __utm.gif has HTTP GET parameter data appended to the end which is picked up by the Google Analytics server, parsed and then added to your GA account. 

You can take a look at the GIF request that gets sent to Google Analytics by using a tool like Firebug on Mozilla Firefox or the Google Analytics Debugger extension for Google Chrome. Here's an of some of the data which gets appended to the GIF request below. 

Using mobile Google Analytics tracking

It used to be the case that mobile phone browsers didn't always execute JavaScript, so to overcome this issue, Google helpfully provided an alternative mobile tracking code.

This didn't rely on JavaScript and simply used a GIF request instead. Instead of the GET parameters being populated by JavaScript, it utilises a server-side language, such as PHP, to populate the values and create the GET parameters required by GA. 

When a mobile device views a page containing the GIF, the server grabs the data it needs from the device and server, populates the GET parameters and sends the data to Google Analytics. 

Since email clients are a bit like old mobiles in that they don't execute JavaScript, you can use the same kind of technique to send data when an email is viewed. 

While you might be able to tweak the standard __utm.gif, I think the easiest way to utilise this technique is to use the mobile tracking code, which works using PHP. 

Setting up the mobile tracking

To get started, the first thing you'll need to do is obtain a copy of the Google Analytics tracking code for mobile devices.

To find this, click Admin > Tracking code > Advanced > PHP and you should find the PHP code required, and the MO-123456-1 web property ID required in the textarea below. 

The other vital thing to obtain is a copy of the ga.php Google Analytics mobile code, which you can download from here and install on your web server. 

Save the example code to a PHP file and load it in your browser, making sure you've pointed to the ga.php Google Analytics file you installed in the previous step.

After loading this, you should be able to see the GIF/GET request in the Google Analytics Debugger extension for Chrome and your data should appear in GA a few minutes later. 

Creating campaign tracking

What we have now is a script (ga.php) that can be loaded in a HTML image tag by a web browser and which passes a string of GET parameters back to the Google Analytics servers, and serves up a transparent GIF for the browser. 

However, since we can't run PHP inside an email, we need to manually populate the GET parameters. As well as the standard ones used by the mobile tracking, you can also utilise Google Analytics' other tracking methodology, including the campaign tracking parameters.

Don't forget that anything you include in your URL needs to be URL encoded so it doesn't break the URL. This means converting & to & and spaces to %20, among other things.  

Here's a working example which tracks a newsletter campaign, to which I've added some spacing and line breaks to improve readability. Simply copy this and replace the values with your own. 

You will need to replace the utmac value with your own mobile web property ID; the utmn value should ideally be a random number assigned to each email; the utmr is the referral field which you can leave empty; the utmp is the address of the page, to which you can pass any appropriate value; while the utmp parameter contains your URL encoded campaign parameters. 

For the campaign tracking, I used the standard utm_source (newsletter), utm_medium (email), utm_campaign (July) and utm_term (Offer1).

Now, all you need to do is add the HTML code for this GIF request to your email inside a standard HTML element, ensuring that you're pointing to the ga.php you hosted earlier.

Then stick it at the bottom of your email and send a test mailing to yourself. If you open the email, it should send a request to your pixel tracking code and pass the data to your GA account, where it should appear shortly afterwards.

Don't forget...

While it works, in theory, the results you get in practice may be different to those you get from your ESP's open-rate tracking.

This reports gross-opens and might require the utmn value to be populated with a unique numeric identifier in order to pick up unique opens.

It might not be hugely reliable without this, because if the same user repeatedly opens an email it will skew the gross opens, making you think the email has been opened by more people than it really was.

As a result, I'd recommend testing it thoroughly before you rely on the data. 

This makes email marketing a challenge, because pretty much the only way you can target them with relevant emails is to ask them to fill in a preference centre style form, and people hate filling them in. 

Relevant emails are much more likely to be opened and generate more purchases and fewer unsubscribes, so it's worth taking the effort to improve in this area.

I built a tool to do just this, and ditched the preferences centre by extracting the data I needed using RFM. Here's a quick run down of how it works. 

1. Create a segment

I created a segment for each subcategory within the taxonomy of the site. These typically use some default RFM settings I came up with based on rules related to our business and the type and value of products sold, however, I also added the ability to over-ride this and adjust the RFM settings on a per-subcategory basis. 

2. Managing segments

Sometimes it's useful to know how many customers you've got in each segment, and what the average RFM score is. You might also want to adjust the RFM settings for the segment, so I added a tool to handle this. You need to delete the data from the segment if you're changing the settings, since the RFM analysis will obviously be utilising different data.

3. Creating groups

In order to create groups of customers to target, I created a tool allowing the user to select which of the segments they wanted to include. If you want to target only owners of mountain bikes, you'd select all of the subcategories related to this and create a name for the group to identify the cluster. 

You'll also need a tool for managing the groups you've created. In this I included the ability to edit and delete them, and listed the number of customers in each one, the number contactable (due to opt-in settings) and the number for which the group is their top scoring group. 

4. Scoring customers

The complicated part is the scoring of the customers. To do this, I grabbed a copy of the site's historical basket data and the customer data and stored it on the server that does the number crunching.

The basic concept is that you create some algorithms which loop over each customer's purchases and score them in every segment using the settings you defined when each segment was created.

Obviously, this is really intensive and the data crunching can take several days for a large data set. However, you do end up with some really useful data. 

At the same time, I also do some other ecommerce data crunching, and look at things like purchase latency. For this part, I'd highly recommend getting yourself a copy of Jim Novo's excellent book Drilling Down, which covers all aspects of mining customer data using such techniques. 

5. Create your lists

The whole point of this system is that you use it to cluster customers so they're contacted with offers most appropriate to their current interests. If you spend lots of money on skiing, but go surfing in the summer, you won't want to receive skiing-related emails when it's hot outside.

When exporting lists, the tool clusters customers so they're placed in their top-scoring RFM group and don't get placed in multiple lists. When you're creating your custom emails, you simply select the lists you want to mail to, and the tool exports the email addresses of the customers included. 

I also added an RFM-powered tool to the exporter which allows you to drill-down a little further, by selecting only customers who have shopped within a certain time period, purchased a certain number of times, or spent over a specified amount.

This means you can create a highly targeted email allowing you to contact only repeat customers who have, for example, spent over £100 on shoes over the past 30 days. The next step is simply to use the exported list data and create the email targeted at the cluster. 

6. Analysing the customer data

The neat thing about having all this customer data is that you can mine it to extract other useful things, such as details on purchase latency. This can be useful for email marketing, because it means you can identify customers who are lapsing and target them specifically based on what you know about the typical behaviour of the individual and other customers who shop in a given segment.  

But oddly, it hasn't yet done the same with the vast amounts of data it holds on ecommerce retailers' product inventories through the Google Merchant Center feeds. 

Google Merchant Center feeds, if you're not familiar with them, are XML files containing vast quantities of information on a retailer's product inventory. 

What's the point? 

The existing Google Merchant Center tools are pretty limited, but tying the two together could help retailers improve the return they get from their feed. 

For Google, it would also make sense for them from a commercial point of view, because they're controversially going to start charging for inclusion on Google Products (the new name for Google Shopping or Froogle).

Much of their revenue comes from Adwords, which GA covers well, so the same strategy ought to apply equally to monitoring and optimising Google Merchant Center feeds. 

However, on a practical level for practitioner analysts and ecommerce managers, integrating the Google Merchant Center feed data with Google Analytics would have several benefits, beside the obvious one of allowing better feed optimisation. 

It would also greatly improve the rather limited ecommerce functionality in GA at present, which in my opinion is perhaps its weakest area from a practical perspective. 

1. Product conversion rates

If you're an ecommerce manager one of the key things you'll want to measure is the conversion rate of individual products, but this is not something Google Analytics really provides, which I find rather odd.

If you know the conversion rate of each product you can use it to identify those which aren't performing well, indicating potential problems with content, imagery, merchandising or price, which means you're better able to fix them. 

Both the URL of the product and its SKU are used appear in both the Google Analytics data and in the Google Merchant Center feed data, so either can be used as a primary key to join the two together. 

I've previously built a proof-of-concept application using the Google Analytics Core Reporting API and the Google Content API for Shopping which does just this, allowing me to see accurate product conversion rates for any SKU or page, without the need to do anything in Excel.

I can even pull in the product image and description and other data to see if there are issues with that. It makes the whole process of checking product content much easier.

2. Brand analysis

As an ecommerce manager, one of the other things you'll want to look at is how well different brands are performing across your site. At the moment you can do this using advanced segments, or the excellent new dashboard tools, but it's still very much a manual process and it's lacking in some information. 

However, the Google Merchant Center feed data contains a brand field, so if it were incorporated into GA, it would be able to show you the performance of every brand stocked without the need for you to manually create an advanced segment and multiple reports to examine each one. 

You'd then be able to see what proportion of revenue or visits were to different brands, see how each was performing and examine the customer segments associated with each one. It could be really useful for retailers.  

3. Feed quality analysis

The bit that will probably appeal most to Google from a commercial angle would be the incorporation of feed quality data. At the moment, you can get this (at a very basic level) in the Google Merchant Center application, but it's not brilliant. 

It would be nice to be able to see, for example, which of your products were missing key parts of information such as barcodes, and identify what difference that had to conversion rates.  

From past work on Google Merchant Center feed optimisation, I'm fairly certain that feed errors are used as a ranking factor in Google's Product search, so showing these in GA would also help retailers fix problems and spot what was happening more easily.  

4. Cost data

The other thing I've felt Google Analytics and the Google Merchant Center feeds have long required is cost data. At the moment, there's no way to see how profitable your ecommerce activity has been, or whether your Adwords campaigns broke even, turned profit or ran at a loss, unless you export the data and add your own external cost data. 

However, since the Google Merchant Center feeds are made available only to Google, it would be really simple to add an additional field containing the cost price, which would then greatly extend the functionality and usefulness of GA.

I appreciate that many people say GA isn't a financial tool, but nonetheless, having this data available within GA would be really, really handy, and I think pretty easy to introduce. 

I'm already doing something similar using the ecommerce tracking, but using server-side code to push the cost/margin data into GA rather than using the client-side code, which obviously shows confidential data in the page source.

It's particularly good for handling real time data, such as lots of inserts, updates and queries, which makes it well suited to powering API services for SaaS software and it's already being used by big sites like Craigslist and Foursquare. 

In many ways it's rather like Apache CouchDB, but it requires its own drivers (rather than using a REST API), it scales using sharding rather than replication, and it supports more datatypes. It's one of the easiest of the "big data" technologies to get started with and it works very well with PHP.

1. Install MongoDB

To install MongoDB on a Debian-based system open a terminal and enter "sudo apt-get install mongodb" to install Mongo and its dependencies. Once it's finished installing, enter "mongo" and you should see MongoDB's command line in your shell. 

2. Create MongoDB's data directory

Oddly, MongoDB uses a default data directory called /data/db but it does not create this when you install it. To create the directory get a prompt and enter "sudo mkdir -p /data/db/" then "sudo chown `id -u` /data/db".

3. Install the PHP drivers for MongoDB

To use MongoDB with PHP you need to make sure that the "mongod" server is running on your machine (just type in mongod into a shell to find out) and that you have the Pear installed. 

You can install the Pear PHP system by entering "sudo apt-get install php5-dev php5-cli php-pear". 

Once you have Pear installed, you can use it to install the MongoDB drivers using PHP Extension Community Library (PECL) simply by entering "sudo pecl install mongo". 

Once that's installed, you'll then need to manually modify your php.ini configuration file to include the line "extension=mongo.so" to ensure the Mongo drivers are loaded as a PHP language extension.

On Ubuntu, you can edit php.ini by entering "sudo gedit /etc/php5/apache2/php.ini". You'll need to restart Apache to get the module to load so enter "sudo service apache2 restart" and you should be ready to go. 

4. Creating or selecting a MongoDB database

Creating a new database is very easy in Mongo, instantiate the Mongo class with "$mongo = new Mongo();" then select the Mongo database you wish to use with "$db = $mongo->customers;". 

If a database doesn't already exist to be selected, Mongo will create one, so the above code example creates a database called customers, or if it already exists, selects it so we can use it.

5. Create a collection in your Mongo database

MongoDB doesn't use tables, but it does have an equivalent which is called a "collection". Collections are essentially collections of data files or "documents".

Extending our above script, we can select a collection called purchases from our customers database using "$collection = $db->purchases;". Again, if the collection doesn't exist, Mongo will create it. 

6. Inserting a record

To add a record to the collection you've created you use the insert() function. First you assign the data you want to add to an associative array, which I've called $object in the example below, then you add it to the collection with $collection->insert($object);

If you want to see the records you've added, you can run the find() function on your collection and assign the output to the MongoDB cursor and then loop over it to see the list of entries. 

7. Deleting a record

To delete all records matching a certain criteria, you need use the remove() function and pass it an array with the array parameters you want to match against. 

Like SQL, if you have multiple records that match the criteria MongoDB will delete all of those that match. However, if you use add the true parameter to the remove function it will only delete the first match.

8. Selecting records

To perform the MongoDB equivalent of a SQL SELECT statement you need to use the find() function. You pass an array of search parameters to find() and it returns any matching documents in the cursor. There are lots of different ways to use the find() function and there are some handy examples shown here. 

9. Updating records

The final basic thing you'll want to do is update a record, which rather helpfully is done via the update() function. However, the way it works was, to me, initially a bit confusing. 

You pass two arrays to update(). The first one contains the details on the item you want to match against (in the example below I've matched records with the name 'Steve Jobs') while the second array contains the $set key and an array of the keys/values you want to update. 

You can also add additional fields to an existing document with update(), or you can do an "upsert" in which you update an existing object or create a new one if it doesn't already exist. 

R is becoming popular with analysts who want to do a bit more with the data they export from Google Analytics. While the learning curve is initially a bit steep, there's an R package for Google Analytics which can make working with, and visualising, API data a little less challenging.

I've recently been dabbling with R and Google Analytics and thought I'd share some pointers on how to get started with some simple projects. 

1. Install R and R-XML

Besides R itself, the R for Google Analytics code also uses a couple of extra extensions for fetching the data from the webservice and for parsing the resulting XML. 

I'm a Linux user and on Ubuntu/Debian you can install these by entering: sudo apt-get install r-base r-cran-xml libcurl4-gnutls-dev libxml2-dev

2. Start R and install RCurl and XML

To install the required RCurl package you need to start R by opening a terminal and typing R then entering a command to download the package from one of the CRAN mirrors. These are much like the CPAN mirrors used in Perl, but contain R packages rather than Perl packages.

The Google Analytics R documentation and code comments suggest using the www.omegahat.com mirror, however, this domain appears to no longer be in use, so I went for Imperial College's CRAN mirror, which is at http://cran.ma.imperial.ac.uk.

You can download RCurl from your chosen mirror by entering the following command in the R command line interface: install.packages("RCurl", repos = "http://cran.ma.imperial.ac.uk")

Once you've done that, issue the command below to install the XML libraries in R: install.packages("XML", repos = "http://www.omegahat.org/R")

3. Install R Google Analytics

There's a pre-built package for using R with Google Analytics, but it's not in Cran, so you'll need to download it the RGoogleAnalytics code libraries from Google Code. However, as of September 2012, this had not been updated to work with the new Google Analytics Core Reporting API and doesn't support the namespaces required in R versions greater than 2.13, so either won't install or won't work. 

The very helpful Richard Fergie (@richardfergie) has kindly pointed out that there's an unofficial version of RGoogleAnalytics available on GitHub, which does work and should install. Thanks, Richard.  

To install RGoogleAnalytics start R (just type R in a terminal) and then enter this command (with the location and file name replaced with where ever you've put the RGoogleAnalytics tarball): install.packages("/home/matt/Downloads/RGoogleAnalytics_1.1.tar.gz",repos=NULL,type="source") 

4. Install RStudio

Next, I'd recommend installing RStudio, a very powerful graphical client available for R, which is free and open source and available for Linux, Mac and Windows. You candownload it from www.rstudio.org as a package for your OS. 

Not only does it provide simple access to the command line, but it will also display any visualisations generated within the application itself, which is nice. The full documentation for RStudio will tell you more. 

5. Load the RGoogleAnalytics library 

If you wish to install any additional packages, click the Packages tab in the lower right hand corner of the RStudio GUI and RStudio will automatically install the selected package and its dependencies with a single click. This makes extending R really, really easy. In order to use one of the packages, you need to tick the checkbox so the libraries are loaded. 

When you load the RGoogleAnalytics library, you should see it fire up the dependencies in the console window.

> library("RGoogleAnalytics")
Loading required package: RCurl
Loading required package: bitops
Loading required package: XML

6. Connect to the Google Analytics Core Reporting API

Using the source editor window at the top right of RStudio write a few lines of code to start up RGoogleAnalytics in R, pass your login credentials to the Google Analytics Core Reporting API and fetch your profile data. 

ga <- RGoogleAnalytics()
ga$SetCredentials("you@yourdomain.com", "qkjshdjkdsksk")
profiles <- ga$GetProfileData()

Highlight the text you've written and then click the "Run" button to execute this chunk of code - you should see some results in the Workspace window pane.  

Click the "Profiles" option and you should get a pop-up window containing all of the profiles for your ID and all of the table IDs for your GA tables. You'll need to grab one of the table IDs to use to query GA. 

7. Running basic GA queries

Once you have an open connection to GA, you can now compose a Google Analytics API query to pass to the Core Reporting API using the QueryBuilder() function. Here's a basic example to fetch visitors and pageviews for a given date period. 

When the query runs you should see it pause for a second while it queries the Google Analytics servers and then display the results in the console window on the bottom left had corner of RStudio.

While handy, at the moment, this isn't anything you couldn't achieve either via the Data Feed Query Explorer or via the Core Reporting API in any other language that takes your fancy. 

8. Plotting data

R has tons of features for quickly plotting data and applying statistical calculations to data sets. Basic plots are achieved using the plot() function while trend lines can be added using lines().

You can do this on your GA data with:

# Set up RGoogleAnalytics

ga <- RGoogleAnalytics()

ga$SetCredentials("YOURUSERNAMEHERE", "YOURPASSWORDHERE")

profile.data <- ga$GetProfileData()

my.table.id <- "ga:YOURTABLEIDHERE"

# Build your GA query

query <- QueryBuilder()

query$Init(

start.date = "2011-01-01",

end.date = "2011-05-20",

dimensions = "ga:week",

metrics = "ga:itemRevenue",

sort = "ga:week",

table.id = my.table.id)

# Run the query

ga.data <- ga$GetReportData(query)

head(ga.data)

# Create chart

require(stats)

plot(ga.data$data, type="o", col="blue") 

Google sets strict criteria on data quality and error limits and if you don't monitor these you risk having your feed penalised and some or all of your products de-listed.

In September 2011, Google started gradually enforcing new feed specification requirements and if you didn't update your feed then it's possible you may have experienced a traffic and revenue drop from this source.

Among the new additions were the requirement to provide data on product availability, the Google Product Category and images.

If you sell clothes and shoes there are additional things you need to provide too.

Step 1: Examine the Dashboard

Login to the Google Merchant Centre and open the Dashboard for the site you want to examine.

The Dashboard shows some charts outlining the number of products that are active, awaiting review, disapproved or expiring for the Product Search and Product Ads; the number of search clicks; a summary of data quality and the total number of items inserted.

We're concerned with three main things: the number of items inserted, the overall data quality and the number of errors in the data feeds. The more effort you put into the data quality and errors, the more products you'll get listed.

Step 2: Submit a test feed

Step 2: Submit a test feed

While the standard Google Merchant Center web interface will show you errors and warnings, it won't show them all on one page and there's no immediate option to download them all in an easily digested spreadsheet.

Therefore, the first step I'd suggest is to download your existing feed from the Data feeds page and rename it under a new name in order to create a test data feed.

On the Data feeds page click "New Test Data Feed" and upload the copy of the feed you downloaded to create a new test data feed which will allow you to check your changes and download all errors and warnings in a CSV file.

After a few minutes, if you click the list of errors on the Data feeds > Status section, you should find an option to "Download full report". Click the link and you should receive a huge CSV file to open in Excel allowing you to search for errors and warnings.

Step 3: Examine data quality  

The next step is to look at your data quality, which covers things like missing or invalid product identifiers, or long titles. The issues Google discovers in your feed will be separated into Critical Errors and Suggested Optimisations.

Critical Errors are obviously really bad and you must fix them quickly to prevent your feed being suspended.

Typical reasons for critical errors are products that have been added without the required product identifier or GTIN, which is usually a product code or barcode.

Rather unhelpfully, Google doesn't inform you by email when it discovers such issues, so you'll need to check the Data quality report regularly.

It's also worth setting up an alert in Google Analytics to inform you of traffic drops from this source. If you click the "How to fix" links, you'll find details on how you can resolve the problem.

You'll need to work through the errors one by one, examine your feed data for similar issues, and then upload it to assess what's left to fix.

Step 4: Examine data feed errors

Data feed errors are the next issue to tackle. You can examine these by viewing the reports at Data feeds > Status > View errors.

If you need to download a copy of your XML feed to examine, you can use the link on this page and then open it up in a text editor.

You can also find individual errors by using Ctrl + F within the full report you downloaded in Step 2 and searching for "error".

The item errors report on the web interface will show any products that have errors and have been de-listed. None of these will be shown on Google Shopping or Google Ads, so you'll want to rectify these first.

Typical reasons for errors are "non-unique items", which are usually different products with the same product code; image URLs which contain unusual characters such as square brackets, or items with product descriptions that contain too much text.

Step 5: Testing your changes 

Once you've edited your product content to resolve the issues the Google Merchant Center has flagged, you'll need to use your e-commerce platform to regenerate the feed.

Once you've done this, go to Data feeds > Daily upload and click the "Update and fetch now" button.

After a few minutes, you should see the new data appearing in your Status report and you'll be able to work out what work is left to complete.

Solving common errors

Any products in your feed that throw an error won't be displayed in Google Product Search or Product Ads, so fixing these should be your top priority.

"Item too big"

The item is too big error is caused by product descriptions which are over 10KB in size. Flash or JavaScript code, or just shedloads of text can push you over this limit, so you'll simply need to cut the length of the copy in the description field to fix it.

"Invalid URL in attribute"

The Invalid URL in attribute error is triggered by dodgy URLs and filenames present in the Google Merchant Centre feed. Typical causes for this are spaces in URLs that haven't been escaped with "%20", the presence of backslashes instead of forward slashes, the use of relative URLs (ie. file.html) and missing http:// at the start of the URL.

"Duplicate item"

Duplicate item errors are usually caused by product administrators inadvertently using the same product code for multiple SKUs. This is easily fixed by identifying the duplicates and adding the correct product codes, or the additional EAN/GTIN for the SKUs.

"Missing required attribute: title"

Oh dear, one of your staff has added a product without a page title, which isn't very clever, but thankfully it is easily fixed.

Solving common warnings

Warnings won't prevent products from being listed, but they do affect your overall feed quality, so might feasibly have an impact upon ranking. It makes sense to improve these where you can.

"Invalid UPC value"

The Universal Product Code (UPC) is a US barcode based on 12 numerical digits. Check that yours meets these criteria if you're supplying one. Make sure there are no non-numerical characters present. All barcodes passed to Google get validated, and it will throw an error if it finds an invalid code, which means the product won't be displayed.

"Invalid string value in attribute: description"

The description field in the feed contains the product copy for the item you're promoting. The usual reason for the "invalid string value" error is that the description is completely missing, or contains an invalid character. If the description is completely missing you'll also get another warning for "Missing recommended attribute: description".

"Missing recommended attribute: google_product_category"

The Google Product Category is a string which tells Google Shopping how to categorise the item. It's typically something like "Media > DVDs & Videos" and is based on Google's taxonomy, not yours.

However, although it's wise to add it to every item you list it's only essential for feeds in the UK, US, Germany, Japan and France which sell products in certain categories listed here.

"Missing recommended attribute: product_type"

Unlike the google_product_category, the product_type field is based on your site's taxonomy for the given item. You'd usually use this in conjunction with google_product_category to place the item in Google's taxonomy, and then provide additional breadcrumb information to further categorise the product.

"Missing recommended attribute: image_link"

The image link is a recommended field to provide, so you'll need to source an image and include the URL in your feed to get rid of this notice.

"roboted images"

This error occurs because there is a User-agent Disallow rule in your robots.txt file which is blocking Googlebot or Googlebot-Image from accessing the images required for the feed. To fix this add the following lines to your robots.txt file:

User-agent: Googlebot 

Allow: /images/

User-agent: Googlebot-Image

Allow: /images/ 

7 helpful tips for troubleshooting feeds

1. Check the Google Merchant Center Dashboard regularly to ensure your feeds are working optimally. Google won't tell you when your products or feed gets de-listed and you'll lose money if you don't check regularly.

2. Keep a close eye on the Google Merchant Center Announcements blog. This is where you'll find more information on forthcoming changes to feed requirements and how they might affect you.

3. Track the revenue you're getting from Google Merchant Centre feeds using Google Analytics campaign tracking parameters in your URLs. You can create dashboards (above), custom reports, advanced segments and alerts to help you can an eye on things.

4. If you need help diagnosing problems with your feed, check out the Google Merchant Center forums at Google Groups.

5. The errors and warnings which apply to you will depend on what you sell, and in which countries you operate. You'll need to check the guidelines at Google Merchant Center Help to find out which ones matter to you.

6. You can find GTINs that Google recognises by searching for the product in Google Shopping and clicking the "Compare Prices" button which usually appears on the first result. Scroll to the bottom and you should find the EAN you need.

7. Dodgy GTINs are among the trickiest issues to solve and your item will be rejected if you supply an invalid code. Fortunately, you can validate them yourself directly within your platform using code like this. 

8. Provide as much detail as possible and make sure content editors always maintain high standards of data quality. Miss out your EANs, for example, and you'll get far less exposure.  

Why bother? 

Here's what can happen to your performance if you optimise properly! A 200%+ performance boost. Simply fixing the errors and warnings and improving taxonomy helps increase the clicks. 

We're now seeing a huge increase in revenues from this channel which is making a massive financial contribution.  

Knowing which 20% of your products are having the greatest impact on your revenues can help you reduce the revenue you lose to stock outs and help you optimise your product sales for the slower turning lines.

It's not directly possible to do a Pareto analysis in Google Analytics, but you can use the Google Analytics Data Feed Query Explorer to extract your annual sales data and pull it into Excel for further analysis. 

Step 1: Fetching the data 

Log into the Data Feed Query Explorer and create a query to fetch the product name and item revenue for every product you've sold over the past 12 months, and order the results by descending item revenue. 

You can save time by clicking here and then authenticating your Google Analytics account. If you highlight the data produced with your cursor, you can just copy and paste it from the Data Feed Query Explorer straight into Excel. 

Step 2: Sorting the data

Now that we've got our revenue data for all of the products we've sold over the past year, we need to count the number of products and split them into three classes, A (the top 20% of sales), B (the next 75% of sales) and C (the bottom 5% of sales). I'd recommend placing each class on its own sheet so you can go over them more easily.  

You should find that approximately 80% of your revenue comes from the top 20% of products sold (it was 75% in the case of my site), while the other two classes (80% of your stock) make up the remaining 20% of your revenue.  

Step 3: Analysing the class data

The products in class A are the ones that are currently of the greatest importance to your revenue. These are the ones you need to keep a really close eye on in terms of your inventory control as stock outs will lose you sales. 

However, if anything, the really interesting stuff lies within the B and C classes. If you spot products in here that you'd expect ought to really be fast-moving products that shift in large quantities worthy of class A, then you've spotted a potential problem. Are they priced too high? What's stopping your customers purchasing them as often as you'd expect? 

If everything is definitely in order regarding pricing and other on-site issues, then perhaps you could reduce your inventory levels of these slower moving items, make them special order items or get them drop-shipped to free up working capital? 

The other really useful information lies in what you can't see. Google Analytics will obviously only tell you about the products you have sold.

If you compare the total number of products you've exported to the actual number of products in your range you may spot a number of products that haven't sold at all - so called class D items. In this case, there's something very wrong and you definitely ought to consider clearing these, unless they're simply range completers.

The next step is to display your data.

However, if you're creating different queries to run which have varying numbers and types of metric and dimension selected, you need a flexible way of formatting data without the need to code a new table each time you modify your API query.

The Smarty templating engine for PHP is ideal for this. 

Step 1: Install Smarty and set it up

Firstly you'll need to install Smarty and get it set up so you can use it within your Core Reporting API application. You can download it from www.smarty.net and check the excellent Smarty Crash Course for some tips on set-up and basic usage. 

Once you've set it up, you'll want to assign a title variable to pass to Smarty: $smarty->assign('title','Data for ');

Then you need to get the column headers, which are your metrics and dimensions, and add them to an array to pass to Smarty so we can use them as the table headers: $smarty->assign('headers',$data['columnHeaders']);

The rest of the data can be passed to Smarty as $data: $smarty->assign('data',$data); and then you can send it all to the display template with $smarty->display('../templates/table.tpl');

Step 2: Create your flexible table template

In order to get your content to display clearly, with the right column headings and the right number of columns, we need a flexible structure that works for any combination of metrics and dimensions. 

Create the basic table structure using the standard tags and then create a loop construct within your first set of table row tags to add the headings based on the selected dimensions and metrics.  

The next step is to loop through the actual data. I used a foreach to loop over all of the results and display them in the appropriate table cells. 

There are some tweaks that are worth making to make the data more presentable.

I used the Smarty cycle value to add a class "on" or "off" to each table row and then added CSS to change the colour to create a zebra-table effect of alternating row colours. 

I also used is_numeric and stristr to check to see whether the value was a string or an integer and rounded the values that had lots of trailing decimal places, and truncated long strings, such as URLs, to make the data a bit cleaner.

Add some CSS to style it up and you've got a quick and easy way to display lots of data from the Core Reporting API, without the need to edit anything other than your original query.  

Next time I'll explain how you can use refresh tokens to create applications that can run without user interaction.  

It relies upon two key bits of data held in the e-commerce platform's database: the current stock holding for each product, and the low-stock level for each product.

When an item is sold, the stock holding goes down according to the number of units sold. Each product is then re-ordered when its stock level reaches its low-stock level or re-order point (ROP).

The stock holding level is manually increased when the ordered goods arrive from the supplier and stock levels are replenished to the maximum stock level.

This technique works but its success is dependent upon a number of things, and there are various business considerations to take into consideration to ensure it works optimally.

Here's a textbook representation of what happens to stock levels.   

The problem?

Well, two main things can go wrong.

You can run out of stock because your stock holding isn't large enough to meet the demand, or you can have a low-stock level which is too low, which means you run out of stock while waiting for the goods you've re-ordered to arrive.

These things need checking continually, because demand for products can rise (and fall), causing you to run out of popular products (or have a surplus of slow sellers you need to turn).

I've explained before how I have used Google Analytics to identify the revenue you lose to stock-outs, and it's well worth implementing this if you suspect stock control needs improving.

However, looking at stock-outs only gives you part of the picture.

What you really want to see, is how close you get to your buffer inventory, how much stock you have, what the demand is like, and how much working capital you have tied up in your stock in relation to sales.

Most e-commerce platforms don't tell you this, and our systems don't report it, so I spent a day devising a hack to obtain the data and pull it into Google Analytics for monitoring, reporting and analysis. 

What will this hack provide?

Step 1: Setting up server-side analytics

In order to obtain data on these things you need to query the database at a particular time each day and send the data to Google Analytics.

However, that's not feasible using the standard client-side tracking code - you need to do it on the server-side from a command-line application.

To achieve this I developed a command-line application in PHP which runs on the PHP-CLI and queries a SQL database to obtain the metrics required.

It then injects them into Google Analytics using the excellent PHP-GA class. If you've not used this before you can get some general usage tips from my post on the hack I made for testing a cookie-less Google Analytics implementation.

Using a command-line application with server-side analytics allows you to run the code from a server and can bypass the 10 event limit imposed for the client side GATC. 

You could also use SSGA server-side Google Analytics if you prefer, but it's not been updated for a while and, annoyingly, it encodes spaces in event category, action and label strings. 

Step 2: Obtaining aggregate daily totals

The first thing I wanted to obtain was aggregate daily totals on various metrics to give an overview on the inventory not provided by our e-commerce platform.

This includes metrics like the daily levels for: total categories, total products, total SKUs, total stocked SKUs, total drop-shipped SKUs, total inventory units and total inventory value.

I wrote a PHP class containing a series of functions which ran SQL queries against the e-commerce platform's database and returned the required data in variables I could pass to GA and echo to the command line output.

If anything, this was the most complicated bit, due to the nature of some of the joins required to query the database. These will obviously vary according to your platform, so I've skipped them out here.  

Step 3: Configuring your events

Once I had the metrics, I then used server-side event tracking to add the data to GA to a separate profile - the latter point is important, for reasons I'll explain in a minute.

Here's the format I used for the PHP-GA events to ensure data is placed in the correct place in GA for easy analysis. These events place all data in an event category called 'Aggregate daily totals'. Drilling-down to action and label levels gives you access to the rest of your data.

Categories
Event format: _trackEvent('Aggregate daily totals','Category','12',12).
Goal: Monitors the total number of categories over time.

Products
Event format: _trackEvent('Aggregate daily totals','Products','1234',1234).
Goal: Monitors the total number of products over time.

SKUs (stocked)
Event format: _trackEvent('Aggregate daily totals','SKUs (stocked)','12345',12345).
Goal: Monitors the total number of SKUs stocked over time.

SKUs (direct)
Event format: _trackEvent('Aggregate daily totals','SKUs (direct)','1234',1234).
Goal: Monitors the total number of drop-shipped SKUs over time.

SKUs (out of stock)
Event format: _trackEvent('Aggregate daily totals','SKUs (out of stock)','123',123).
Goal: Monitors the total number of out of stock SKUs over time.

Inventory units
Event format: _trackEvent('Aggregate daily totals','Inventory units','1234567',1234567).
Goal: Monitors the total number of units of stock held over time.

Inventory value
Event format: _trackEvent('Aggregate daily totals','Inventory value','8787878',8787878).
Goal: Monitors the total value of units of stock held over time.

Customers
Event format: _trackEvent('Aggregate daily totals','Customers','1234567',1234567).
Goal: Monitors the total number of customers over time.

Newsletter subscribers
Event format: _trackEvent('Aggregate daily totals','Newsletter subscribers','123456',123456).
Goal: Monitors the total number of newsletter subscribers over time.

Here's an example of the code required to create the new session in PHP-GA and to send the event tracking data. Again, there's more information and sample code in my other post on PHP-GA. 

Providing you're sending fewer than 499 events in a single PHP-GA session, you should be able to use a single $tracker->trackPageview($page,$session,$visitor); call.

Send more and you risk hitting the 500 hit limit for a session, and you'll need to split events over multiple sessions in the same script.

Echoing the output of the application to the terminal is useful during testing because it allows you to see that the data are being obtained correctly and gives you the opportunity to look for PHP-GA's errors, since PHP-CLI applications don't run through Apache and errors, warnings and notices don't appear in its error logs.

Once you've sent some test data, you should see it appearing in the GA web interface within about 10 minutes.

It's then possible to look at the charts produced showing the daily event values to see how they change over time. Some of the other totals obviously won't be usable, but that's easily fixed by creating custom reports.

Step 4: Analysing inventory by category

Now that I had an overview of the entire inventory, the next thing I wanted to examine was the inventory at a product level. 

The data should be useful for finding out which categories were underperforming and should show what return we were getting from the stock held, when you compare sales revenue to inventory.

How many products and SKUs were in each category? How much stock were we holding? How much working capital was tied up in stock? What return did we get from our inventory for the category?

If we had £2000 of working capital tied up in our stocks of purple mohair ties but monthly sales were only £100, did we really need so much stock? Would it be better to drop-ship these instead?

Extracting these data is achieved in similar ways, but obviously involves a bit of looping and for my e-commerce platform some of the database queries involved some complicated joins and concatenation.

However, the principle is easy. First you need to get your list of categories, then for each category you need to obtain the total number of products, SKUs, inventory held and its value. Here's an example of the event tracking code used.

Subcategories in category
Event format: _trackEvent('Aggregate daily totals','Subcategories in category','Tablet PCs',3).
Goal: Monitors the total number of subcategories in a category over time.

Products in category
Event format: _trackEvent('Aggregate daily totals','Products in category','Tablet PCs',123).
Goal: Monitors the total number of products in a category over time.

SKUs in category
Event format: _trackEvent('Aggregate daily totals','SKUs in category','Tablet PCs',3234).
Goal: Monitors the total number of SKUs in a category over time.

Inventory units in category
Event format: _trackEvent('Aggregate daily totals','Inventory units in category','Tablet PCs',39393).
Goal: Monitors the total number of units of stock in a category over time.

Inventory value in category
Event format: _trackEvent('Aggregate daily totals','Inventory value in category','Tablet PCs',123456).
Goal: Monitors the total value of stock in a category over time.

If you have lots of data to send, consider using sleep(1) to pause for a second between the POSTing of data to prevent hammering the GA servers with a shed-load (that's English for a lot) of traffic all at once. 

You'll also need to ensure you use PHP-GA to create a new session for each set of related events you send, otherwise you'll hit the 500 hits per session limit and execution of the script will stop. 

It's worth making sure that error messages are set to display to standard out because you may get "MySQL has gone away" timeout messages if your data takes a long time to crunch. Opening a new connection for each chunk of data should solve this.  

Step 5: Sending product-level data

Now that we've done with collecting the aggregate data and the category-level data, it's time to do the bulkier task of obtaining the product-level data which provides your actual inventory profile timelines. 

The principle for this bit is the same as for the category-level data. You run a SELECT query against your database to get a full list of your SKUs and return the product name, SKU code, cost price, quantity of stock being held and then create a while loop. 

Within the while loop you need to create a new PHP-GA session for each SKU returned and then send four events containing the data we want to send to Google Analytics.

It's vital to create a new session for each one, because you'll otherwise hit the 500 hit limit in no time. You'll also want to use a sleep() function to minimise the speed at which data is sent to Google's servers.

Since we're adding a bit more data to these, and allowing more drilling-down, these data are going in separate event categories. 

Inventory units
Event format: _trackEvent('Inventory units','Apple Macbook Pro 15','AMBP15',5).
Goal: Monitors the total holding of a SKU over time.

Inventory value
Event format: _trackEvent('Inventory value','Apple Macbook Pro 15','AMBP15',4500).
Goal: Monitors the total value of SKU inventory over time.

Re-order level
Event format: _trackEvent('Re-order level','Apple Macbook Pro 15','AMBP15',1).
Goal: Monitors the re-order or low-stock level over time.

Step 6: Automating data collection

In order for the data to be accurate you need to ensure that the queries are only run once per day. 

I configured a cron job to run my script as a scheduled task on my Linux server every night at one minute past midnight with this command: 01 00 * * * php /var/www/stockprofiler/profiler.php

Every morning when I check GA, I have fresh data from the previous day. I don't need to touch the code again - it just keeps sending me new data on the inventory just as the client-side code does. 

Step 6: Accessing your inventory data

Once you've followed the above steps, you'll be able to access the data in your new profile via the following event categories. You can drill-down among them to see aggregate totals for each day, category or product.

Don't forget that because we're sending daily totals to GA, the only field that will really make sense if the event value and the timeline for the metric.

The total event value metrics will be calculated based on the combined sum of daily events, so won't mean much. It's best to create some custom reports. 

Handy tips

• Put this data in a separate Google Analytics profile to prevent the large quantity of hit level data sent pushing you over your data limit. It would be nice to have the data in the same account as your web data, as you'd then be able to compare web metrics to inventory metrics, but there are risks to that if you have a large site.

• Don't send more than 10 million bits of hit level data per month, otherwise you'll hit Google Analytics' data limit. If you have a big product range, calculate the number of events your script will be firing to determine whether this could be an issue.

• Don't exceed more than 1 million hits per day, otherwise your data will only be refreshed once a day, and you'll exceed your monthly data limit.

  

• Use the new real time reports in Google Analytics 5 to get an idea of the amount of data being sent and the rate, and to check that everything is working as intended during testing.

• If you're using this hack with Google Analytics Premium there is no data limit, so theoretically you could use this with quite a big product inventory. 

• Don't send more than 500 bits of hit level data in a single session. If you're looping over data, use PHP-GA to start a new session in each loop, which will increase the number of sessions and prevents you hitting the 500 hit limit. 

• Use a sleep() function to pause the sending of session data by your script for a second to prevent it hammering the GA servers. 

• Make sure your cron job only fires once a day, as most of these are daily totals which will be ruined if you inadvertently send the data multiple times.

• Create custom reports to make the data more meaningful, since some of the metrics won't apply with this hack. 

Avinash Kaushik has confirmed that analytics providers have been discussing the legislation, but hasn't said whether they have a solution to the problem, and if they do, what it will entail. 

If you're affected, your options are to get users to opt-in to being tracked using cookies (in which case you can kiss goodbye to 90% of your web analytics data and rely on sampling), do nothing in the likely event that the ICO don't bother to prosecute you simply for using GA, or switch to a server-side tracking system which doesn't rely on cookies. 

While I'm aware of other cookie-less tracking systems, I haven't used any since the pre-GA/Webtrends days. 

Personally, I'd be extremely surprised if they came anywhere close to the functionality of Google Analytics, and like many other people in the industry, I've got data in the tool that I want to keep and features I rely upon for my work.

GA does what I need it to do, and it's tightly integrated into my sites so I can capture additional data to help me make the right decisions. 

If browser manufacturers aren't going to step-in to allow this issue to be handled on the browser side, the best solution for me seems to be the development of a server-side Google Analytics tracking system.

That could be one that has official backing, or one that's been built from scratch to help overcome the legislation in a legal and responsible manner.

Server-side Google Analytics?

The current Google Analytics tracking solution is a client-side system and the JavaScript code is executed by the browser.

Tracking data is sent to GA via GIF image requests generated by the code, with certain items of data, such as timestamps and unique identification hashes, being stored in a number of first-party cookies in the visitor's browser.

Google does not yet provide a cookie-less or server-side alternative to this client-side tracking system, although its mobile tracking code comes close.

I'd be really surprised if its engineers weren't working behind the scenes on a server-side API. Indeed, the client-side code probably already interfaces with something a server-side solution could connect to as well.

Thankfully, given the lack of an authorised API, there are at least two fairly robust solutions which achieve the same thing: Server-Side Google Analytics (SSGA) and PHP-GA. 

These are both PHP classes which pass similar data back to the GA server by faking the GIF requests, which trick Google Analytics into thinking the data is coming from a client-side tracking tag.

While some things don't work, and while some actions are harder to track and require more coding effort, they're both actually quite decent and it is possible to get quite a bit of the core tracking data into GA without relying on cookies.

Server-side GA isn't cookie-less by default...

You can use either SSGA or PHP-GA for this. Both work in very similar ways.

I've used SSGA on a couple of projects and it works really well, but unlike PHP-GA it hasn't been under active development for quite a while (since April 2009) and it supports fewer features. 

PHP-GA is more modern (it's written in fully namespaced OOP PHP), it supports pretty much all of the main features (including new ones like trackSocial) and is probably the best choice of the two.

It is server-side, rather than client-side, so the code is hidden from source code viewers and is triggered by PHP on your server, however, it can use cookies for some features. They're just created by PHP, rather than by JavaScript.

It hasn't been designed to be a cookie-free alternative to GA. It's been designed to replicate the functionality of Google Analytics ga.js JavaScript tracking code in PHP. As a result, if you use it as-is, it could create cookies. Most of the main features don't use cookies, though.

Example 1: Basic page tracking

For really basic pageview tracking you can use the code below. Don't forget to replace your own UA profile ID, otherwise you'll fill my test account with your data!

This example simply adds the user IP address (required by GA but not shown or accessible) the user agent (ie. Windows Vista, Internet Explorer 6) and the details on the page title and file accessed. (Use REMOTE_ADDR, not REMOTE_ADDRESS, as I stupidly did!)

You can pass additional data to GA using the Visitor() function, which I'll explain in more detail later. This code doesn't set any cookies whatsover and will provide only basic tracking. 

Example 2: Event tracking

Event tracking, probably the best overall feature of Google Analytics, is also very easy to achieve and again doesn't use any cookies.

You simply call the GoogleAnalytics\Event() function and then pass it the required data using setCategory(), setAction(), setLabel() and setValue() - even the new opt_noninteraction feature is supported via setNoninteraction().

The data can then be passed to the tracker using trackEvent(). Interestingly, the client-side version of GA prevents you from sending more than 10 events at once, but you can send 20 or more simultaneously with both PHP-GA and SSGA.  

Example 3: Ecommerce tracking

Ecommerce tracking works much like the client-side equivalent. First you need to populate the order ID with a random number which is used as the transaction ID. Then you call Item() and set the orderId, the SKU, the name, quantity and price.

You can then create the transaction, pass the items, the basket total and any details you have on the customer location and shipping charges and it populates the ecommerce tracking data in exactly the same way as the client-side version. Again, no cookies are used.

Other features

There are many more things you can do with PHP-GA, including passing additional visitor data (such as screen resolution, Flash version, Java enabled etc), setting custom variables (with cookies), and using trackSocial.   

This only really scratches the surface of testing what you could do with PHP-GA without the need to use cookies, and what you'd probably need to do to get close to the full features sans cookies.

Granted, it falls short of client-side GA if you don't use cookies, but it provides the basics of page tracking and ecommerce tracking which could be enough to get by if the worst does happen. 

Sadly, it isn't capable of recognising repeat visitors, presumably because GA only uses the ID stored in the __utma cookie, and not a combination of other visitor features.  

It's not going to solve every problem, but it could record most activities without cookies and it's certainly going to be a lot better than having 90% of your data missing. Obviously it needs more work, though...