Extending the Layout Service Context

Out of the box, the Layout Service returns the Context and Route objects (see my earlier post for more info on the Layout Service.) On my current JSS React project we wanted to add various metadata fields to the route and render them in the HEAD tag via Helmet, which was fine until we hit the need to add a URL for the OpenGraph og:url metadata tag.

The requirement was:

if no value is supplied in the Canonical URL field or the OpenGraph URL metadata field in the Sitecore route item, then show the current URL of the route.

In server-side MVC land we would do this all the time, and in Sitecore JSS you’d think we could use Window.location (because JavaScript). But (and there’s always a “but”) this project uses React with Server Side Rendering (SSR), so the Window.location object isn’t available when the app is first rendered and delivered to the browser by Node.js. Now maybe there’s a clever way to handle this with React, Node, and a handful of magic beans, but that’s not something I am particularly comfortable with, so I chose to do it in .NET on the server by modifying the data returned by the Layout Service. And aside from that, this approach can be used to populate the context with other data that isn’t available client-side.

Where does this bit go?

Given that the Layout Service returns something much like this:

Then where is the right place to put the URL in the layout service JSON in order to populate the og:url metadata?

The first option I considered was to add it to the Route, but this appeared to entail overriding the Sitecore.LayoutService.ItemRendering.Render method and extending the RenderedItem class so that it returned a URL property, but the amount of change required was extensive, or so it seemed to me anyway.

Option 2 was to add it to the Context which was much simpler and used some typical pipeline processor code that can be patched in. Looking at the pipeline config for GetLayoutServiceContext:

it was apparent that it would be simple to add another pipeline processor that would add the data to the Context. Using LinkManager to calculate the URL for the context item, and patching the processor into the Layout Service pipeline required minimal code:

ItemContext.cs

using Sitecore.Diagnostics;
using Sitecore.LayoutService.ItemRendering.Pipelines.GetLayoutServiceContext;
using System.Collections.Generic;
using Sitecore;
using Sitecore.Links;

namespace JSSDemo.Feature.Metadata.PipelineProcessors
{
    public class ItemContext : IGetLayoutServiceContextProcessor
    {
        public const string Key = "itemUrl";

        public void Process(GetLayoutServiceContextArgs args)
        {
            Assert.ArgumentNotNull((object)args, nameof(args));
            IDictionary<string, object> contextData = args.ContextData;
            var options = LinkManager.GetDefaultUrlBuilderOptions();

            options.AlwaysIncludeServerUrl = true;
            string link = LinkManager.GetItemUrl(Context.Item, options);

            args.ContextData.Add(Key, link);
        }
    }
}

Layout Service Pipeline Patch

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <pipelines>
      <group groupName="layoutService">
        <pipelines>
          <getLayoutServiceContext>
			  <processor type="JSSDemo.Feature.Metadata.PipelineProcessors.ItemContext, JSSDemo.Feature.Metadata" />	
		  </getLayoutServiceContext>
		</pipelines>
      </group>
    </pipelines>
  </sitecore>
</configuration>

Which resulted in the expected JSON output with an “itemUrl” property containing the URL of the context item:

And the Sitecore config was patched to be:

Another approach might be to use a rendering with a datasource, but that would not be a great editor experience.

You could argue that the context item URL doesn’t belong in the Context and ideally the route is the best place for this data, but given the changes needed to change the rendering of the route, this seemed the best option. I’d be interested to hear of a low-touch method to amend the route properties.

Changes to xConnect Client API in Sitecore 10

During an upgrade of my Marketing Automation custom activity demo code to version 10, Visual Studio flagged a warning that the xConnect Client getAsync method was marked as obsolete when passing in ContactExpandOptions.

The warning was:

XdbContext.GetAsync(IEntityReference, ExpandOptions)' is obsolete: 'Please use overload that accepts ExecutionOptions parameter.'

It turns out that, as of Sitecore 10, when doing a getAsync() or get() using the xConnectClient you need to pass an object that extends ExecutionOptions such as Sitecore.XConnect.ContactExecutionOptions. These are:

  • ContactExecutionOptions
  • DeviceProfileExecutionOptions
  • InteractionExecutionOptions

As an example, retrieving a contact changes from:

var contactTask = client.GetAsync(reference, new ContactExpandOptions(personalInfoFacetKey));

to:

var contactTask = client.GetAsync(reference, new ContactExecutionOptions(new ContactExpandOptions(personalInfoFacetKey)));

See Sitecore xConnect Client API docs for more.

Creating a custom rendering contents resolver

In my previous post, I covered the basics of how the Layout Service and Rendering Contents Resolvers work together to deliver component rendering data via the API for consumption by rendering hosts using Sitecore’s headless delivery options such as the JSS or .NET Core SDKs.

In this post we will look at why we might want to customise the structure and content of component rendering data returned via the API.

Let’s start with an example.

Dangerous Creatures of Australia

Everyone knows that Australia is stuffed to the gills with lethal creatures such as sharks, spiders, jellyfish, snakes, ants, and scorpions. In fact it is so dangerous that you should never come here. To warn people of the dangers, I’ve created a blog about the perils of these creatures.

Within a JSS site and JSS tenant hosted in SXA there is a root Blog node based on a Blog route template (which extends the default JSS App Route) and some child routes created using a Blog Article route template:

On individual Blog Article routes I’ve added a rendering via Standard Values presentation details which is called Recommended For You and which renders out some promo cards that point to recommended blog articles.

Recommended blogs rendering

This rendering uses a datasource containing a Multilist field which content managers can use to assign specific articles as “Recommended” articles:

The JSS app will iterate through these and render a promo card with an image, teaser text, and a link to the actual article.

Let’s take a look at what the output of this looks like using the default features of the Layout Service.:

As you can see, there is a single rendering Recommended Blogs that has a datasource assigned which contains 2 fields: moduleTitle and recommendedItems.

The recommendedItems field is a Multilist and that presents in the field value as a | separated list of item IDs representing the recommended blog articles. The first problem here is, well, what the heck are we supposed to do with a bunch of IDs? Our front end devs won’t be able to render a list of promo items that link to the recommended blogs when they only have some ID values to work with. We could use GraphQL but I’m going to use a rendering contents resolver because it will provide a more consistent, familiar data structure for the devs to work with (and if I didn’t, then this blog post would be called something else.)

Extending the Datasource Resolver

As I mentioned in my previous post, the rendering contents resolvers (except Sitecore Forms) all use the same code, just with different parameters. That code is in:

Sitecore.LayoutService.ItemRendering.ContentsResolvers.RenderingContentsResolver, Sitecore.LayoutService

To extend this functionality we can override the ResolveContents method (or implement IRenderingContentsResolver):

public override object ResolveContents(Sitecore.Mvc.Presentation.Rendering rendering, IRenderingConfiguration renderingConfig)

From there it’s a simple matter to get the datasource using GetContextItem. For this to work you can either set UseContextItem to false in the code, or uncheck the checkbox in the Sitecore Content Editor:

The first version of the code looks like this:

using Newtonsoft.Json.Linq;
using Sitecore.Data;
using Sitecore.Data.Items;
using Sitecore.Diagnostics;
using Sitecore.LayoutService.Configuration;
using System.Collections.Generic;
using System.Linq;

namespace JSSDemo.Feature.Blog.Extensions
{
    public class RecommendedBlogsContentsResolver1 : Sitecore.LayoutService.ItemRendering.ContentsResolvers.RenderingContentsResolver
    {
        private List<Item> items = new List<Item>();

        public override object ResolveContents(Sitecore.Mvc.Presentation.Rendering rendering, IRenderingConfiguration renderingConfig)
        {
            Assert.ArgumentNotNull(rendering, nameof(rendering));
            Assert.ArgumentNotNull(renderingConfig, nameof(renderingConfig));

            Item ds = GetContextItem(rendering, renderingConfig);

            var recommendedItemsFieldId = Templates.RecommendedForYouTemplate.RecommendedItemsFieldId;

            //if the rendering datasource has curated items
            if (ds.Fields.Contains(recommendedItemsFieldId) && !string.IsNullOrWhiteSpace(ds.Fields[recommendedItemsFieldId].Value))
            {
                List<string> recItemIds = ds.Fields[recommendedItemsFieldId].Value.Split('|').ToList();
                foreach (var id in recItemIds)
                {
                    var item = Sitecore.Context.Database.GetItem(new ID(id));
                    items.Add(item);
                }
            }
            
            if (!items.Any())
                return null;

            JObject jobject = new JObject()
            {
                ["items"] = (JToken)new JArray()
            };

            List<Item> objList = items != null ? items.ToList() : null;
            if (objList == null || objList.Count == 0)
                return jobject;
            jobject["items"] = ProcessItems(objList, rendering, renderingConfig);
            return jobject;
        }        
    }
}

The code is pretty straightforward, it:

  • gets the datasource item
  • checks that the recommendedItems field exists and retrieves the value
  • splits the list of Ids and iterates over it
  • gets each item from the context database and adds to a list of items
  • serialises the list of items and returns them to the layout service

[The Nuget packages required for this code are Sitecore.Kernel and Sitecore.LayoutService]

A new rendering contents resolver is added in Sitecore using this assembly:

And then applied to the rendering:

The rendering data in the Layout Service JSON now looks like this:

Which is much better. We now have an array of Sitecore Items and all the fields that we need (and some we don’t).

Improving the code

There’s a couple of things about the above output that are not ideal:

  • It is returning ALL of the fields. Ideally we want to only return what we need, otherwise the Layout Service JSON is unnecessarily bloated (especially if the articleContent field is populated.)
  • There is no URL to the item

You could look at using ContentSearch for retrieving the data, but the Rendering Contents Resolver serializer expects only items so a content search results model would still have to be mapped onto an item before serialization, which seems kind of pointless. I looked at extending the serializer code, but it was Items all the way down and not worth the effort. In terms of efficiency, this rendering will probably take advantage of the item cache and other caching options can be applied so that it doesn’t hit the database as frequently.

Another option you might think would be to modify the Fields property of the Item…..

20 Funny Memes On How To Say No | SayingImages.com

So in the end I settled on creating a model Sitecore template for the blog promo card which contained only the fields that I wanted to return, and then mapped the Blog item data onto an in-memory “fake” Item and serialized that. The model template has only a few fields:

The new code looks like this:

using Newtonsoft.Json.Linq;
using Sitecore.Data;
using Sitecore.Data.Items;
using Sitecore.Diagnostics;
using Sitecore.Globalization;
using Sitecore.LayoutService.Configuration;
using System.Collections.Generic;
using System.Linq;

namespace JSSDemo.Feature.Blog.Extensions
{
    public class RecommendedBlogsContentsResolver2 : Sitecore.LayoutService.ItemRendering.ContentsResolvers.RenderingContentsResolver
    {
        private List<Item> items = new List<Item>();

        public override object ResolveContents(Sitecore.Mvc.Presentation.Rendering rendering, IRenderingConfiguration renderingConfig)
        {
            Assert.ArgumentNotNull(rendering, nameof(rendering));
            Assert.ArgumentNotNull(renderingConfig, nameof(renderingConfig));

            Item ds = GetContextItem(rendering, renderingConfig);

            var recommendedItemsFieldId = Templates.RecommendedForYouTemplate.RecommendedItemsFieldId;

            //if the rendering datasource has curated items
            if (ds.Fields.Contains(recommendedItemsFieldId) && !string.IsNullOrWhiteSpace(ds.Fields[recommendedItemsFieldId].Value))
            {
                List<string> recItemIds = ds.Fields[recommendedItemsFieldId].Value.Split('|').ToList();
                foreach (var id in recItemIds)
                {
                    var item = Sitecore.Context.Database.GetItem(new ID(id));
                    var articleModel = MapItemToRenderingModelItem(item, GetItemUrl(item));
                    items.Add(articleModel);
                }
            }
            
            if (!items.Any())
                return null;

            JObject jobject = new JObject()
            {
                ["items"] = (JToken)new JArray()
            };

            List<Item> objList = items != null ? items.ToList() : null;
            if (objList == null || objList.Count == 0)
                return jobject;
            jobject["items"] = ProcessItems(objList, rendering, renderingConfig);
            return jobject;
        }

        private Item MapItemToRenderingModelItem(Item item, string itemUrl)
        {
            var newId = new ID();
            var def = new ItemDefinition(newId, item.Name, Templates.BlogCardRenderingModel.TemplateId, ID.Null);

            //Populate fields
            var fields = new FieldList();
            fields.Add(Templates.BlogCardRenderingModel.Fields.ArticleTitle, item.Fields[Templates.BlogArticlePage.Fields.ArticleTitle].Value);

            //Use SuccessImage if available, else use HeaderImage
            if (item.Fields[Templates.BlogArticlePage.Fields.SuggestedImage].HasValue)
            {
                fields.Add(Templates.BlogCardRenderingModel.Fields.Image, item.Fields[Templates.BlogArticlePage.Fields.SuggestedImage].Value);
            }
            else
            {
                fields.Add(Templates.BlogCardRenderingModel.Fields.Image, item.Fields[Templates.BlogArticlePage.Fields.HeaderImage].Value);
            }

            fields.Add(Templates.BlogCardRenderingModel.Fields.Categories, item.Fields[Templates.BlogArticlePage.Fields.Categories].Value);
            fields.Add(Templates.BlogCardRenderingModel.Fields.Url, itemUrl);
            fields.Add(Templates.BlogCardRenderingModel.Fields.Teaser, item.Fields[Templates.BlogArticlePage.Fields.Teaser].Value);

            var data = new ItemData(def, Language.Current, Sitecore.Data.Version.First, fields);
            var db = Sitecore.Context.Database;

            var resultItem = new Item(newId, data, db);
            return resultItem;
        }

        private string GetItemUrl(Item item)
        {
            return Sitecore.Links.LinkManager.GetItemUrl(item);
        }
    }
}

Instead of adding raw Blog Article Items to the items collection, this code maps the retrieved Items onto a BlogCardRenderingModel which is created on the fly in code and returns a collection of those instead. I used Alistair Deneys’ (@adeneys) excellent blog post on this as a guide for creating the “fake” model item, although there are a few changes here. In particular:

  • Uses Version.First() instead of Version(1) – needed for Sitecore 10
  • This uses Language.Current instead of parsing the EN language
  • Uses Sitecore context database instead of the “dummy” database approach which did not work for me in Sitecore 10
  • Uses a “real” Sitecore template ID so that the necessary fields are available. Using a fake template ID does not work in this situation because the fields won’t exist for the mapping code to populate.

The mapping also calculates the URL for the item using LinkManager. The final JSON output now looks like this:

Each item in the rendering data now only contains the fields articleTitle, categories, image, teaserText, url, id and name, which is a lot leaner and prevents the Layout Service from becoming unnecessarily bloated.

Conclusion

Modifying or extending the contents resolver allows us to reduce bloat and return exactly the data we need, but it also opens up opportunities to pull data in from other sources for use in renderings, combine data from the Content Search API with Sitecore item queries, and filter, sort, and group the data in different ways.

[The code and content for this post and the previous one are on Github]

Sitecore Layout Service and Rendering Contents Resolvers

When developing JSS or.NET Core headless solutions, one of the key components is the Layout Service. The Layout Service composes and serves a JSON description of the context, route, placeholders, renderings (components), and datasources that can be used to populate the data which the rendering engine uses to render the final UI of the application. The Layout service is part of Sitecore Headless Services (previously known as JSS Server Components) and powers both JSS and the newer .NET Core headless rendering approaches. You can find out more about the Layout Service here.

In this post we’ll look at the Layout Service payload and at how Sitecore uses a Rendering Contents Resolver to determine the contents and structure of the component rendering data returned via the Layout Service. In a subsequent post we will look at why and how we might extend a Rendering Contents Resolver to suit our needs.

The Rendering Contents Resolver is not the only way to return rendering data to the API – you can also use GraphQL, however that will be the topic of a future post.

Layout Service data

Sitecore Headless Services can be used to expose endpoints that return Sitecore route data as JSON, but you need to create and configure a Sitecore Services Client (SSC) API key in order to query it. For details on how to create an SSC API key, see section 2 of this page. Once created, don’t forget to publish your API key if you are using the API endpoint to query the Web database.

The Layout Service API call has the following structure:

https://[siteUrl]/sitecore/api/layout/render/jss?item=[item ID or sitecore path]&sc_apikey=[your api key]

For example:

https://sc10.sc.dev.local/sitecore/api/layout/render/jss?item=/&sc_apikey={6369DDCB-6A77-439F-8A73-C47496DA1CEE}

or

https://sc10.sc.dev.local/sitecore/api/layout/render/jss?item={66C56C12-F526-4E61-9D75-D76895AC2587}&sc_apikey={6369DDCB-6A77-439F-8A73-C47496DA1CEE}

In the first example above the API will return the Layout Service representation of the site root (since we’ve provided the path “/” ). In the second example it will return the layout for a specific item ID (in this case it’s the Blog Home route). You don’t actually have to use curly braces {} to wrap the ID nor the hyphen in the ID – it will work without them. i.e. jss?item=66C56C12F5264E619D75D76895AC2587&sc_apikey=6369DDCB6A77439F8A73C47496DA1CEE will work just the same.

Let’s look at the JSON output:

In the example above, the layout service represents a route called Blog in the context of the jss-demo site and we can see the item Name (Blog), ID, and a few other properties such as databaseName (in this case it’s the Sitecore master database.)

The root of the JSON contains the context and the route data:

Looking at the route, we can see it contains route-level fields such as pageContent, pageTitle, as well as item name, version, ID, etc:

Basically this is all the stuff you’d expect see about the item, although quite a lot has been filtered out for us already, since we don’t want to send ALL the item data down the pipe in the JSON because that would (a) increase the size of the JSON payload, and (b) expose a lot of irrelevant data or data that should be hidden from the browser. I’m using JSS on top of SXA, so the Page Design field has been returned as well, however there isn’t a page design applied currently.

Within the route, we can also see 3 placeholders:

  • jss-demo-ph-content
  • jss-demo-ph-content
  • jss-demo-ph-content

Each placeholder can contain an array of zero or more renderings (aka “components” in JSS terminology.)

In this case, only one of these placeholders has been populated (jss-demo-ph-content). That placeholder contains a rendering “Blog Promo” which has a datasource {621726E6-4D0C-4C4E-9081-44A874C98A38} that represents a blog article:

As we can see, the fields on the datasource have been serialised and returned in the Layout Service data, as well as the (currently empty) params object which would contain rendering parameters if populated. Back on the Sitecore server the datasource item has been retrieved and serialised for us to use in the rendering engine. This is where the Rendering Contents Resolver fits in.

Note that you can return whatever JSON data you like as component rendering data, but the structure above is what the default serialised item data looks like.

Rendering Contents Resolvers

Out of the box, Sitecore Headless Services provides 6 resolvers, and these are in sitecore/System/Modules/Layout Service/Rendering Contents Resolvers

The default resolver is the Datasource Resolver which, as the name suggests, serializes the datasource item.

The various OOTB resolvers are summarised below

ResolverFunction
Context Item Children ResolverSerializes the children of the context item
Context Item ResolverSerializes the context item
Datasource Item Children ResolverSerializes the children of the datasource item
Datasource ResolverSerializes the datasource item
Folder Filter ResolverSerializes the descendants of the datasource item, excluding folders
Sitecore Forms ResolverRetrieves and serializes a Sitecore Form and its associated items
Rendering Contents Resolvers included with Sitecore Headless Services

[tip of the hat to Gary Wenneker who covered this a while ago]

It is important to note however, that although the default is the Datasource Resolver, it only uses the resolver code with default properties and does not reference the actual Datasource Resolver item. Therefore if you change the properties of the Datasource Resolver item in the Layout Service/Rendering Contents Resolvers section of the tree it will have no effect on your rendering output unless you go into the rendering settings and set the value for Rendering Contents Resolver as per below:

All of the resolvers above (except Sitecore Forms Resolver) use exactly the same code, just with some properties passed in that define what to retrieve and how to serialize it. The type is: Sitecore.LayoutService.ItemRendering.ContentsResolvers which is in the Sitecore.LayoutService assembly.

The properties that can be set are:

Type: the .NET type that retrieves and serialises the output. As is usual with Sitecore, you can create your own, which we will cover the in my next post.

Include Server URL in Media URLs: Fairly self explanatory, fully qualifies the URL for the media item src.

Use Context Item: retrieves the context item instead of the datasource item. You might want to use this to populate route level data into a rendering (e.g. for route metadata values perhaps, although you could do this client side without needing to set this value.) In the code, there is a not very well named method GetContextItem(rendering, renderingConfig) which will return the context item if this property is true, otherwise it will do a GetItem on the datasource item (if one has been provided).

Item Selector Query: Sitecore query to select and filter items for inclusion in the rendering data

All of the above will cover simple rendering data requirements, but there may be situations where it is preferable to modify the structure and the contents of the data returned to a rendering. In my next post I will look at extending the OOTB resolver code and why this might be useful.

[The code and content for this post and the subsequent one are on Github]

SUGCON Global 2020

With in-person SUGCON events cancelled all across the world due to the pandemic, the Sitecore community decided to hold a global SUGCON online. This was a great initiative and being global made it open to everyone – either as a presenter or an attendee – without the need for travel.

There was a huge amount of quality content delivered and all available online for free! You can see the videos on YouTube. I’d be hard pressed to pick the best (IMO), but the presentations that had most relevance for me were from Vincent Lui, Alastair Deneys, @sitecorey, Alex Shyba, and the highly entertaining JSS/SXA shootout from Mark van Aalst and Anastacia Flynn, but there are many other good ones in there too. Oh and Jax Baxter’s personalisation one. In fact just watch them all and decide for yourself.

I was also fortunate enough to have the opportunity to present alongside these great presenters, and delivered an updated version of the Marketing Automation customisation session that I took to Sitecore Symposium in Orlando last year: Automate all the things!

The slides are available here.
Kudos and thanks to the organisers who put a massive amount of effort into this event.

More resources are available here:

Github sample code
https://github.com/parrya/ma demo

Troubleshooting tips
https://andypaz.com/2019/11/11/troubleshooting custom activity types and marketing automation/

Dude, where’s my contact?

Dude, Where's My Car? Where's Your Car, Dude? - YouTube
Dude.

Ever wondered how to find the xDB Contact that was created for a user that visited your site ? No ? Well that’s too bad because I’m going to tell you anyway.

When a user visits a Sitecore site that has tracking enabled, a cookie named SC_ANALYTICS_GLOBAL_COOKIE is created in the response:

The key bit here is the string of hex digits “7828c8717c3d4594be641d919ae2af8c” (but not the |True or |False suffix which is related to robot detection).

This value represents a Device Profile ID in Sitecore which identifies the device from which the user browsed the site. This information gets added to their session and remains there until the session is flushed to xConnect/xDB on expiry/timeout.

Sitecore uses a Device Profile Id rather than a Contact Id because users (contacts) can visit from more than one device and, if we are able to “identify” them in Sitecore (i.e. link a browser session to a specific xDB Contact with some degree of certainty), then the interactions that occurred on different Device Profiles can be merged into the Contact data in xDB. This means that a user’s actions from different sessions and devices can be linked, providing a broader view of a Contact’s interactions and behaviour over time.

So, how to find the Contact record? First, you will need to wait until the session expires and the data is flushed to xDB…

Mr Bean Waiting GIF - MrBean Waiting StillWaiting GIFs

If this is a bit tedious then (outside of production environments anyway) you can change the session timeout in the Web.config to a much shorter period so that the data flows through more quickly.

To check if the data has arrived in xDB, run the following query in xDB:

/* Retrieve all device profiles from shards */

SELECT DeviceProfileId, LastKnownContactId
FROM [Xdb.Collection.Shard1].[xdb_collection].[DeviceProfiles]
WHERE DeviceProfileId = cast('7828c871-7c3d-4594-be64-1d919ae2af8c' as uniqueidentifier)

  UNION ALL

SELECT DeviceProfileId, LastKnownContactId
FROM [Xdb.Collection.Shard0].[xdb_collection].[DeviceProfiles]
WHERE DeviceProfileId = cast('7828c871-7c3d-4594-be64-1d919ae2af8c' as uniqueidentifier)

This query will return the last known xDB Contact Id for that Device Profile. Note that the ID needs to be converted to a string representation of a GUID format (8-4-4-4-12 digits) and cast to a UniqueIdentifier.

If the session has timed out and data is flowing correctly into xDB then this query should return at least one record:

Which contains the last known Contact Id for that Device Profile. In this case “37E8135C-4953-0000-0000-05FE63C6DB57”.

From there it’s a simple matter to retrieve Contact and Facet (or other) data via the Contact Id:

SELECT c.ContactId, FacetKey, FacetData
FROM [Xdb.Collection.Shard1].[xdb_collection].[Contacts] c
     INNER JOIN [ma-demo._Xdb.Collection.Shard1].[xdb_collection].[ContactFacets] cf
     ON c.ContactId = cf.ContactId
WHERE c.ContactId = '37E8135C-4953-0000-0000-05FE63C6DB57'

    UNION ALL
 
SELECT c.ContactId, FacetKey, FacetData
FROM [Xdb.Collection.Shard0].[xdb_collection].[Contacts] c
     INNER JOIN [ma-demo._Xdb.Collection.Shard0].[xdb_collection].[ContactFacets] cf
     ON c.ContactId = cf.ContactId
WHERE c.ContactId = '37E8135C-4953-0000-0000-05FE63C6DB57'

In a follow up post I’ll look at how to get contact Identifiers out of xDB.

Don’t delete the Click Email Link goal if you use EXM

If you delete the “Click Email Link” goal:

Then your EXM click-through events will not be tracked. This came to light when looking in the contact interactions when trying to find out why the click events were not being processed. All the other events such as Email Opened were showing up, but the text “Page event definition Click Email Link not found” was appearing as an event in the Interactions array on the xDB contact.

Historically, as far as I am aware, these goals were just examples, but it appears that EXM has a hardcoded dependency on this goal. It should probably be protected or put into a sub-folder for EXM. For now, though, if you want click tracking on emails to appear in your stats, don’t delete it.

Delete xDB contacts and interactions from Collection databases

To clean out unwanted contact data from your collection databases, there are some referential integrity constraints that mean the deletions have to happen in a specific order. The following query will delete all contacts, facets, interactions, interaction facets, and contact identifiers from xDB.

  delete
  FROM [xdb_collection].[ContactIdentifiers]
  
  delete
  FROM [xdb_collection].[ContactIdentifiersIndex]

  delete
  FROM [xdb_collection].[ContactFacets]
    
  delete
  FROM [xdb_collection].[InteractionFacets]
  
  delete
  FROM [xdb_collection].[Interactions]

  delete
  FROM [xdb_collection].[Contacts]

You will need to run this on both Collection shards.

The mysterious case of the Zombie Automation Plan

Yesterday we worked with our customer to deploy some integrations that trigger Sitecore Marketing Automation plans. The plans do their thing and then push messages onto an Azure Service Bus topic and the messages end up on a Dynamics Contact Activity feed. To test the integrations, I created an Automation Plan that would be triggered from an event and then pushed some test content to the service bus. All appeared to be working, so I deleted the plan and uploaded the “real” plans ready for action.

Next morning, lots of messages had been processed but the Activity Feeds on the Dynamics contacts had double the number of activity records. One which was correct and the other which had some content that looked a lot like my test automation plan output. No worries, I thought, it must still be there. Go and check in /System/Marketing Control Panel/Automation Plans. Nope, not there.

OK, so maybe it’s orphaned in the Master database. Search for the Item by ID (which conveniently enough was included in the service bus message). Nope, not there either.

Hmm, so it must be cached by the maengine.exe. Go to Azure Portal, restart ma-ops app service. Turns out this doesn’t recycle the WebJob, so I edited the maengine.exe.config and that recycled the WebJob.

Trigger a plan as a test. Nope, still getting duplicate messages on the bus.

Alright, what about Deploy Marketing Definitions ? Cool, go do that, trigger the plan again. No luck – still getting duplicate messages.

What was the cause?

Well, Automation Plans are not actually retrieved from the Master database, since the MA engine doesn’t have access to the Sitecore databases directly. It uses the REFDATA service which in turn pulls from the refdata database (which is where the data is sent when you click the magical Deploy Marketing Definitions button), so that was the next place to check.

The refdata database contains a table called xdb_refdata.Definitions which has some data that looks like:

A query of the xdb_refdata.DefinitionTypes table revealed that Automation Plans have a TypeID of 2DE34E0A-5FB6-4AE0-ACFC-69ACF97076B4

so the query to find all Automation Plans in the refdata Definitions table was:

select *
FROM [xdb_refdata].[Definitions]
WHERE TypeID = '2DE34E0A-5FB6-4AE0-ACFC-69ACF97076B4'

I can see the plans, but which is the zombie plan? The Data column is varbinary(max) and the ID column has no relation to the ItemID in the Master database, so to find out which plan doesn’t belong we need to get the plan data from the Data column:

SELECT id, cast(data as varchar(max)) as planData
FROM [xdb_refdata].[Definitions] 
WHERE TypeID='2DE34E0A-5FB6-4AE0-ACFC-69ACF97076B4' 
ORDER by LastModified

Then it was just a case of looking at the planData output and finding the zombie plan. Great but what now ? Kill the zombies with fire ? The Definition items were referenced in 2 other tables xdb_refdata.DefinitionCultures and xdb_refdata.DefinitionMonikers and I wasn’t keen to start deleting stuff in Production, so instead I marked the offending Definition items as IsActive=0:

UPDATE [xdb_refdata].[Definitions]
SET IsActive=0
WHERE id in ('B143F382-7014-4E55-A764-0180D82AED74','33BFBC5E-C3B9-49C0-9E37-710BA146AA79');

Run the tests again and no more duplicate messages coming through. Just to be sure I deployed the marketing definitions again but the data remained set to IsActive=0 so all was well..

The moral of the story is this: Don’t delete Marketing Automation Plans without deactivating them.

Postscript:

The above approach still had issues, some contacts were enrolling in plans and were blocked and the following error was occurring (although not all the time):

[12/02/2019 00:03:54 > fdf166: INFO] 2019-12-02 00:03:54 ERR An exception occurred during processing for worker '35f908d6-638e-433e-aef6-9184a80378f4'
[12/02/2019 00:03:54 > fdf166: INFO] Sitecore.Framework.Conditions.PostconditionException: Postcondition 'planDefinition should not be null' failed.
[12/02/2019 00:03:54 > fdf166: INFO] at Sitecore.Framework.Conditions.EnsuresValidator1.ThrowExceptionCore(String condition, String additionalMessage, ConstraintViolationType type) [12/02/2019 00:03:54 > fdf166: INFO] at Sitecore.Framework.Conditions.Throw.ValueShouldNotBeNull[T](ConditionValidator1 validator, String conditionDescription)
[12/02/2019 00:03:54 > fdf166: INFO] at Sitecore.Framework.Conditions.ValidatorExtensions.IsNotNullT

The key was in the planDefinition should not be null message. The zombie plan was still taking new enrolments.

To fix this, I needed to re-activate the plans in refdata and set the EndDate of the plan, but the Data column is varbinary(max) so a bit of SQL was needed:

SELECT id, cast(data as varchar(max)) as planData
FROM [xdb_refdata].[Definitions]
WHERE id ='33BFBC5E-C3B9-49C0-9E37-710BA146AA79'

declare @planData varchar(max);
declare @binPlanData varbinary(max);
declare @test varchar(max);
SET @planData = '{-- insert the modified planData from SELECT query above with EndDate set to a date in the past --}';
SET @binPlanData = cast(@planData as varbinary(max));
SET @test = cast(@binPlanData as varchar(max));

PRINT @test;

UPDATE [xdb_refdata].[Definitions]
SET Data = @binPlanData, IsActive=1
WHERE id = '33BFBC5E-C3B9-49C0-9E37-710BA146AA79';

Next time I am going to use a flamethrower.

Automate Everything!

Recently I had the privilege and good fortune to present at the 2019 Sitecore Symposium in Orlando, Florida. Below are links to the slides and notes from my presentation on customising Sitecore Marketing Automation plans.

Slides

Code

Github repository containing sample code, utilities and content package: https://github.com/parrya/ma-demo

Troubleshooting and debugging tips

See this blog post for some tips on creating custom activity types in Sitecore Marketing Automation plans.

Resources

Sitecore docs:
https://doc.sitecore.com/developers/92/sitecore-experience-platform/en/activity-types.html
Custom Activity Types in 9.0.x:
https://www.brimit.com/blog/sitecore-9-custom-marketing-automation-action
Automation deep dive:
https://www.linkedin.com/pulse/how-sitecore-9-marketing-automation-works-deep-dive-emmerzaal/
Marketing Automation Achitecture:
https://www.youtube.com/watch?v=0HxUph2YLZc&list=PL1jJVFm_lGnyicywCcwcWa8RtsoiJEbC9&index=5