Power of configuration rules

TL;DR;

Scroll down to Environment Rules section that takes you through env:define/:require and whatever:define/:require configuration out of the box extensions.

Intro

Sitecore ecosystem is known for flexibility and numerous features ‘out of the box’, but as we know everything comes with the price. Specifically for Sitecore Experience Commerce 9.0.2 the price includes approximate 350 configuration files that combined consume enormous 3 MB of xml text with comments.

How much this pile of configuration will grow during real app development? It is hard to say as it all depends on numerous factors such as scale, variety of environments, development practices and tools used in process. And just like in real life, when something is growing it’s better to keep control of it. So in this post I will try to describe fairly new successful way of organizing configuration matter using configuration rules engine.

Evolution of Roles

During whole lifetime of Sitecore 6 developers had little issues with configs due to size of them and limited set of roles:

• Content Delivery aka CD
• Standalone aka Content Management aka CM
  Note while there could be several CD instances in the solution, statistically there always was only single CM (also known as Standalone in this case). Later these two roles became very different.

The most typical developer’s challenge was to understand the main concept itself that application may need more than one working instance at a time, and how to switch master to web.

As Sitecore product was getting maturer the demand on splitting system into smaller chunks was growing even faster. Until Sitecore 7.5 there were two main roles that and two optional:

• Dedicated Publishing
• Dedicated Email Dispatch (for optional ECM/EXM module)

While Sitecore 7.5 didn’t get any major popularity, it was compensated by Sitecore 8.0 that flipped Sitecore world and made developers struggle with enormous, overdetailed and often inconsistent documentation of converting Standalone Sitecore instance type (which was preconfigured in distribution package with no other options) into one of following roles:

• Content Management aka CM
• Processing
• Reporting
• Content Delivery aka CD
• … and various mixtures of first 3 of them:
  • CM with Reporting
  • CM with Processing
  • CM with Processing and with Reporting aka Standalone
• … and various mixtures with:
  • Dedicated Publishing
  • Dedicated Indexing (which is worth separate article)
  • Dedicated Email Dispatch (for optional ECM/EXM module)

Sitecore-Configuration-Roles

Thanks to small initiative group with people from Sitecore Ukraine and Sitecore Australia, Sitecore 8.2.3 was released with unblocked configuration engine that allowed to implement prototype Sitecore-Configuration-Roles that extended default XML namespaces of Sitecore configuration with new one that allowed annotating independent configuration elements with instructions when to enable or disable them.

<configuration xmlns:role="http://www.sitecore.net/xmlconfig/role">
  <sitecore>
    <someElement someAttribute="someValue"
                 role:require="ContentManagement && !Processing">
      <!-- 
        the element itself and all children are processed only 
        when role:require="expr" expression is true 
      -->

There were a couple of key aspects that brought the project to success:

1. Embracing familiar syntax of commonly used XML namespaces
2. Providing annotated version of default Sitecore configuration files
3. Support of a few partners who fearlessly used it in production
4. Support of community for reporting issues and submitting fixes

Configuration Rules

Followed by success of Sitecore-Configuration-Roles the initiative group acquired the prototype implementation and improved it, making it even more helpful with generalized extensible configuration rules engine which is explained in official documentation. Out of the box Sitecore offers several important features:

• Same role as before with all configuration files annotated by default
• Extra search and exmEnable to switch search provider and disable EXM
• Concept of env environment rules that lets replace MSBuild config transforms, or at least limit their usage only to the web.config file.

While it’s crystal clear of how to use first three options from common sense, official documentation and the prototype’s readme file, the last but not least env:define concept unfortunately lacks any info sources. Jump to Environment Rules section if you cannot wait, but it’s better go discuss the mechanics of the rule based config reader first:

<configuration>
  <configSections>
    <section 
      name="sitecore" 
      type="Sitecore.Configuration.RuleBasedConfigReader, Sitecore.Kernel" />

The logic of engine is fairly straightforward:

1. During website startup stage, create a matrix of defined categories of rules:

   Check ASP.NET configuration for app settings in the given format:

   <configuration>
     <appSettings>
       <add name="banana:define" value="african, american, australian"/>
       <add name="bird:defined" value="cockatoo|penguin"/>
       ...
   

   and build in-memory dictionary of the following kind:

   {
     "banana": [ "african",  "american", "australian" ],
     "bird":   [ "cockatoo", "penguin",               ],
   }
   

2. Extract <sitecore> node from ASP.NET configuration and merge patch files into it, sourcing App_Config contents according to the order defined in the App_Config/Layout.config file.

   By default, it’s almost driven by common sense:

   • App_Config\Sitecore (non-alphabetically, check Layouts.config)
   • App_Config\Modules (alphabetically)
   • App_Config\Include (alphabetically)
   • App_Config\Environment (alphabetically)

3. When recursively processing XML elements of particular *.config file, check attributes for presence of required with XML namespace that follows the http://www.sitecore.net/xmlconfig/KEYWORD/ pattern. If any of the rules fails, the entire XML element with children is skipped.

   Rule evaluation code is simple: all defined tokens are replaced with true and the rest unknown words - with false, and then boolean expression is being evaluated.

   In accordance with banana-bird sample above, only C element will survive:

   <configuration 
     xmlns:banana="http://www.sitecore.net/xmlconfig/banana/"
     xmlns:horse="http://www.sitecore.net/xmlconfig/bird/">
      
     <sitecore>
       <A banana:require="(african or australian) and !american"/>
       <B banana:require="european" horse:require="cockatoo"/>
       <C banana:require="african" horse:require="cockatoo"/>
   

   Important! Even though it is a good practice to have the namespace prefix matching last word in the schema URL, but technically it is not enforced.

   So in this sample horse: works with bird:define because of the appropriate namespace declaration: xmlns:horse="http://www.sitecore.net/xmlconfig/bird/"

Environment Rules

As it is shown in previous Configuration Rules section, Sitecore supports any banana:define and banana:require configuration rules a developer finds useful. Out of box, there are a few of them used and only one is not documented anywhere, even in the web.config itself.

In fact, there is a place in Sitecore files where it is used out of box and it is the App_Config/Environment/Sitecore.PipelineProfiling.config file where env:require="Profiling" rule is used. Quite interesting that the official documentation mentions similar localenv: token which can be a documentation issue and it was supposed to use env: instead.

The sweetest part of the story is that you can mostly replace MSBuild transforms usage with the env:require annotations in Sitecore patch files and only use transforms with web.config to inject environment name:

<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
  <appSettings>
    <add key="env:define" value="Local, Win10, Development, Debug"
         xdt:Transform="Insert"/>

<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
  <appSettings>
    <add key="env:define" value="Azure, UAT, Debug"
         xdt:Transform="Insert"/>

What types of configuration parts can be controlled with env:require? Please share you thoughts via DM on LinkedIn or Twitter. The list will be extended with your samples, current set is:

• Site hostnames
• Unicorn mode and sourceFolder
• MailServer localhost or SendGrid
• FridayCore features

Let’s talk about benefits of configuration rules engine over traditional transformation files:

• Having single set of configuration files make it absolutely easy to make deployments: copy entire App_Config sub-folders and it’s all done. This makes deployments reliable and also leave little room for mistakes like deploying when MyPatchFile.Debug.config to production.

  In addition to that, it is also easy to see difference between Sitecore instance when troubleshooting - there is no need to use file comparison software to see the difference between two or more sets of files because rules are in fact the diff itself.

• Transformation files usually are executed at the compile time, which makes preparing 4 sets of deployment packages (CM, CD, PR, RE) 4 times longer.
• Rules unlock extraordinary flexibility by combining identical transforms for different environments into single block with env:require rule which naturally complies with DRY.
• While transforming the web.config file is super easy thanks to great Visual Studio tooling, it can be tricky to do the same for other files.

Security

While it was never good idea to store passwords and tokens in unencrypted configuration files, having configuration rules engine makes it even more important, because in this case compromising one set of config files compromises credentials for all environments. That’s why it is critical to use right vault for storing sensitive data.

Sceptics might think that having access to configuration files is already terrible enough (which indeed makes some sense), but in security is so much more complex subject than ‘good or bad’, there are different layers and there could be different consequences for breaching different layers.

Examples

To be updated with current setup using web.config transform for octopus variables.

References

The proposed approach works with Sitecore 9.0.0 or later, which means it’s about a year old tech. Google advises there were several blog posts describing nearly same thing soon after public launch of that release in October 2017:

• Sitecore 9 : Custom Config Roles
• A Practical Approach to Structuring Sitecore 9 Config Files
• Getting rid of transformations, allow the same build to run in any environment

Apart from that, author managed to get private conversations with group of MVPs who used similar technique in recent successful projects.

Testing Jekyll and Forestry

Hey, welcome to my new blog which eventually will accumulate some information from my past blogs and of course be a host for new ones. Stay tuned.

Good screenshots

A screenshot is a good source of information especially for troubleshooting issues with no doubt. It is worth noting however that not all screenshots are same helpful. Here is my brief list of rules that most good screenshots comply with:

• It must include entire application, not just obviously essential parts
• If it contains sensitive information, it must be replaced with sample, but similar one. For example, replace actual database password (e.g. Sj2937$wow) with sample (e.g. $amp1e001) which is very close to original
• It should include outlining that helps to focus on problematic part

The Story

• The set of screenshots must represent a story line even if they are not part of any kind of story. This is essential because people find it so much easier to deal with a story rather than bunch of pieces whether they are connected with each other or not
• The story must not miss any important steps even if trivial
• The file names should start with ## masked sequential number followed by brief description of main step highlight

Commands with NotNull-enforced get-only properties

Problem

In real world there are cases when a class needs to hold many properties that must be initialized before class is ready to use. For example:

public class WelcomeGuestCommand
{
    public string Title { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }

    public string Execute()
    {
        return $"Welcome {Title.ToUpper()} {FirstName} {LastName}";
    }
}

Having the class defined as above it is easy to come to an error with NullReferenceException thrown:

public void WelcomeTonyStark()
{
    var welcome = new WelcomeGuestCommand
    {
        FirstName = "Tony",
        LastName = "Stark"
    };

    var message = welcome.Execute();
    Console.WriteLine(message);
}

A NullReferenceException will be thrown because we forgot to initialize Title property.

Of course, we can resolve this kind of issue by checking for null every time we use our properties, for instance, in the beginning of every method using guard:

public string Execute()
{
    // Guard.ArgumentNotNull method actually is a full equivalent of simple 
    // checking value for null and throwing new ArgumentNullException otherwise
    Guard.ArgumentNotNull(Title, nameof(Title));
    Guard.ArgumentNotNull(FirstName, nameof(FirstName));
    Guard.ArgumentNotNull(LastName, nameof(LastName));

    return $"Welcome {Title.ToUpper().Trim()} {FirstName.Trim()} {LastName.Trim()}";
}

Or, it can be solved via null-checks in properties:

public class WelcomeGuestCommand
{
    private string _Title;
    
    [NotNull]
    public string Title
    { 
        get => _Title ?? throw new ArgumentNotNullException(nameof(Title));
        set => _Title = value ?? throw new ArgumentNotNullException(nameof(value));
    }
    
    private string _FirstName;
    
    [NotNull]
    public string FirstName
    { 
        get => _FirstName ?? throw new ArgumentNotNullException(nameof(FirstName));
        set => _FirstName = value ?? throw new ArgumentNotNullException(nameof(value));
    }
    
    private string _LastName;
    
    [NotNull]
    public string LastName
    { 
        get => _LastName ?? throw new ArgumentNotNullException(nameof(LastName));
        set => _LastName = value ?? throw new ArgumentNotNullException(nameof(value));
    }
}

These are valid solution, but it is not elegant at all, especially when certain properties must always be initialized.

Proposed Solution

Thanks to R# and C# 7 we now have can use the following construct (well, in previous versions it worked too, but not that elegant):

// welcome guest interface describes command
public interface IWelcomeGuest
{
    string Title { get; }
    string FirstName { get; }
    string LastName { get; }
    
    [NotNull]
    string Execute();
}

// our command exposes NotNull on get-only properties which is valid
// because they can only be initialized in constructor and we can
// ensure their not-null values using "?? throw ex" C# 7 syntax
public class WelcomeGuestCommand : IWelcomeGuest
{
    // marking with NotNull helps R# to assist when property is not
    // initialized in constructor (and you can configure it to treat
    // that as error) and use R# null-ref analysis.
    [NotNull]
    public string Title { get; }
    
    [NotNull]
    public string FirstName { get; }
    
    [NotNull]
    public string LastName { get; }
    
    public WelcomeGuestCommand([NotNull] IWelcomeGuest args)
    {
        Title = args?.Title 
            ?? throw new ArgumentNullException(nameof(args.Title));
        
        FirstName = args?.FirstName 
            ?? throw new ArgumentNullException(nameof(args.FirstName));
        
        LastName = args?.LastName 
            ?? throw new ArgumentNullException(nameof(args.LastName));
    }

    public string Execute()
    {
        return $"Welcome {Title.ToUpper()} {FirstName} {LastName}";
    }
}

// welcome guest class is used to pass values to constructor
// and to invoke default command implementation
public class WelcomeGuest : IWelcomeGuest
{
    public string Title { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }
    
    // executes default implementation
    public string Execute()
    {
        var command = new WelcomeGuestCommand(this);
        var result = command.Execute();
        
        return result;
    }
}

public class Tests
{
    [Fact]
    public void WelcomeTonyStark()
    {
        // the ArgumentNullException will be thrown at the moment
        // when the command is constructed: Value cannot be null.
        // Parameter name: Title
        var welcome = new WelcomeGuest
        {
            FirstName = "Tony",
            LastName = "Stark"
        };

        // so the Execute call will fail with human-friendly exception details
        Assert.Equal("Welcome MR Tony Stark", welcome.Execute());
    }
}

Benefits

There are a number of benefits that the approach gives:

Read-Only Properties

You can use get-only properties to ensure command does not have state.

Static NotNull analysis

You can rely on R# NotNull static analysis as all necessary properties are marked not-null and cannot change (see above).

Copying constructor

You can use either arguments or command to create another command with the same parameters.

var args = new WelcomeGuest
{
    FirstName = "Tony",
    LastName = "Stark"
};

var welcome1 = new WelcomeGuestCommand(args);
var welcome2 = new WelcomeGuestCommand(welcome1);

Don't get surprised by automatic publish

It came to us from the dark times when Sitecore was young and developers were brave. In fact, it was decided that once one of the special items is modified the engine must start a publishing process immediately. There were only a few types of these items, and these types were defined as follows:

<!-- PUBLISHING -->
<publishing>
  <smartPublishTriggers>
    <trigger templateId="{CB3942DC-CBBA-4332-A7FB-C4E4204C538A}" note="proxy" />
    <trigger templateId="{AB86861A-6030-46C5-B394-E8F99E8B87DB}" note="template" />
    <trigger templateId="{455A3E98-A627-4B40-8035-E683A0331AC7}" note="template field" />
  </smartPublishTriggers>
</publishing>

Luckily, as many other great things in Sitecore, this one can be disabled via Publishing.AutoScheduleSmartPublish setting. This might be a good idea to switch it off in any case.