Property Configuration Guide

Overview

This document describes the structure of Instart Logic's property configurations which are used by the proxy servers in the service network to identify the requests coming in addressed to specific customer properties as they enter the service, and to define how to the service should handle them.

It further describes how to use the property configuration API to view and update configuration files.

Structure of configuration files

Each file is defined as a JSON "object" (that is, an unordered set of name/value pairs).

JSON (JavaScript Object Notation) is a lightweight data-interchange format, based on a subset of the JavaScript programming language. Similar to XML, JSON is a text format that is reasonably easy for humans to read and write and is easy for machines to parse and generate. It is completely language-independent, but follows familiar programming conventions and so is easy for developers to work with. They are commonly used in REST APIs. If you are new to JSON, it would be good idea to become familiar with its structure. There are many good references available online.

Each configuration object defines the behavior of requests for a single customer property. A customer account may have multiple property configurations. No two configurations may share the same property name.

When the service receives a HTTP request, it first examines the Host header to determine which config file applies to that request, and then treats it according to the rules defined in that file.

When the service prepares a HTTP response, it likewise is processed according to the rules defined in that file.

Property configurations contain a couple of main elements:

Property configurations have a tree structure. The various configuration settings are applied using action blocks that can exist at different levels throughout the tree.

Property configuration tree diagram

At the root is the property level. An action block here is used to describe the configuration data that will apply to the entire property.

The next level is the domain level. An action block here will apply to traffic that matches for the specific domain.

Next comes the paths level. An action block here will apply to traffic that matches for specific path patterns within the domain.

Within paths can also be predicates, which contain sub-blocks on_request and on_response; within these are conditional expressions that are matched for requests or responses respectively. When the expressions are matched, action blocks within describe the configuration data that will apply to the matching traffic.

In each case all the action blocks that apply at the upper levels are inherited by the lower levels unless overridden by an applicable action block.

There are a couple of additional mechanisms that allow more flexibility for complex configurations:

Property level

At the root or property level, a typical action block will set an ID and a short name used internally by the service to refer to the entire property. It typically contains elements such as an origin block that defines a default property host name and any other origin settings that will apply to any of the domains present, default caching behavior, enabling content optimizations, and so on.

Domain level

The domain level is the next set of nodes in the tree. Properties might have multiple domains, and at this level you can apply domain-wide actions. For example, here you would be likely to specify the domain origin, specify the path to the domain's SSL certificate, and you might apply an overall cache policy.

The domain level is made up a list of one or more domain objects wrapped in a JSON list named domains, as follows:

"domains" : [
   {
      "patterns": ["acme.com"],
      ...
   }
   ...
]

Each domain in the list contains a patterns object, which is a JSON list. Typically this is a list with a single element, a string that with the domain's name used to match to requests. If you are sharding domains, then you would use additional patterns in the list.

The pattern strings can be exact, or can use an asterisk (*) as a wildcard. For example:

"domains" : [
   {
      patterns: [
         "www.acme.com",
         "f1.devops.acme.com"
       ];
      
   },
   {
      patterns: [
         "*.devops.acme.com",
      ];
      
   },
   ...
]

Requests that match the domain pattern will then have any actions or conditional predicates that are specified in the block applied to them.

NOTE In case a request matches two domain patterns – as in the above example, where a request for f1.devops.acme.com would match *.devops.acme.com as well as the exact domain – the specific match takes precedence.

Path level

At this level, you use patterns to distinguish parts of the website so you can treat them individually. A common approach would be to use a wildcard to match everything within the webroot of the domain by default and apply actions, such as bypassing the cache. Then, another path block might be set up that has a pattern that matches all images that have the extensions *.jpg, *.jpeg, or *.png, and applies a cache policy with a fixed TTL of 7 days. The net effect of such a configuration is simply that all images files with the specified extensions are cached for 7 days while any request for any other file on the site is not cached.

The path level is made up a list of one or more path objects wrapped in a JSON list named paths, as follows:

"paths" : [
   {
      patterns: [
         "<pattern1>",
         "<pattern2>",
         ...
         "<patternn>"
       ];
      ...
   },
   ...
]

Each path in the list contains its own list of one or more patterns to match to requests. Requests that match one of the patterns in the list will then have any actions or conditional predicates that are specified in the block applied to them.

There are two types of patterns supported: wildcard and regex. Wildcard patterns suffice for most uses. Regexes are complex and difficult to construct so we recommend that you avoid them unless you have a compelling need. If so, please contact Support to request assistance with developing regexes for patterns.

Wildcard patterns are preceded by the identifier W followed by a : (colon) followed by the pattern itself. The syntax is as follows:

The following are examples of typical patterns:

What to match Pattern
Exact match with path /foo "W:/foo"
Match with paths beginning with /foo "W:/foo*"
Match any path ending with .jpeg "W:*.jpeg"
Match /images/ directory anywhere in the path "W:*/images/*"
Match any files in the /images directory and ending with .jpeg "W:/images/*.jpeg"
Match any files in a directory anywhere named /images/anything/ followed by /user/ and ending with .gif "W:*/images/*/user/*.gif"

Predicates

The root, domain, and path levels apply configuration settings based on the implicit condition of where they occur. That is, you can define settings that apply to anything in the property with an action block at the root level, anything that matches a specific domain at the domain level, and anything that matches a specific path in a specific domain at the path level. But what if you want to apply settings that depend on more complex conditions?

Predicates are used to define such conditions, and apply specific actions based on whether the conditions match for the request. For example, you might want to do something special for any request that has a certain query string key present, or a certain value set for a certain key.

You use regular expressions to define the conditions which the request must match for the conditions to be applied. This allows a tremendous amount of flexibility in apply settings for all kinds of needs.

Here is the basic form of a predicates block:

"predicates" : {
   "on_request" : [
      {
         "id" : "html_str",
         "expression" : "string.match(request.header(\"User-Agent\"), \"htmlstr\") ~= nil",
         "action" : {
            <do stuff>
         }
      }
   ]
}

The block can contain an on_config_load, on_request and on_response conditional. Then within any of these blocks, there can be one or more expressions that will be evaluated. If the expression matches, the "stuff" in the action block contained within will be applied to the request.

In this example, if the request's User-Agent header's value contains the string htmlstr, this request matches. So if the User-Agent header is foohtmlstr or htmlstrbar of foohtmlstrbar, any of these will match the expression.

Defaults

Action blocks at the root level will apply as defaults to the entire property, but they don't allow conditions within them. A defaults block can be used to apply conditional defaults to the property using predicates.

Here's an example defaults block:

"defaults" : {
   "on_request" : [
      {
         "id" : "watch_query_argument_present",
         "expression" : "(request.user_agent().browser == http.ua.browser.safari
            or request.user_agent().platform == http.ua.platform.ios)
            and request.user_agent().browser_major >= 8",
         "action" : {
            <do stuff>
         }
      }
   ]
},

If the expression evaluates to true when the request is examined – that is, in this example, if the browser in the request's User-Agent header is Safari OR the platform in the same header is iOS AND the major version number is 8 – then the "stuff" in the action block contained within will be applied to the request by default. This can of course be overridden explicitly if the current active domain and path specify otherwise.

Overrides

A overrides block at the root level can allow conditional overrides using predicates to be applied at the end of the traversal of the configuration tree. That is, a setting that is normally applied at any level of the configuration tree can be overridden by this block. An overrides block provides a simple means to quickly and easily turn features off for specific domains or for the entire property if neccesary.

For example, you could can turn off HTML Streaming features for all domains in a property for requests that come from Safari version 8.0 or greater as follows:

{
   "domain" : [
     {
       ...
     },
     {
       ...
     }
   ],
   "overrides": {
      "on_request" : [
         {
            "id" : "disable_html_streaming_ios_safari"
            "expression" : "(request.user_agent().browser == http.ua.browser.safari
               or request.user_agent().platform == http.ua.platform.ios)
               and request.user_agent().browser_major >= 8",
            "action" : {
               <disable HTML Streaming feature>
            }
         }
      ]
   }
}

If the expression evaluates to true when the request is examined – that is, in this example, if the browser in the request's User-Agent header is Safari OR form in the same header is iOS AND the major version number is 8 – HTML Streaming will be turned off, overriding whatever the current active domain and path might be.

Precedence

To summarize, the result of the application of configuration to a given request follow from the order of precedence of the settings and where they are in the JSON tree. Where a setting occurs determines if it will be applied at the end, or be overridden by another setting.

It's also important to note that there are two levels above the actual property config file that affect what how a request get handled – the service as a whole, and the set of customer properties, each have a defaults.json file, and the set of customer properties also has a overrides.json file.

The order of precedence is

  1. Settings specified in the service global defaults
  2. Settings specified in the customer property global defaults
  3. Settings specified in the root-level action block and in a defaults block
  4. Settings specified at the domain level
  5. Settings specified at the path level (note the caveat below)
  6. Settings specified in predicates
  7. Settings specified in a property overrides block
  8. Finally, settings specified in the customer property global overrides.json

It's also important to realize that if two or more settings are applied within the same level of the tree, they are applied in top-down order. That is, the last setting is the one that will be applied as the config is parsed. For example, if there is a setting to cache objects for 30 days and turn HTML Streaming on, then a second setting to cache objects for 15 days and turn HTML Streaming off, the second setting will override the first for any objects in the directory.

Also note that the path statement within a domain does not follow the normal precedence order. There can be only one path statement applied. All path blocks are evaluated and the one that is the best match is the one that will be applied.

Editing a property configuration

The process of editing a property configuration with the API is as follows:

  1. Retrieve the configuration
  2. Edit the configuration
  3. Update the configuration
CAUTION The property configuration you retrieve with the API is a JSON object which contains the entire tree and all its nodes. So you can modify anything in the configuration and, assuming the changes you make are valid JSON, the configuration will be applied to your traffic. Be certain that you understand the implications and know what you are changing and its effects.

You need to use an admin account to edit a property configuration.

We'll show a detailed example of this process using a simple config change, and then move on to describe the settings that you are able to edit with this current release and some examples.

In this example's commands we'll use placeholders for your username, password, customer name and property name; replace with your own actual information to try it yourself (without the enclosing angle brackets, of course). The config file in this example is also simpler than a typical configuration for the sake of brevity and clarity. We'll use the command line tool cURL and the Advanced REST Client plugin for the Chrome web browser to illustrate the process. (If you want to use Advanced REST Client in Chrome you can install it from the Google App Store.)

Retrieving the configuration

Using cURL:

Execute the command

$ curl -k -u '<username>:<password>' 'https://api.instartlogic.com/<company_name>/v1/properties/<property_name>/config'

where you would of course use your own username, password, company name, and property name in place of the placeholders.

Using Advanced REST client plugin for Chrome:

  1. In the URL field at the top enter

    https://api.instartlogic.com/<company_name>/v1/properties/<property_name>/config

  2. Click the GET radio button.
  3. In the Headers field, add a header named Authorization with a value

    Basic <authorization string>

    where your authorization string consists of your username and password separated by a colon (:) and then base 64-encoded.

  4. Click Send.

If successful the API server will respond with a JSON object similar to the following (using the company name "acme" and the property name "myprop"):

{
   "content": {
      "config": {
         "action": {
            "customer_id": 999,
            "property_short_name": "myprop",
            "edge": {
               "cloud_cache": {
                  "policy": "FIXED_TTL",
                  "ttl": "8d"
               }
            }
         },
         "domains":[
            {
               "patterns" : [
                  "s01.acme.com",
                  "s02.acme.com"
               ],
               "paths" : [
                  {
                     "id" : "default_path",
                     "patterns" : [
                        "W:/*"
                     ],
                     "action" : {
                        "edge" : {
                           "cloud_cache" : {
                              "policy" : "FIXED_TTL",
                              "ttl" : "30d"
                           }
                        }
                     }
                  },
                  {
                     "id": "cache_imgs",
                     "patterns": [
                        "W:/assets/images/*"
                     ],
                     "action": {
                        "cache": {
                           "policy": "FIXED_TTL",
                           "edge_ttl": "30d"
                        },
                        "edge": {
                           "cloud_cache": {
                              "policy": "FIXED_TTL",
                              "ttl": "30d"
                           }
                        }
                     }
                  },
                  {
                     "id": "stream_js",
                     "patterns": [
                        "W:*.js"
                     ],
                     "action": {
                        "js": {
                           "enabled": true
                        }
                     }
                  }
               ]
            }
         ]
      }
   }
   "id": "/config",
   "uri": "/customers/acme/properties/myprop/config",
   "etag": "cf6982a425a8a2eeb39fe21270a5580cf7996d0c",
}

Note that in addition to the configuration object itself, which is inside the config block, the returned JSON object contains some identifying information at the end. The value of the etag field is important because you will need to use when you submit the edited configuration. (This allows the API server to make sure that the configuration has not changed between the time you requested it and the time you attempt to update it.)

Editing the configuration

Save the returned JSON object as a file to make it easier to work with. Then open it in your preferred text editor.

First, save the value of the etag field to use later, then remove the three fields at the end and replace them with a single field, commit_message. The value of this field is a brief string describing the change you are making. For this example, let's say we realized that the list of patterns for the path with an id of stream_js has only a match for files with the js extension, and we want to add a pattern for axd files too. So our commit message might be something like this:

{
   "commit_message": "Add axd extension to JS Streaming path patterns",
   "content": {
      "config": {
         ...
      }
   }
 }

Edit the configuration to change what you need. For this example, we would add an additional pattern to the list of patterns for the path with an id of stream_js as follows:

                      "patterns": [
                         "W:*.js",
                         "W:*.axd"
                      ],

Save the file. For this example we'll use the name updateconfig.json.

Updating the configuration

Using cURL:

Execute the command

$ curl -k -u '<username>:<password>' -d @updateconfig.json 'https://api.instartlogic.com/<company_name>/v1/properties/<property_name>/config' 
-H "Content-Type: application/json" -H "If-Match: cf6982a425a8a2eeb39fe21270a5580cf7996d0c" -X PUT

where the @updateconfig.json for the -d option specifies that the contents of the file be sent, and the eTag from the earlier GET command is used for the value of the If-Match header.

Using Advanced REST client plugin for Chrome:

  1. In the URL field at the top enter

    https://api.instartlogic.com/<company_name>/v1/properties/<property_name>/config

  2. Click the PUT radio button.
  3. In the Headers section, keep the header named Authorization and add a new header named If-Match, copying and pasting the eTag value from the response to your GET command earlier.
  4. In the Payload section, click Files, then Add new file field and then click Choose Files. Browse to the location of the file updateconfig.json and select it. The filename will appear to the right along with its size in bytes when it has uploaded.
  5. Click Send.

Receiving, parsing, and validating the config change might take on the order of a couple of minutes. When successful, the API server will respond with a JSON object similar to the following (using the company name "acme" and the property name "myprop"):

{"id":"acme-myprop","uri":"/customers/acme/properties/myprop/config","deployment_handle":"1db3fd08-2868-4f39-9ac2-80b4005b014f"}

where the deployment_handle value is an ID for the back end task that will be deploying the configuration on the service. You can use this handle to query the Tasks API for progress on the deployment. Deployment of the configuration should take on the order of a couple of minutes.

Example configurations

This section contains a couple of example configurations. This should give you a general idea of how configurations are structured. The descriptions that follow discuss what happens in the configuration. Don't worry at this point if some of the settings are somewhat mysterious. Following this section, we'll summarize all the settings that are possible and link to separate documents that provide further detail.

The examples are expandable. Click on opening brackets – { or [ – to expand/contract the configuration elements. Note also that the quotes around the keys and values are removed for readability.

Example 1:

Analysis of Example 1:

This property config has several fields at the root level that identify the property – the customer ID and reporting ID. Then there is an action block that further defines several overall settings for the property at large: a timeout of 90 seconds is defined for the first byte of a response to requests sent to the origin, and several image adaptation features are enabled and configured – inside the smart_vision block, JPG and PNG Transcoding are enabled.

Next comes the domains block. This property contains two domains, one with an ID of ruraljuror, which matches for requests to ruraljuror.net, the other with an ID of assets.ruraljuror, which matches for requests to assets.ruraljuror.net.

In the first domain, an action block does the following:

Then comes a paths block. This contains two paths, one with an ID of default_path, the other with an ID of 12h_static_obj.

The first of these paths matches for any traffic to ruraljuror.net to any path using a /* wildcard. It has an action block that sets a Host request header with the value of the Nginx variable $origin_host (identifying the proxy that's making the request to the origin). It also has a predicates block which looks at requests and evaluates a regular expression that if, in the request's User-Agent header, the browser is Safari OR the platform in the same header is iO AND the major version number is 8, then HTML Streaming features are disabled.

The second of these paths matches for any traffic to ruraljuror.net to any path using a /* wildcard and end in one of the file extensions in the patterns list – .css, .js, .jpg, .jpeg",.png, .gif, .ico, .ppt, .swf, .xls, .xlsx, .ttf, and .svg – the edge proxy will cache these with a fixed TTL of 12 hours, and sends the same Host request header as does the first path in this domain.

This finishes up the first domain.

In the second domain, an action block does the following:

Then comes a paths block. This block contains a single default path, and has no further action.

Config settings summary

This section briefly describes the various configuration settings that can be applied inside action and conditional predicate blocks at the various levels of the configuration.

Origin and edge settings

Settings for origins and how the service communicates with them – the address of the origin servers, if GNA is or isn't used, SSL settings, network settings, etc. – are contained in origin blocks.

Likewise, settings for the Instart Logic proxy servers – cache rules, SSL settings, etc. – are contained in edge blocks.

For details see Origin and Edge Configuration.

Caching settings

There are two caches that you can configure behavior for, the cloud cache (the cache on our proxies) and a client cache (the cache in the requesting browser).

Both caches can be configured inside any action block at any level in the configuration tree. Often there will be default cache and cloud_cache blocks at the property level which will apply to all domains unless overridden within action blocks at the domain or path level or inside a predicate.

For details see Caching Configuration.

Real User Monitoring (RUM)

Real user monitoring (RUM) is a passive monitoring technique that records user interaction with a website or client interacting with a server or cloud-based application. It has many uses, including:

Real user monitoring is typically "passive monitoring" – that is, it collects web traffic without having any effect on the operation of the site. This differs from synthetic monitoring with automated web browsers in that it relies on actual inbound and outbound web traffic to take measurements.

For details see Real User Monitoring (RUM) Configuration.

Content optimizations

To learn about configuring content optimization features in detail, follow the links below.

Security

Security settings in the current release consists of controls for throttling (rate limiting) or blocking requests based on user agent, geographical information, and IP address.

For details see Security Configuration.