ControlPro v3.3
Introduction
Multiple ControlPro (CP) instances can be networked together via one of two mechanisms: ControlPro Interface (CPI) and Central Hub (CH). Most of the underlying logic that keeps data justified is the same. The primary difference is the means of communication between the instances: CPI uses XML interface files written and shared between computers while CH uses endpoints registered to the local domain to pass the same information back and forth.
ControlPro networking isn’t like a traditional client-server architecture. Each instance of CP has a complete copy of the data and can continue uninterrupted operation without communication to other CP instances in its network. When communication is restored, justification will eventually bring all data back into alignment.
Network Status and Log
The Network Status indicator in the upper right corner of all CP screens will show the general status of justification. Hover the cursor over the indicator for a descriptive tooltip. Click on the indicator to open the Network Log screen. Both CH and CPI use the Network Log screen to report justification or connection issues.
Justification
Everything in CP networking runs off last modified time stamps. When two versions of a component are justified, the version that was updated most recently is the version that is accepted as the correct one. This works at both the parent level and the child level. For example, Instance A adds Material X to Ticket 123 and changes its description. A few seconds later, Instance B changes the description for Ticket 123 and adds a new reference for Customer PO; without knowing about any of Instance A’s modifications. When these two versions are justified, the accepted Ticket 123 would have Material X, the Customer PO reference, and Instance B’s new description.
Since time stamps are so important to justification, CP instances sync their local times via an external source. By default, this is http://www.microsoft.com. This will give CP a time delta to modify the system clock with when making component modifications. While this is not an exact time delta because of normal internet latency, it can normally get instances to within one second of each other.
Auto-Justification
Every so often, both CH and CPI will request an auto-justification. CP instances in a CH network will auto-justify against the Central Hub while instances in a CPI network will auto-justify with each CPI-enabled CP originator it is networked to individually. This auto-justification is configurable from every 10 minutes to every 24 hours. By default, this will occur every 1 hour.
When auto-justification occurs, each CP will collate lists of all its component IDs and their modified time stamps and transmit that information. Component lists are ordered as follows: originator, equipment, party, material, profile, formula, ticket, ticket batch, and system; reports are not shared between instances of CP. When this information is received, the receiving CP will compare each list to its local component lists. Any internal component not in the list or more up to date will be exported back.
ControlPro Interface (CPI)
CPI uses a simple file exchange to share information; one file for each modified component. In order for two CP instances to be networked, two folders are needed: “Instance A to Instance B” and “Instance B to Instance A”. Basically, a two-way street going in each direction. One CP’s outbox is another CP’s inbox.
File Sharing
File sharing between multiple computers can be done in a number of different ways. Common ways, from best to worst:
- Each CP had a local folder for its outbox. This folder is shared so that other CPs have access. In this way, there will never be an issue with CP failing to export a file as the local folder will always be available.
- All folders are created on a network drive. There can be issues with writing export files if the network connection is spotty but at least the latency is low.
- Each CP has local folders for its inbox and outbox. A 3rd party tool is used to justify the contents of these folders. This mechanism is not preferred as it can sometimes take five minutes or more for the 3rd party tool to push files down to the target computer. From best to worst:
- Microsoft OneDrive
- Google Drive
- Amazon AWS
Networking
When laying out communication between instances of CP, circular communication is not allowed. This will create an infinite loop of interface files.
Generally, a single CP should serve as a “pivot” instance. This instance needs to be running 24-7 to keep the network working. Point all other CPs at this instance. We generally try to use an unattended instance of CP as the pivot as unattendeds should always be running. Here would be an example.
Alternatively, CP instances can be spread out into longer “chains” so that the loadouts have less “hops” from their ticket sources. This helps tickets created in an office instance or a dedicated 3rd party show up in the desired loadout instance quicker. An example of this would be below.
Exporting
When a component is changed, CP exports a file to all CPI-enabled CP originators. Any modified component will have a modified timestamp on it. No response or acknowledgment is returned. This is a one-way communication.
Importing
On each heartbeat, CP will check the import folders for all CPI-enabled CP originators. Any files found are imported in the order they were created. For each imported component, CP will search its local components for one that matches its ID (originator prefix and serial ID). If there is no match, the imported component will be saved internally. If a match is found, the imported component’s modified timestamp is compared to the local component’s modified time stamp. If the local timestamp is more recent, the imported component is ignored. If the imported component is more recent, justification logic will fire and any changes will be applied to the local component; and its modified timestamp will be updated to that of the imported component.
If a new component was created or any modifications were made to an existing component, CP will then re-export that component to all CPI-enabled CP originators other than the source CP of the interface file. In this way, all CPs in the network will eventually get the updated version of the component.
Sometimes an imported component will fail validation for some reason or another. In such an even, the imported component will be ignored and an interface exception will be generated. Any interface exceptions should be investigated immediately before things get out of sync. If the imported component is a system, ticket or material that is currently being used by an in-process batch, the import will silently fail.
Ticket assignments to systems are not shared through CPI.
Manual Justification
Components can be manually exported for justification from each search screen. Select the components to export, or Ctrl-A for all, and click the “CPI Export” button. The selected components will be exported to all CPI enabled originators this instance of CP is communicating with as if it had just been modified.
Central Hub (CH)
CH uses endpoints registered to the local domain to keep instances in the network in sync. The Central Hub instance itself is referred to as the “server” while all other instances are referred to as “satellites”. Although, if communication to the CH server is lost for some reason, CH satellites will continue uninterrupted. CH uses two-way justification to help keep components in sync.
Networking
All satellite instances of CP communicate with the server. It’s recommended that all 3rd parties also communicate with the server as well.
Component Modification Justification
When a component is changed in a satellite, the satellite will justify its new version of the component with the server. The server will then justify this against its internal version and return the accepted version back, which is then re-justified by the satellite. If the satellite’s changes were rejected by the server, the satellite’s version is reverted to the server’s version.
Heartbeat Justification
Every heartbeat, a satellite will send all components modified since the “Last Successful” heartbeat justification to the server for justification. The server will then justify the components and return the accepted versions, which are then re-justified in the satellite. The satellite will then request all components modified in the server since it’s last successful justification; minus the “CH Justification Buffer” setting as a safety margin against time discrepancies. Returned components are then justified locally. If no issues were encountered, the Last Successful timestamp will be updated. If errors were encountered, the timestamp will not be updated, and the satellite will try again next heartbeat. If the Network Indicator remains red after several heartbeats, something is failing to justify and will need manual investigation.
System, tickets and materials that are currently being processed in a batch are ignored during a heartbeat justification. It’s assumed that when the batch is complete, these components will be justified at that point. The Last Successful timestamp will be updated if these components are ignored.
Full Justification
Satellites offer the option for a blunt force, comprehensive, full justification. Every component in CP – in the same order as auto-justification – will be sent to the server for justification, be re-justified locally, and then every missing component in the server is then requested to be locally justified. Full justification offers the most comprehensive mechanism to ensure that all data is in sync that CP networking offers.
Daily Central Hub Cleanup
Since all purged parent and child components are maintained in the server, the server needs to clean them out to keep its dataset from becoming too cluttered. This is accomplished by the CH Daily Cleanup. By default, this happens at 03:00 in the morning. Once a component has been purged from the server, it is gone for good.
CH vs. CPI Considerations
If an instance of CP is not running or its computer is not connected to the network for some reason, CPI files will build up in its inbox quickly. This can end up being thousands of files to import if left uncorrected for a few days or more; tens of thousands if longer. This could mean that when the instance is finally started up, it could be chugging through files for half an hour or longer before it’s useable. If CH were used, a single request for “what’s changed” will bring the instance up to date in seconds.
Reading, writing and transmitting CPI files is a very “slow” operation. Transmitting the same amount of information via the CH endpoints is dozens of times faster.
Because CH relies completely on time stamps, if one component was changed half a dozen times in one instance, when another instance gets the updated version, the data only has to be transmitted once. In CPI, every change generates an interface file that has to be read in individually.
When using CPI, only one CP is allowed to communicate with each physical system. All other CPs should disallow enabling on systems they’re not going to be running. CH allows multiple instances of CP to control the same physical system; although, only one at a time.
During heavy parts of the season, it can take CPI several auto-justify cycles for all data on the network to justify. Because of the two-way street that CH uses, data will justify a lot quicker.
When using CPI, no additional computers or VMs are required to set up a network. CH requires a dedicated computer or VM.
The CH instance will maintain a record of purged components, at least until the daily cleanup. If a component is purged accidentally, it can be restored from the CH. Components purged in CPI are gone as soon as the other CPs in the network receive the purge instruction.