Implementation Life Cycle

To avoid repeating Implementation Life Cycle , I use the acronym ILC throughout this page.

As mentioned at Automation Script, sysopswork script at its simplest form is implemented top-down, line by line. Regardless of the results of each command (or REST API call), it continues to implement until it finishes all script lines.

Although this method is easy and quick for simple implementations, we want to have the ability to validate the implementation and ensure everything is working as intended. We also want to have the ability to revert back the configuration in case something goes wrong. With ILC instructions, we can achieve that.

An ideal ILC would have the following steps:

Pre-Implementation Tests are performed and provided with successful results, meaning the current state is desirable to start implementation.
Implementation commands are applied to change the state of an existing configuration.
Post-Implementation Tests are performed and provided with successful results, meaning the new state is as intended.

If Pre-Implementation Tests fail, we want to have the ability to do not proceed with the implementation.

If Implementation commands or Post-Implementation Tests failed, it means the implementation itself failed. When this situation happens, we want to have the ability to go through the following steps:

Apply Back-out commands to revert the configuration to the original state.
Perform a Final-Test to confirm that Back-out was implemented properly and the state has returned to its original.

If either Back-Out commands or Final-Test failed, the implementation failed to return to its original state.

sysopswork scripts has three major end results for any implementation:

Implementation Applied means that Pre-Implementation Tests (if present), Implementation, and Post-Implementation Tests (if present) all returned intended results.
Back-Out Applied means that either Implementation or Post-Implementation Tests (if present) did not return the intended results and that Back-Out and Final-Test returned expected results.
Automation Failed means that Pre-Implementation Tests, Back-Out, or Final-Test did not return the intended results.

Follow below a table for better visibility:

Automation Status	Pre-Test	Implementation	Post-Test	Back-Out	Final-Test
Implementation Applied	Succeed	Succeed	Succeed
Back-Out Applied		Failed	Failed	Succeed	Succeed
Automation Failed	Failed			Failed	Failed

Automation failure could also happen in the event of a timeout or a REST API response with a status code other than 2xx (and redirects).

The respective instructions of those steps were presented at Automation Script, and in this page, I will provide more details on how sysopswork process the different steps of ILC.

ILC Script

The mandatory instruction line before any command line instruction is OBJECT:. sysopswork has to know at which object the automation will be applied. The exception are REST instructions, which do not need an object.

Among all ILC command instructions, the only mandatory one is IMPLEMENTATION-COMMAND: (or IMPC:). But as previously explained, in order to reach full potential of sysopwork, ideally all instructions must be provided.

Usually, an ILC script will have the following lines:

OBJECT: host

PREC: command_to_test_expected_original_state
PRER: results_verification

IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification

POSTC: command_to_test_expected_new_state
POSTR: results_verification

BACKC: command_to_return_state_of_object_configuration
BACKR: results_verification

FINC: command_to_test_expected_original_state_after_backout
FINR: results_verification

The shorter version of the instruction is suggested.

Notice that the blank lines between the instructions are not necessary.

Also notice that I did not include any of the -SUCCESS: (or S:), neither -FAILURE: (or F:) as those are used in exceptional cases, where you want sysopswork to behave the opposite of the default automation flow.

You can automate several objects within the same script, as per below:

OBJECT: host_1

PREC: command_to_test_expected_original_state
PRER: results_verification

IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification

POSTC: command_to_test_expected_new_state
POSTR: results_verification

BACKC: command_to_return_state_of_object_configuration
BACKR: results_verification

FINC: command_to_test_expected_original_state_after_backout
FINR: results_verification

OBJECT: host_2

PREC: command_to_test_expected_original_state
PRER: results_verification

IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification

POSTC: command_to_test_expected_new_state
POSTR: results_verification

BACKC: command_to_return_state_of_object_configuration
BACKR: results_verification

FINC: command_to_test_expected_original_state_after_backout
FINR: results_verification

One important point to mention here is that sysopswork implements instructions in sets. What does that mean?

The set is based on the life cycle steps: from Pre-Implementation Test to Final-Test. On the example above with two objects, there are 2 sets.

In other words, sysopswork organizes the implementation based on the next instruction. If the next instruction follows the ILC steps explained above, sysopswork will consider the next instruction being part of the same set. If the next instruction is a previous ILC step, then sysopswork understand this is a new set. An example of 2 sets for the same object:

OBJECT: host

PREC: command_to_test_expected_original_state
PRER: results_verification

IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification

POSTC: command_to_test_expected_new_state
POSTR: results_verification

BACKC: command_to_return_state_of_object_configuration
BACKR: results_verification

FINC: command_to_test_expected_original_state_after_backout
FINR: results_verification

PREC: command_to_test_expected_original_state
PRER: results_verification

IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification

POSTC: command_to_test_expected_new_state
POSTR: results_verification

BACKC: command_to_return_state_of_object_configuration
BACKR: results_verification

FINC: command_to_test_expected_original_state_after_backout
FINR: results_verification

Notice that after the first FINR:, it comes a PREC:. This scenario indicates to sysopswork that a new ILC is starting, from Pre-Implementation Test to Final-Test. In other words, this is a new set.

IMPORTANT: Be very careful with the order to the implementation steps. Don’t expect sysopswork to understand that a POSTC: (Post-Implementaiton Test command) entered before an IMPC: (Implementation command) in the script is part of the same ILC steps. In this case, sysopswork understands as non-correlated sets.

You can have as many sets as you want for an object. You can also repeat the same implementation step within a set. For example:

OBJECT: host

PREC: command_to_test_expected_original_state
PRER: results_verification

IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification

IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification

IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification

POSTC: command_to_test_expected_new_state
POSTR: results_verification

BACKC: command_to_return_state_of_object_configuration
BACKR: results_verification

BACKC: command_to_return_state_of_object_configuration
BACKR: results_verification

BACKC: command_to_return_state_of_object_configuration
BACKR: results_verification

FINC: command_to_test_expected_original_state_after_backout
FINR: results_verification

Notice there are 3 IMPC: and IMPR: instructions between Pre-Implementation Test and Post-Implementation Test, and 3 BACKC: and BACKR: instructions between Post-Implementation Test and Final-Test. As mentioned at Automation Script, if you don’t need to analyze the results of all Implementation and Back-out commands individually, you could rewrite the last example as follows:

OBJECT: host

PREC: command_to_test_expected_original_state
PRER: results_verification

IMPCS:
command_to_change_state_of_object_configuration
command_to_change_state_of_object_configuration
command_to_change_state_of_object_configuration
IMPCE:

POSTC: command_to_test_expected_new_state
POSTR: results_verification

BACKCS:
command_to_return_state_of_object_configuration
command_to_return_state_of_object_configuration
command_to_return_state_of_object_configuration
BACKCE:

FINC: command_to_test_expected_original_state_after_backout
FINR: results_verification

You can also have different objects for each ILC steps. For example:

OBJECT: host_1

PREC: command_to_test_expected_original_state
PRER: results_verification

IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification

OBJECT: host_2

IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification

OBJECT: host_3

IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification

POSTC: command_to_test_expected_new_state
POSTR: results_verification

BACKC: command_to_return_state_of_object_configuration
BACKR: results_verification

FINC: command_to_test_expected_original_state_after_backout
FINR: results_verification

Important to notice on this example is that although there are 3 objects, there is only 1 ILC instruction set. The instruction OBJECT: does not reset the ILC set.

This flexibility is very powerful as you can perform a Pre-Implementation Test in one object, and only if the results of the test succeeds, implementation commands are applied in a completely different object.

Some implementations will have the following automation pattern:

OBJECT: host_1

PREC: command_to_test_expected_original_state
PRER: results_verification

OBJECT: host_2

PREC: command_to_test_expected_original_state
PRER: results_verification

OBJECT: host_1

IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification
IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification

OBJECT: host_2

IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification
IMPC: command_to_change_state_of_object_configuration
IMPR: results_verification

OBJECT: host_1

POSTC: command_to_test_expected_new_state
POSTR: results_verification

OBJECT: host_2

POSTC: command_to_test_expected_new_state
POSTR: results_verification

OBJECT: host_1

BACKC: command_to_return_state_of_object_configuration
BACKR: results_verification
BACKC: command_to_return_state_of_object_configuration
BACKR: results_verification

OBJECT: host_2

BACKC: command_to_return_state_of_object_configuration
BACKR: results_verification
BACKC: command_to_return_state_of_object_configuration
BACKR: results_verification

OBJECT: host_1

FINC: command_to_test_expected_original_state_after_backout
FINR: results_verification

OBJECT: host_2

FINC: command_to_test_expected_original_state_after_backout
FINR: results_verification

2 objects, 1 ILC instruction set.

Analyzing Results

Some commands return values, and based on those values, we might know if the command implemented had the intended effect, or if something went wrong.

The -RESULTS: (or R:) instructions analyzes the returned values of the -COMMAND:``(or ``C:) instructions and confirm if an expected result has returned. sysopswork analyzes ASCII character set.

IMPORTANT: Some commands return results in a human-friendly pagination format. For automation, pagination must be disabled, and commands be provided with options to avoid pagination. For inevitable cases, you can use PROMPT: instruction. More details at Prompt

There are 3 methods fo verify the results of commands:

WORD
REGEX
PRECISION-SYNTAX

Both WORD and REGEX analyze all the result lines for a match. PRECISION-SYNTAX could have indications of where exactly to look.

WORD

The method WORD is as the name says, any word (it cannot contain spaces). You provide a word to -RESULTS: (or R:) instruction and sysopswork will look at all the result lines trying to find the word. If the word is found, sysopswork consider the results of the command successful. If it does not find the word, it considers the command failed.

REGULAR EXPRESSION (REGEX)

Regular expression is a sequence of characters that specifies a search pattern. This document does not provide details of how to use regex. There are a vast number of materials available on the internet just about regex.

Similar to method WORD, sysopswork looks for the provided regex pattern on all result lines, one by one. If it finds a match, sysopswork considers the command succeed. If it does not find a match on any of the lines, it considers the command failed.

PRECISION-SYNTAX

The method PRECISION-SYNTAX uses the power of regex and indications of line number and column position to find an exact match, instead of looking line by line. Another advantage of PRECISION-SYNTAX is that you can use multiple regular expressions to identify patterns in different places using a single -RESULTS: (or R:) instruction. The PRECISION-SYNTAX is defined as below:

$/{X1, Y1, "Z1"}REGEX-PATTERN1/{X2, Y2, "Z2"}REGEX-PATTERN2/{Xn, Yn, "Zn"}REGEX-PATTERNn/

X indicates a line number. Y indicates a column number. And Z indicates the column delimeter character (the default is either space or tab), and it has to be enclosed between double quotes.

All X, Y and Z are optional.

Let’s work on some examples. To explain all the options (X, Y, and Z), let’s take the /etc/passwd file example. The script has the following lines:

PREC: cat /etc/passwd
PRER: $/{2, 5, ":"}daemon/

The command cat /etc/passwd returns the following entries:

root:x:0:0:root:/root:/bin/bash
daemon:x:2:2:daemon:/sbin:/sbin/nologin
user:x:3:4:user:/usr/user:/bin/bash

For brevity purposes, several entries were omitted.

The precision-syntax instruction $/{2, 5, ":"}daemon/ looks at the line 2, at column 5, using the delimeter “:”. If you take line 2 (daemon:x:2:2:daemon:/sbin:/sbin/nologin), and break the line by columns using the delimeter provided “:”, the first column has the value daemon, the second columns has the value x, the third column has the value 2, the fourth column has the value 2, and the fifth column has the value daemon, which is what we provided with the regex daemon. In other words, the command is considered successful.

Let’s now use a match on multiple lines still with the example above. Consider the script has the lines:

PREC: cat /etc/passwd
PRER: $/{1, 5, ":"}ro.*/{3, 1, ":"}daemon/

The first part of the precision-syntax is {1, 5, ":"}ro.*, which means look at line 1, column 5, with delimeter “:” for the regex ro.*. The command cat /etc/passwd returned on line 1, column 5, the word “root”, so the regex ro.* matches. So this test passed.

The second part of the precision-syntax is {3, 1, ":"}daemon. The command cat /etc/passwd returned on line 3, column 1, the word “user”, so the regex daemon does not match. Therefore, the test failed, and even the first part of the precision-syntax matched, if there is a failure with any of the parts, the command is considered to have failed.

IMPORTANT: Result method PRECISION-SYNTAX does not work if the output of a command is paginated. On those cases, you have to use either a WORD or a REGEX.

Implementation Failure

Backout starts at the first BACKC: instruction, followed by any consecutive BACKx: and finally FINx: instructions (if provided). After sysopswork completes to implement BACKx: and FINx: instructions of the current set, it starts to perform the same procedure on previous sets (if any) in a reverse order. For example, in the script:

OBJECT: host_1

PREC: command_to_test_expected_original_state
PRER: results_verification

IMPC: impc_command_1
IMPR: impr_results_1

POSTC: postc_command_1
POSTR: postr_results_1

BACKC: backc_command_1
BACKR: backr_results_1

FINC: finc_command_1
FINR: finr_results_1

OBJECT: host_2

PREC: command_to_test_expected_original_state
PRER: results_verification

IMPC: impc_command_2
IMPR: impr_results_2

POSTC: postc_command_2
POSTR: postr_results_2

BACKC: backc_command_2
BACKR: backr_results_2

FINC: finc_command_2
FINR: finr_results_2

OBJECT: host_3

PREC: command_to_test_expected_original_state
PRER: results_verification

IMPC: impc_command_3
IMPR: impr_results_3

POSTC: postc_command_3
POSTR: postr_results_3

BACKC: backc_command_3
BACKR: backr_results_3

FINC: finc_command_3
FINR: finr_results_3

There are 3 objects, 3 sets. If impr_results_1 or postr_results_1 fails: 1. sysopswork implements backc_command_1; 2. verifies backr_results_1; 3. if backr_results_1 succeeds, it implements finc_command_1; 4. it verifies finr_results_1. In this case, sysopswork does not proceed to object host_2; 5. sysopswork terminates the implementation as “Back-Out Applied”.

If both impr_results_1 and postr_results_1 succeeds, sysopswork skips BACKx: and FINx: of host_1 and proceeds to PREC: of host_2. If at host_2, either impr_results_2 or postr_results_2 fails: 1. sysopswork will proceed to backc_command_2 all the way to finr_results_2. It does not proceed to host_3; 2. then implements any BACKx: and FINx: of the previous sets in reverse order, starting at the first instruction of the previous set, meaning that: 3. it will implement backc_command_1 at host_1; 4. verifies backr_results_1; 5. implements finc_command_1; 6. verifies finr_results_1; 7. sysopswork terminates the implementation as “Back-Out Applied”.

To emphasize the “reverse order of back-out sets” procedure, in case sysopswork has to apply BACKx: and FINx: instructions on host_3, it will when apply the BACKx: and FINx: instructions on host_2, then apply the BACKx: and FINx: instructions on host_1.

IMPORTANT: The reverse order refers to the sets, not the instructions, meaning that sysopswork will never implement FINx: before BACKx: of the same set. The same applies if there are more than one BACKC: instruction within the same set. The order of the instructions within the set are respected during back-out implementation.

Forcing Back-Out

There are situations where sysopswork automates an implementation, all values analyzed through -RESULTS: (or R:) instructions are as expected, but you still need need to revert back the implementation.

There are 2 ways to perform this forced back-out:

If the script was fully implemented, and the status returned was Implementation Applied, sysopswork automatically creates a backout script with the filename implementationID_backout, located at subdirectory /script of sysopswork main directory. You must use that script as sysopswork sorted the backout commands in the right order (based on ILC sets).
You want to implement the backout in the exact order of a specific script.

Either way, you can use sysopswork option -b to force sysopswork to apply the top-down backout instructions of a script, as per example below:

sysopswork -b script/implementationID_backout

If you don’t use option -b, sysopswork understands this is a regular implementation and skips backout instructions until the results of Implementation command or Post-Implementation command fails.

Checking Logs

You can find logs of an implementation at subdirectory logs/ of sysopswork main directory. They are ideal to identify issues that might have happened during an implementation.

There are 3 files generated as part of a regular implementation:

File with name IMPLEMENTATION_ID.log containing timestamps at UTC timezone and which steps sysopswork processed.
File with name IMPLEMENTATION_ID_http.log with details of all HTTP requests and respective responses as provided by REST instructions.
File with name IMPLEMENTATION_ID_cli.log with output of SSH / PowerShell Remoting sessions.

In the case of a forced back-out, the 3 files generated are:

File with name IMPLEMENTATION_ID_backout.log containing timestamps at UTC timezone and which steps sysopswork processed.
File with name IMPLEMENTATION_ID_http_backout.log with details of all HTTP requests and respective responses as provided by REST instructions.
File with name IMPLEMENTATION_ID_cli_backout.log with output of SSH / PowerShell Remoting sessions.