What is the Wonder Machine?

It’s been long in coming, but we are now ready to get into the meat of things over the next few posts.

The Problem

Wonder Machine is a design of a demo environment that is particularly suited to demonstrating end-to-end integrated scenarios. The problem the Wonder Machine is meant to solve is a large memory footprint of an integrated setup (two or more applications, middleware, databases) contrasted with limited memory in a demo laptop. While the obvious solution would be to build the integrated setup on a server with abundant memory, we keep running into situations where we do not have a reliable connection from the client site where we do the demo – either because we can't get a reliable Internet connection or because of the client's security policies. For those situations, we need to have a setup that we can carry to the demo.

We'll see shortly that the Wonder Machine design also helps us when running server-based demo environments – by making it possible to clone it and run multiple copies concurrently in a data center.

How is it going to work? Consider as an example the integration between Oracle Utilities applications Customer Care and Billing (CC&B), Meter Data Management (MDM) and Operational Device Management (ODM). All the integration flows are routed through an instance of Oracle SOA Suite (SOA). We cannot fit an integrated setup that requires a total of 9 GB memory to run into a laptop that has only 8 GB memory. What we may be able to do is to split the setup so that it runs on two 8 GB laptops. If the integrated setup is made up of several discrete components, interacting with one another over a network, we can host some components in VM1 running on Laptop1, and others in VM2 running on Laptop2, like this:

Here the arrows indicate network connections between components, i.e. CC&B, MDM and ODM each connect to the SOA Suite server, but do not connect to one another directly.

Distributing the environment between two laptops solves the limited memory problem, but how do we specify the network connections in the integration points? We could run the two laptops on a private network (set up a dedicated portable router and assign static IP addresses) or we could use the DNS names of the two VMs. This will work for the components laid out as in the picture, but it will not be easy to change when we have different hardware available (laptops with more or less memory). For example, if we have to reduce the VMs to no more than 4 GB each, we would want to be able to run our integration on three VMs:

and once we upgrade to a laptop with 16 GB of memory, we would like to run everything in a single 12 GB VM:

In general, the URLs of the integration points are configured at installation time, and are not easy to change afterwards (at least, not easy for a functional sales consultant who did not perform the installation). Specifying URLs with explicit IP addresses or DNS names makes it harder to reconfigure the environment to match available hardware configuration.

It turns out there is a better way.

Key Design Principles

The Wonder Machine design boils down to these key principles:

1. Modularity– each of the components of an integrated setup should be installed in a separate directory, and should not access files or otherwise depend on other modules. There is a special common module that holds data and programs shared by all modules – so module installations may assume that the common module exists, but may not rely on the existence of other, non-common modules. It is OK to connect to other components over the network, but an installation should not reference a file or directory in another, non-common module. To preserve modularity, we need to adhere to a specific directory structure.

2. Logical server names– any connection URLs and hostnames specified as part of the installation should use logical server names instead of DNS names, hostnames, IP addresses, anything that is tied to the physical layout of the components or networking landscape. We want to be network-independent! Logical server names are abstract names tied to the function of a component, rather than any specific host on the network. For example, when specifying the URL to connect to the CC&B application server, we would use the logical server name ccbappserver; to connect to the SOA Suite server we could use soaserver, and so on – whether these components happen to be running on the same physical or virtual server, or on different servers. The logical server names are translated to the IP addresses using the hosts file. Then, if the networking landscape changes, all we have to do is update the hosts file with the new IP addresses – no need to reconfigure the applications or integrations!
   Note that network independence is a very strong requirement – when implemented correctly, there should be no references to any IP addresses, hostnames, DNS names or any other names beside the logical server names anywhere in the installation of the applications or the integrations. All the integration/functional scenarios should work fine after the environment is moved to a different IP address, or if the hostname or DNS name of the server is changed, or if the server is cloned on the same network. Any network-dependent names or addresses can cause problems when the environment is cloned, or when components are split apart onto separate servers. For example, after the components are split apart, the connections may no longer work as the target component is no longer running on the same VM or IP address as the calling component. Worse, a component of the cloned environment could be making connections to a component of the original environment. This type of situation can be very hard to test and troubleshoot (it all appears to work while you are building it), so you should take utmost care when installing and configuring software in this fashion. There are situations where the hostname will slip into the configuration without your active participation – it may be defaulted for you or substituted without your knowledge by the installation program. When installing a module, it is a good practice to set the hostname of the server to match the logical server name of the module (i.e. set the server's hostname to ccbappserver). This helps ensure that even configuration items that would be silently defaulted to the hostname are made to use the logical server name instead. After the installation and deployment of the component is completed, the server's hostname can be changed back to the original (or to the logical server name of another component that will be installed next).
   Using localhost in the configuration of the server is not allowed. Because localhost is always mapped to IP address 127.0.0.1, it implies that the target of the connection resides in the same server (same OS instance) as the program making the connection. This may be true when the environment is being built, but it may no longer be true after it is split among multiple VMs.
   Normally, if all the components are installed on a single VM, all the logical server names can be pointed to 127.0.0.1 in the hosts file – all the network connections between components are internal to the VM. This way, multiple copies of the environment can run on a single network with no need of reconfiguration and no host file updates. We only need to update the hosts file when splitting the environment to run on two or more VMs.

Implementation Features

The above two points define the essence of the Wonder Machine. In addition, there are some design choices that we have made in the Wonder Machine implementation at Oracle Utilities Global Business Unit (UGBU). They are not absolute requirements, but they make the Wonder Machine easier to distribute to end users, easier to clone and deploy on hosted servers, and more practical to run on laptops with limited memory.

1. Wonder Machine is meant to be a virtualized environment. This is driven by the ease of distribution (non-technical end users can download a VM and run it); this also allows us an easy way to archive environments and build a library of environments. At Oracle, we use Oracle VM VirtualBox to build Wonder Machine environments and to run them on laptops and desktops, but the Wonder Machine itself is not dependent on a particular virtualization tool. Once an environment is built, we often transfer it to a server as a runtime copy (to be used in demos). We use virtualized servers that run on Oracle VM. In some cases it may be more convenient to deploy the Wonder Machine directly on a physical server or laptop, without a virtualization layer.

2. All the components needed for an integrated scenario are installed on a single VM. It would be possible to create separate VMs for each application and for the integration layer, but this would have several practical drawbacks. We would usually need to run two or more VMs per laptop when demonstrating integration, and that would incur unnecessary overhead. We would be running multiple copies of the guest operating system – which would increase the memory and disk space footprint. An integrated setup that would just fit in a 16 GB laptop when built as a single VM, might require two or more laptops when each component is a separate VM.
   This makes the Wonder Machine VMs rather large. Fortunately, we are able to deliver slimmed down VMs to the end users by distributing subsets of the original master environment. Thanks to the modular design, we can simply remove the modules that are not needed in a particular situation and deliver only what is required – this reduces the disk space requirement and the download time for the end user. For example, we may build a CC&B - MDM - ODM integrated environment on our master Wonder Machine VM. From there we can create several end user packages: the full CC&B - MDM - ODM integration, the two-point CC&B - MDM and MDM - ODM integration packages, and the single application packages for CC&B, MDM and ODM. There is no need to build the single application VMs separately from the integrated environment – with the modular design we can reuse the work.

3. All the components needed for the largest integrated scenario are installed on a single VM. This makes the Wonder Machine VM even larger. We want to be able to show the complete end to end business process. Although CC&B - MDM and MDM - ODM are two distinct integrations, we want to be showing them in a single environment, because that's what makes sense from the business point of view. Thus our demo environment needs to include all the applications and all the integrations that might be needed for the largest end to end process. If the solution relevant to a particular client does not include some applications, it is easy to cut the bigger integrated setup down to a subset (e.g. CC&B - MDM). It is much harder to combine two-point integrated environments that were built separately (e.g. CC&B - MDM and MDM - ODM) into a single environment (CC&B - MDM - ODM).

4. Wonder Machine environments are built on Oracle Linux, in a VirtualBox VM. Although, with the right level of expertise and effort, a similar environment could be built on Windows, with Linux we get a universe of useful free tools and readily available online advice. One tool that turned out to be particularly important to us is rsync – we use it to clone and transfer the environments to hosted servers.

5. Wonder Machine environments need to be reliable and easy to use. Most of the intended users of our demo environments are non-technical, and they are generally not familiar with a UNIX or Linux environments. They use the environments under stressful conditions – the portable VM hosted on a laptop needs to be started at the client site, minutes before the demo is to start – when chances of human error are high. Therefore, making the Wonder Machine environments reliable, intuitive, simple and easy to use is critical for the success of the Wonder Machine project, and the user experience aspect is a very important part of building Wonder Machine environments. Here are some design features meant to support usability:
   • Graphical desktop. All Wonder Machine environments should provide the user with a graphical desktop environment (using GNOME) starting by default. Servers can be started and shut down using desktop shortcuts.
   • Server start/stop scripts. For those users who prefer a command line interface, and to facilitate management of the environments hosted in a data center, Wonder Machine environments provide simple shell scripts to start and stop the servers. Desktop shortcuts and automatic startup/shutdown call these scripts to ensure consistent results, regardless of the method used to start or stop a server.
   • Automatic startup and shutdown of the applications on system startup and shutdown. This is normally used for environments hosted in a data center, to minimize management workload and improve availability (the environments should start automatically following data center maintenance or reboot for other reasons, without manual intervention).
   • Supporting utilities. Wonder Machine environments come with a set of utilities (database autostart/stop, SQL*Developer, scripts for export/import of OUAF databases, migration scripts, Samba for file sharing, VNC for graphical remote access) that the end user can rely on.
   • Minimize size. We try to minimize the size of complete Wonder Machine package – both the disk space required to run the VM on the user's laptop and the size of the compressed package that needs to be downloaded. To start, we should clean up the installation and do not leave behind files that are no longer needed (such as the installation media). As part of the packaging process, we try to recover the unused space by compacting the virtual disks in VirtualBox, and replace disks that do not contain valuable information (swap, tmp and work disks) with the original version that has not been written to (and is therefore very small). Finally, we compress the VM using "maximum" or "ultra" settings of the compression utility. We'll cover the packaging process in detail in a later post.
   • Consistent user experience. It is very important that the look and feel, and the ways users interact with the environment (e.g. start/stop scripts, supporting utilities, application URLs and login credentials) are consistent between different Wonder Machine packages and between different applications within a package. This includes the end user experience, and extends to administration and maintenance (e.g. applying patches). It implies following patterns in directory structure, and scripting, and deviating from the patterns only when there is a good reason. There are often multiple ways to accomplish the same result, and they may be equally good from the technical perspective. It is often much easier, especially when the work is done by multiple people, to just make a choice and move on with the task; it takes more effort to make sure that the pattern is consistently followed – but following the pattern results in a higher quality product. Consistency makes the environments easier to learn and more intuitive to use. Consistency reduces human error. This is a point that cannot be stressed enough.
   • Managed change. The above is not to mean that there must not be any change. Great is the enemy of good. When making changes, we simply need to consider the impact on the users and roll them out so as to minimize disruption.

It follows that, although the concepts are broad and open, the guidelines for building a Wonder Machine environment need to be quite prescriptive.

How Does This Work?

Let us revisit the example of the CC&B - MDM - ODM integration with an environment built as above. We have a single VM that has been built to contain the entire integrated setup: three applications and the integration layer. If we have a laptop with sufficient memory (16 GB), we can just run all components of the VM within the single laptop, no changes required.

Here is the hosts file inside the VM:

127.0.0.1    tugbu-olvm-1 localhost
127.0.0.1    ccbappserver ccbdbserver
127.0.0.1    mdmappserver mdmdbserver
127.0.0.1    odmappserver odmdbserver
127.0.0.1    soaserver soadbserver

If we don't have a laptop with 16 GB of memory, but have two laptops with 8 GB each, then we can make a copy of the Wonder Machine on each laptop. We decide that on the first laptop we'll run a VM with only CC&B and MDM, and on the second laptop we'll run a VM with only ODM and SOA Suite – so as not to exceed the available memory:

We connect the two VMs to the network, and update the hosts files within the VMs as follows:

127.0.0.1    tugbu-olvm-1 localhost            
127.0.0.1    ccbappserver ccbdbserver
127.0.0.1    mdmappserver mdmdbserver
192.168.0.5  odmappserver odmdbserver
192.168.0.5  soaserver soadbserver

127.0.0.1    tugbu-olvm-1 localhost 
192.168.0.4  ccbappserver ccbdbserver
192.168.0.4  mdmappserver mdmdbserver
127.0.0.1    odmappserver odmdbserver
127.0.0.1    soaserver soadbserver

Now, because we have built our VM using the logical server names from the beginning, all the integrations work as before, even though initially all connections were made inside a single VM, and now we are connecting between two networked servers, running on two separate pieces of hardware! And because we have built it in a modular way, we can simply delete the inactive modules (marked in light grey) to save disk space. In effect, we have an integrated setup that is a logical collection of modules, which can be run on an arbitrary collection of physical servers (with the constraint that each module must fit in some server, i.e. we cannot split a module that is too big). We did not have to plan for the specific hardware configuration when building the integrated environment, but we did have to plan for this flexibility by building it according to the Wonder Machine guidelines.