Introduction
Image Packaging System (IPS) is a single tier packaging architecture which in Oracle Solaris 11, and other Oracle Sun products such as Oracle Solaris Cluster 4.x, replaces the previous SVR4-based dual tier packaging and patching architecture.
IPS has a number of significant advantages over the old SVR4-based architecture, including:
- Eliminating the need to manage which patches out of a population of thousands should be installed on which systems
- The replacement of free format patch install scripts, which were a common source of error, with predefined actions
- The consolidation of multiple packaging and patching commands into a single 'pkg' command.
In addition, it's implementation in Solaris 11 offers:
- Monthly consolidated bug fix Support Repository Updates (SRUs) providing a regular predictable cadence of engineered together, tested-together release baselines in contrast to the almost daily ad hoc patch release previously
- Significant advantages in Boot Environment cloning (snapshots) enabling low-overhead, rapid, backups/restores, thanks to the mandatory ZFS Root (/) filesystem and associated commands such as 'beadm'.
- Install groups defining the packages needed for common Use Cases, currently: solaris-small-server, solaris-large-server, and solaris-desktop
- Incorporations defining a functional surface, specifying the versions of particular packages which were engineered together and tested together to provide a defined set of feature and bug fix functionality
As we get used to Solaris 11 and IPS, it's natural that users will encounter some issues.
As a novice user myself, I'm documenting here some of the more common Solaris 11 / IPS issues.
This is not designed to be an exhaustive list, but rather the "gottchas" which temporarily stumped either myself (easy to do!) or other non-IPS-expert colleagues over the past year.
Some of the "issues" are more to do with users getting used to conceptual changes.
Some are Caveats resulting from bugs or sub-optimal choices made in early releases. While these have been fixed, their residual impact may still be felt on systems with the affected software installed.
Much of the solutions knowledge below is thanks to two Solaris 11 IPS-expert colleagues of mine, Pete Dennis and Albert White, who I've been pestering unmercifully about IPS issues over the past year. It was either that, or I'd have to RTFM!
Concepts
Repositories, Publishers, and the 'pkg' Command
The 'pkg' command is functionally rich. See 'man pkg' and other documentation. When installing or updating packages, it dynamically analyzes the constraints on the target system, including dependencies and other factors defining what may be installed.
IPS is network repository based.
It is expected that most production customers will set-up their own repository behind their firewall and update it periodically with content from the Support Repository published by Oracle.
Many issues where 'pkg' is unable resolve all constraints imposed on a system, is because the required package versions are not available from the Repositories specified.
Sometimes, it is not immediately obvious why a particular package version is required to resolve a constraint, which can leave users scratching their heads.
Therefore, when a 'pkg install' or 'pkg update' command does not provide the anticipated results, check the specified Publishers (i.e. which Repositories are available to that system) and the content of those Repositories.
For example, Solaris 11 bug fix updates are provided by Support Repository Updates (SRUs) which are released monthly. They contain only the incremental changes relative to their base release, e.g. Solaris 11.1. They are designed to be used in conjunction with a Repository containing that base release.
If the system is already installed with that base release, and the user is just updating existing installed packages, as opposed to installing additional packages, then the user can often get away with just using the SRU on its own.
However, if a bug fix in the SRU has added a dependency on a package which is not installed on the target system, and that package is in the base release rather than the SRU, then an update to that SRU will fail if the base release is not available to enable the dependent package to be pulled in and installed.
For example, a bug fix to the 'thunderbird' package in Solaris 11 11/11 SRU4 to fix font displays resulted in a new dependency being added to the Solaris 11 11/11 'fonts' package. Since the 'fonts' package hadn't changed since the initial Solaris 11 11/11 release, it wasn't included in the SRU, so access to the base Solaris 11 11/11 release in a Repository was required to resolve the dependency. There was a similar dependency addition in a later Solaris 11 11/11 SRU.
Similarly, if a Publisher is specified but is unavailable, or is not specified but is needed because that Repositroy contains a required package, then 'pkg' will be unable to resolve the constraints and will fail.
Making sure the correct Repository Publishers are defined and accessible, and the content of those Repositories is complete will resolve many package install and update issues.
Install Groups and Incorporations
The concept I've had the most difficulty getting straight in my own head is the relationship between Install Groups and Incorporations.
Install Groups simply specify a list of packages to be installed for common Use Cases. They do not specify the versions of packages to install. Currently, the following Install Groups exist in Solaris 11:
- solaris-small-server - the current minimum Install Group, for use by security conscious customers
- solaris-large-server - a superset of solaris-small-server which includes additional useful Sys Admin utilities and network protocols
- solaris-desktop - for use where Solaris will be providing a desktop environment to users
Note the Install Group names solaris-small-server and solaris-large-server have nothing to do with the size of the server, rather it's the size of the solaris footprint on the server. Note also, that solaris-desktop is not a superset of the other two.
The use of Install Groups is not mandatory. They are simply provided for ease of use. Additional packages can be specified in addition to these Install Groups, for example, to resolve additional application dependencies.
Incorporations specify the versions of packages which should be installed together to provide a set of functionality, called a surface. Incorporations exist for various consolidated sub-components of Solaris 11, such as the osnet-incorporation for the core Operating System and NETworking:
gerryh@dublin:~$ pkg info osnet-incorporation
Name: consolidation/osnet/osnet-incorporation
Summary: OS/Net consolidation incorporation
Description: This incorporation constrains packages from the OS/Net
consolidation.
Category: Meta Packages/Incorporations
State: Installed
Publisher: solaris
Version: 0.5.11
Build Release: 5.11
Branch: 0.175.1.3.0.4.2
Packaging Date: Wed Jan 02 19:28:00 2013
Size: 6.22 kB
FMRI: pkg://solaris/consolidation/osnet/osnet-incorporation@0.5.11,5.11-0.175.1.3.0.4.2:20130102T192800Z
The "entire" Incorporation defines what constitutes the version of the
entire Solaris Operating System, for example the "entire" Solaris 11.1 SRU3.4
release:
gerryh@dublin:~$ pkg info entire
Name: entire
Summary: entire incorporation including Support Repository Update (Oracle Solaris 11.1 SRU 3.4).
Description: This package constrains system package versions to the same
build. WARNING: Proper system update and correct package
selection depend on the presence of this incorporation.
Removing this package will result in an unsupported system. For
more information see https://support.oracle.com/CSP/main/article
?cmd=show&type=NOT&doctype=REFERENCE&id=1501435.1.
Category: Meta Packages/Incorporations
State: Installed
Publisher: solaris
Version: 0.5.11 (Oracle Solaris 11.1 SRU 3.4)
Build Release: 5.11
Branch: 0.175.1.3.0.4.0
Packaging Date: Wed Jan 02 19:31:02 2013
Size: 5.46 kB
FMRI: pkg://solaris/entire@0.5.11,5.11-0.175.1.3.0.4.0:20130102T193102Z
While it's possible to remove other Incorporations, hence removing the package version constraints they impose, it is unsupported to remove the "entire" Incorporation.
Also, please don't remove other Incorporations unless you are confident you know what you're doing!
When installing a Solaris system, it is common to specify both an Install Group - i.e. which packages to install - and a version of the "entire" Incorporation - i.e. which versions of those packages to install.
For example, this could be specified in an AI (Automated Installer) manifest, along with any additional IPS products or packages required. Here's part of an AI manifest my team uses to install SPARC SuperClusters with Solaris 11 11/11 SRU12.4 as well as other tools from a separate "Exa-family" tools Repository which is specifically for Engineered Systems:
<software type="IPS">
<source>
<publisher name="solaris">
<origin name="http://pkg.us.oracle.com/solaris11/support/"/>
</publisher>
<publisher name="exa-family">
<origin name="http://pkg.us.oracle.com/exa-family"/>
</publisher>
</source>
<software_data action="install">
<name>pkg://solaris/entire@0.5.11,5.11-0.175.0.12.0.4.0:20121002T161728Z</name>
<name>pkg://solaris/group/system/solaris-large-server</name>
<name>pkg://solaris/compatibility/packages/SUNWxwplt</name>
<name>pkg://solaris/developer/java/jdk-6</name>
<name>pkg://solaris/developer/library/lint</name>
<name>pkg://solaris/library/libfuse</name>
<name>pkg://solaris/library/motif/libdpstkxm</name>
<name>pkg://solaris/library/perl-5/database-584</name>
<name>pkg://solaris/library/perl-5/net-ssleay-584</name>
<name>pkg://solaris/library/perl-5/pmtools-584</name>
<name>pkg://solaris/library/perl-5/xml-simple-512</name>
<name>pkg://solaris/library/perl-5/xml-simple-584</name>
<name>pkg://solaris/runtime/perl-584</name>
<name>pkg://solaris/system/file-system/uvfs</name>
<name>pkg://solaris/system/header</name>
<name>pkg://solaris/x11/server/xvnc</name>
<name>pkg://solaris/runtime/tk-8</name>
<name>pkg://exa-family/system/platform/exa-family/oswatcher@0.5.11,5.11-01.0.11:20120726T024155Z</name>
<name>pkg://exa-family/system/platform/supercluster@0.5.11,5.11-1.1.0.1:20121114T163635Z</name>
<name>pkg://exa-family/system/management/hmp/hmp-ipmitool@1.8.10.4,5.11-3:20111102T203714Z</name>
<name>pkg://exa-family/system/platform/exadata/compmon@0.5.11,5.11-1.1.0.13:20121121T024216Z</name>
<name>pkg://exa-family/system/platform/supercluster/iscsi@0.5.11,5.11-1.1.0.3:20121128T031803Z</name>
<name>pkg://exa-family/system/platform/supercluster/setcoremem@0.5.11,5.11-1.1.0.1:20121113T173705Z</name>
<name>pkg://exa-family/system/platform/supercluster/ssc_exavm@0.5.11,5.11-1.1.0.3:20121128T031805Z</name>
<name>pkg://exa-family/system/platform/supercluster/ssctuner@0.5.11,5.11-1.1.0.1:20121113T173712Z</name>
<name>pkg://exa-family/system/platform/supercluster/vnet@0.5.11,5.11-1.1.0.3:20121128T1808Z</name>
</software_data>
</software>
</ai_instance>
</auto_install>
Now for the bit which always confuses me. Strong coffee helps!:
Installing an Incorporation does not, by itself, install any packages. Rather, the Incorporation specifies the constraints on package versions if they are present on the system.
So 'pkg install entire' on a bare metal system does nothing, unless other packages are specified upon which the constraints specified in the Incorporation are to operate - e.g. an Install Group package such as solaris-large-server.
# create a 'bare' metal image to play with
$ pkg image-create -p http://pkg.us.oracle.com/solaris11/release bare_metal
# what is in this image:
$ cd bare_metal
$ pkg -R `pwd` list
pkg: no packages installed
# Install entire
$ pkg -R `pwd` install --accept entire
:
Packages to install: 28
:
# It installed 28 packages! What are they ?
$ pkg -R `pwd` list
NAME (PUBLISHER) VERSION IFO
consolidation/SunVTS/SunVTS-incorporation 0.5.11-0.175.1.0.0.14.0 i--
consolidation/X/X-incorporation 0.5.11-0.175.1.0.0.24.1317 i--
consolidation/admin/admin-incorporation 0.5.11-0.175.1.0.0.5.0 i--
:
consolidation/xvm/xvm-incorporation 0.5.11-0.175.1.0.0.5.0 i--
entire 0.5.11-0.175.1.0.0.24.2 i--
# These are all Incorporations specified by 'entire'. There is no actual software payload installed at all.
But once Solaris is installed on a system, updating an Incorporation, for example, using 'pkg update entire', updates the constraints, causing the relevant packages which are installed to be updated by IPS to the later functional "surface" specified by the Incorporation.
So if, for example, the new version of the Incorporation specifies 'foo@1.24' and specifies it has a new dependency on 'bar@1.13' and package 'foo' is already installed, say @ Version 1.20, then updating the Incorporation tells IPS to update 'foo' to Version 1.24 and install 'bar' at Version 1.13 if it hasn't already been installed (from whichever specified Repository/Repositories contains these packages at these versions).
Ask and You Shall Receive - In Abundance!
You can have too much of a good thing. Like information. Which can make it hard to see the wood for the trees when trying to debug a 'pkg update' issue.
When issues occur, 'pkg' is verbose about the potential issues that could be causing the problem.
Packages will have dependencies upon other packages. These dependencies may not only be satisfied by the explicit version mentioned but also by a later version of a package.
This means that if 'pkg' is unable to solve for all dependencies given the available Publishers specified, the contents of those Repositories, and the constraints specified for the target system, then 'pkg' will produce a list of the all the dependencies that could not be satisfied. While these errors are all true they, due to the amount of them, can obscure the underlying error.
One way to reduce the amount of errors is to specify the version of the packages that you need to update to.
This is because, by default, 'pkg' will attempt to move to the latest set of packages. If this update fails then it will produce errors for each possible set of packages that could have been updated but were not.
By specifying an explicit version of the packages to update then the errors produced will be relevant to that particular version.
Therefore, rather than just saying:
$ pkg update entire
...explicitly state which SRU you want to update to...
$ pkg update entire@0.5.11,5.11-0.175.0.12.0.4
...which specifies Solaris 11 11/11 SRU 12.4.
There's another good reason to explicitly specify which SRU or package version you want to update to.
If you don't specify a version, 'pkg' will try to update to the latest version which satisfies the constraints on the target system.
If the repository has been updated, this could produce a different result than the same command issued prior to the repository been updated. This may be undesirable if you are trying to update a number of systems to the same level.
Also, if 'pkg' is unable to update to the latest available release due to the constraints on the target system, it will recursively try to update to versions higher than what is already installed.
For example, one of my team issued a 'pkg update entire' to update a test system to SRU4. Only days later did he realize it had actually updated the system to SRU3 as there was a constraint on the system preventing update to SRU4.
Since IPS is not telepathic, it's best to explicitly state what version you want it to update a system to.
All 'pkg' commands are logged.
The use of 'pkg history' is useful to examine the history of the system but additionally to print out the previous errors messages without having to rerun a command that you know is going to fail.
When looking at the history use the -l option ask for details of the interesting command invocation.
All errors are indented with, in most cases, the significant error being indented to the right hand side.
Therefore when looking at issue, use the 'pkg history' command to print out the previous errors and look at the errors that are indented to the right.
Note that the errors themselves may be repeated for each and every package that the update has failed on, so the output may still be verbose.
But the errors are likely to be caused by just one or two issues, such as having the incorrect Publishers specified (too few, too many, or not available) or insufficient content in the Repositories.
Solaris 11 Release and Support Repository Relationship
The Release Repository contains just the Solaris Releases such as the original Solaris 11 11/11 release and the Solaris 11.1 update release.
The Support Repository is only available to customers with a valid support contract. It contains all releases and all monthly Support Repository Updates (SRUs) providing bug fix updates to customers.
As discussed in my previous blog posting, we've implemented a process improvement to remove any 'blackout' period on the release of bug fixes by tweaking the relationship between Update releases and bug fix releases in Solaris 11 compared to Solaris 10 and older.
We still produce periodic Update releases such as Solaris 11.1, containing support for new hardware and enhanced software features (e.g. VM2.0). Update releases also contain a significant number of bug fixes for issues found internally during Solaris testing and more complex customer reported issues which required more test "soak" time than is possible in an SRU. Update releases are intensely tested and hence provide high quality Solaris Baselines. The Solaris Binary Compatibility Guarantee still applies, so users should not experience any compatibility issues crossing a Solaris Update boundary.
Support Repository Updates (SRUs) primarily deliver bug fixes, although they may include some feature enhancements. They too, are intensely tested prior to release. SRUs go through several internal builds prior to release. Once released, additional critical bug fixes can be "back-published" to SRUs. The build number of the SRU is now included in its name to uniquely identify it, e.g. Solaris 11 11/11 SRU13.4 is Build 4 of SRU13 on top of Solaris 11 11/11. Earlier SRUs had a letter suffix to denote "back-published" additional content.
We've improved the process in Solaris 11 so that we can continue to deliver bug fixes for critical issues in SRUs while the content for an Update release is being finalized. This implies that an Update release may not be a superset of the SRU(s) immediately preceding it. Rather, it is the SRU after the Update release which is effectively the superset of both the Update the SRUs preceding the Update release.
The relationship between Update releases and SRUs can be drawn as follows:
Solaris 11 11/11 Solaris 11.1 Solaris 11.2...
\ \ \
SRU1, SRU1A, SRU2, ... SRU12.4, SRU 13.4, SRU1.4, SRU2.5, SRU3.4, SRU3.4.1, ...
Installed systems with a valid support contract should always be updated using SRUs from the Support Repository.
The SRUs are contiguous, just as Kernel patches were contiguous in Solaris 10 and earlier. That is, the next SRU to Solaris 11 11/11 SRU13.4 is Solaris 11.1 SRU1.4. This is similar to the Kernel PatchID progression in Solaris 10 and earlier.
Known Caveats
As developers and release engineers have become used to Image Packaging System and the Solaris 11 eco-system, bugs, "features", and caveats continue to diminish.
Nevertheless, users may be impacted by the residual effects of some of these items.
Here's a non-exhaustive list of potential issues and their workarounds:
Need to accept Java 7 license
Legal insist that users explicitly accept the revised license terms in Java 7.
This means that users must add "--accept" to 'pkg install' or 'pkg update' commands when moving to versions with revised license terms. For example:
$ pkg install --accept entire
"Bunny Hop" update required from pre-Solaris 11 11/11 SRU 10.5 before updating to a Solaris 11.1 SRU
An IPS bug in early Solaris 11 11/11 versions resulted in some x86 packages being installed on SPARC systems and vice versa due to the incorrect resolution of indirect dependencies. This is now fixed.
Solaris 11 11/11 SRU 10.5, SRU 11.4, SRU 12.4, and SRU 13.4 contain functionality to remove the residual effects of the issue - i.e. remove the incorrect architecture packages. This must be completed prior to updating to a Solaris 11.1 SRU.
That is, users must first perform a "bunny hop" update to SRU 10.5, SRU 11.4, SRU 12.4, or SRU 13.4 prior to updating to a Solaris 11.1 SRU.