% ISSUES:
%
%       * some places talk about active interfaces -- and in theory should
%         be active interfaces -- but aren't in implementation because
%         the code doesn't check for up logints.

\newcommand{\cvdocsdir}{/net/clearview.east/export/apache/clearview/docs/}
\newcommand{\teamauthor}{Peter Memishian (meem)}
\newcommand{\docversion}{1.14}

\title{IPMP Rearchitecture: High-Level Design Specification}

\input \cvdocsdir/comfmt.tex

\begin{document}

\input \cvdocsdir/comstart.tex
\section{Introduction}
\subsection{Brief Overview of IPMP}

IPMP is a popular Solaris-specific multipathing technology which operates
at the IP layer.  Specifically, IPMP attempts to insulate IP-based
networking applications from changes to the underlying networking hardware
on the system and the system's connectivity to the network as a whole.
For instance, if two networking interfaces are put into an ``IPMP group'',
then the failure or removal of either interface from the system will not
affect applications using the IP addresses hosted on those networking
interfaces.  Further, the inbound\footnote{Inbound load-spreading has
  never been explicitly documented but many customers are aware of the
  feature.} and outbound networking traffic using those IP addresses will
be load spread across the networking interfaces in the group, providing
greater network utilization.

As networking availability is a widespread concern, several other
multipathing technologies exist in Solaris today: 802.3ad (link
aggregation), Sun Trunking, Routing (OSPF-MP), CGTP, and SCTP.  While a
detailed comparison of these technologies is beyond the scope of this
overview, it's worth summarizing where IPMP fits among them in order to
separate the key goals of IPMP from the current implementation details.

Specifically, IPMP is intended to provide high-availability and improved
network utilization for IP-based networking applications\footnote{This
  includes Java-based networking applications as well.}, while also
satisfying the following requirements:

\begin{itemize}
\addtolength{\itemsep}{-0.5\baselineskip}
\item Must work with all networking cards supported on Solaris.
\item Must not require changes to existing hosts and hardware on the network.
\item Must not require changes to existing applications.
\item Must support all IP-based protocols.
\end{itemize}

The first two requirements are not met by 802.3ad and Sun
Trunking\footnote{Sun Trunking is an add-on product -- not part of base
  Solaris.}, both of which require an 802.3-based link-layer and hardware
(switch) support.  Further, due to implementation limitations, only a few
Solaris network interfaces currently support 802.3ad or Trunking.  The
second requirement is also not met by a routing-based solution, which
requires that all participating systems run a routing protocol, and also
does not support all IP-based applications (e.g., multicast-based
applications are not supported).  Likewise, CGTP\footnote{CGTP is also an
  add-on product -- not part of base Solaris.} requires that all
participating systems run CGTP (e.g., in order to filter duplicate
packets), and is generally only suited to traffic between hosts on the
same link.  Finally, SCTP does not meet the third and fourth requirements,
since applications must be recoded to use it and must use SCTP as the
transport protocol.

Unfortunately, IPMP's high-availability and network utilization features
currently come at a cost.  As detailed in the following section, IPMP is
hard to administer, often impossible to observe, poorly integrated with
other core Solaris features, and quite complicated in implementation.  As
we will see, these problems stem from an improper relationship between IP
interfaces, IP addresses, and IPMP groups.  Since
\htmladdnormallinkfoot{Project Clearview}{http://opensolaris.org/os/project/clearview/} is
specifically focussed on rationalizing the Solaris network interface
abstractions, and moreover, since the current IPMP implementation violates
several of the \htmladdnormallinkfoot{network interface
  requirements}{http://opensolaris.org/os/project/clearview/overview.txt}  imposed by
Clearview, it is both opportune and necessary to rearchitect IPMP as part
of Clearview.

While brief background material is provided herein, {\bf those who have not
regularly used IPMP are {\em strongly} encouraged to read through the
\htmladdnormallinkfoot{IPMP
  chapters}{http://docs.sun.com/app/docs/doc/816-4554/6maoq027c} of
the Solaris 10 System Administration Guide before
proceeding.} For those who need a quick refresher on IPMP terminology,
a brief glossary is provided at the end of this document.

\subsection{Problems with IPMP}

While IPMP has been quite popular with customers, its adoption has been
seriously hampered by a significant number of implementation and design
flaws.  Specifically, IPMP currently:

\begin{itemize}
\item {\bf Provides a frustrating and overwhelming administrative experience.}

  For instance, {\tt ifconfig -a} on a host using IPMP can yield a
  \htmladdnormallinkfoot{bedazzling amount of output}{http://opensolaris.org/os/project/clearview/ipmp/jurassic-ifconfig.txt}, but does not provide easy
  answers to common
  questions such as ``What interfaces are currently active in IPMP group
  {\tt a}?'',  ``What IP addresses are currently available for use in
  group {\tt a}?'' or ``Has IPMP group {\tt a} failed completely?''.
  In addition, despite that many IPMP configurations use a significant
  number of IP addresses, it is not possible to configure these addresses
  through DHCP. Further, key questions such as ``What probe targets are
  being used on group {\tt a}?'' can only be answered through monitoring
  network traffic.

\item {\bf Promotes a misunderstanding of the relationship between
    addresses and interfaces}

  Because {\tt ifconfig} shows specific IP addresses hosted on specific IP
  interfaces, the administrator is lulled into a fundamental
  misunderstanding that IP addresses are actually ``owned'' by a
  particular IP interface.  This is not true: a packet may be sent over
  {\it any} active interface in the IPMP group, regardless of its source
  address.  Despite attempts to document this issue, this remains a source
  of constant confusion and a call generator.

\item {\bf Migrates IP addresses between interfaces during interface
    failure, repair, offline, and undo-offline operations.}

  This migration confuses applications that operate at the
  network-interface level (such as routing daemons), and prevents the use
  of DHCP to manage the addresses.  For instance, {\tt dhcpagent(1M)} has
  no way to realize that an address that it was managing on {\tt ce0}
  is now on {\tt ce1}. Further, DHCP uses the network interface as
  the administrative handle to manage a lease (e.g., {\tt ifconfig ce0 dhcp
  extend}), but with IPMP, this handle is not stable over the lifetime
  of the lease.

  It also frequently startles and confuses Solaris administrators, since
  it spreads the misconception that the addresses are tied to specific
  interfaces (in truth, all the interfaces in an IPMP group share a pool
  of \emph{all} of the group's addresses).

  Similarly, when operating in a zone, IP interfaces seem to shift around
  as if by magic.  For instance, suppose {\tt ce0} and {\tt ce1} are
  together in a group in the global zone, and the IP address used by a
  shared-stack local zone is initially hosted on {\tt ce0:1}.  If {\tt
    ce0} fails, a subsequent {\tt ifconfig -a} invocation in the local
  zone will no longer show {\tt ce0:1}, but instead show the logical
  interface on {\tt ce1} that the address migrated to (e.g, {\tt ce1:2}).

\item {\bf Provides no facility for interacting with the traffic flowing
    through the IPMP group as a whole -- instead, each network interface
    in the group must be operated on individually.}

  Since any interface in the IPMP group may be used to send or receive
  traffic, debugging network problems using tools like {\tt snoop} is made
  much more complicated: each of the interfaces in the group must be
  individually monitored, and the resulting packet traces must be
  carefully merged to get a complete view of the relevant traffic.

  Similarly, this introduces problems for packet filters and other
  facilities that associate policy with network interfaces: since
  administrators cannot apply or observe policies associated with the
  group as a whole, they must instead carefully apply the policy to each
  of the group's underlying interface\footnote{Though the {\tt
      ipmp\_hook\_emulation ndd} tunable provides short-term relief.}.
  Further, this policy must be manually updated whenever interfaces are
  added or removed from the group.

  A more serious problem exists with stateful firewalls: although packets
  may be sent and received using any interface in the group, the firewall
  is unaware of the relationship between the interfaces and thus cannot
  share state across the interfaces in the group.  This renders stateful
  firewalls unusable with IPMP\footnote{Again, the {\tt
      ipmp\_hook\_emulation ndd} tunable provides short-term relief.}.

\item {\bf Introduces confusing and non-deterministic routing table semantics.}

  Routes are added to the routing table using the underlying network
  interfaces that make up the IPMP group.  However, to provide reasonable
  IPMP semantics, those routes actually apply to the whole group, rather
  than just the specified network interface.  Besides being generally
  misleading to administrators and applications, this leads to a number of
  ugly corner cases, such as what to do when the interface is removed
  from the group (or from the system!)

  Additionally, one is not allowed to add routes that use a failed
  interface -- even though the route applies to the group as a whole and
  though there are other functioning interfaces in the group.  Since there
  is no way to predict when an interface will fail, this introduces a
  non-deterministic point of failure.

\item {\bf Provides error-prone and incomplete routing socket
    ({\tt route(7P)}) semantics.}

  Applications such as routing daemons that make use of routing sockets
  get an incomplete and confusing picture.  For instance, if an address
  migrates from {\tt ce0} to {\tt ce1} as a result of a failure on {\tt
    ce0}, then an application will first see the addresses for {\tt ce0}
  disappear (and potentially have to notify other systems that the address
  that was on {\tt ce0} is no longer reachable), only to later see the
  address reappear on {\tt ce1} (and potentially re-notify systems that the
  address is again reachable). For routing, this odd behavior can trigger
  failsafe mechanisms (e.g., in Cisco's IOS), such as route flap dampening
  and hold-downs, can lead to extended outages.

  Further, there is no way to observe the IPMP group as a whole using
  routing sockets -- only the underlying interfaces that comprise it can
  be observed.

\item {\bf Exposes IPMP test addresses and {\tt INACTIVE}, {\tt OFFLINE},
    and {\tt FAILED} IP interfaces to unsuspecting applications.}

  Applications that need to discover and use IP addresses or IP interfaces
  on the system must be explicitly updated to ignore IPMP test addresses
  and IP interfaces that IPMP has deemed unusable.  Specifically, the
  well-known {\tt SIOCGLIFCONF ioctl} returns \emph{all} IP addresses and
  interfaces, even interfaces that have been marked {\tt OFFLINE}, {\tt
    INACTIVE}, or {\tt FAILED}, and also returns IPMP test addresses.  The
  latter is particularly troublesome, because the application appears to
  work normally until the interface hosting the test address fails.

\item {\bf Comprises thousands of lines of complicated, subtle, and
    fragile kernel code.}

  The code to support address migration (failover and failback) in the
  kernel is extremely complicated, involving more than 7500 lines of
  subtle and fragile logic that exists solely to support IPMP.  This
  support impacts the core data structures, such as {\tt ipsq\_t}, which
  have special support for merging with and splitting from other {\tt
    ipsq\_t} structures just for IPMP address migration.

\end{itemize}
\pagebreak
\section{Requirements}
\label{req}
Simply put, the requirements are to address all of the aforementioned
flaws without affecting backward compatibility. Specifically:

\begin{enumerate}

\item IP addresses in an IPMP group must never migrate as part of failure,
  repair, offline, or undo-offline operations.

\item IP addresses in an IPMP group must be administered using the same
  tools and in the same manner as IP addresses not placed in an IPMP
  group.  As a corollary, address autoconfiguration through DHCP must
  work.

\item The IPMP group's core configuration and current state must be able
  to be easily and concisely obtained.

\item The IPMP group as a whole must be observable through packet
  monitoring tools such as {\tt snoop(1M)}.  It must also still be
  possible to observe individual interfaces as well.

\item The IPMP group as a whole must be able to administered as a whole --
  e.g., specified directly to {\tt route(1M)}, and visible directly
  through {\tt netstat(1M)}.

\item IPMP test addresses must not be visible to applications unless
  explicitly requested.

\item Migration of {\tt UP} addresses must be obviated (to prevent complex
  migration code in the kernel).

\item The documented IPMP administrative commands ({\tt if\_mpadm},
  {\tt ifconfig}, and {\tt cfgadm}) must continue work as before.

\item The documented IPMP semantics, e.g., for failure detection, repair
  detection, outbound interface selection and source address selection
  must continue to operate as before.

\item The documented IPMP configuration files
  ({\tt /etc/hostname.{\it if}} files) must continue to operate as
  before.

\end{enumerate}

All of these requirements can be met by changing the core IPMP
architecture such that each IPMP group is modeled as an IP interface
(rather than as a collection of loosely federated IP interfaces), and
introducing a new tool to easily and concisely obtain the core
configuration and current state of each group.

Note that because each IPMP group will be modeled as an IP interface,
\htmladdnormallinkfoot{Clearview's list of interface requirements must be
  followed}{http://opensolaris.org/os/project/clearview/overview.txt} as
well.  This adds one additional requirement: vanity naming.  That is, {\bf
  the name of the IP interface representing each IPMP group must be
  configurable by the administrator.}

The remainder of this document describes the proposed new model, and
contrasts it to the existing model where possible.  The following section
provides an overview of how core IPMP concepts and operations will be
represented in the new model. The next two sections discuss the
administrative and programmatic impact of the new model.  Finally, the
final section revisits the requirements and summarizes how they will be
satisfied by the proposed rearchitecture.  Note that since this is a
high-level design document, a detailed design of each specific change will
not be provided.

\pagebreak
\section{IPMP Rearchitecture: Basic Operation}

\subsection{IPMP Network Interface}
\label{basipmpif}
As previously discussed, the lion's share of the problems with IPMP stem
from not treating each IPMP group as its own IP interface.  As an
example, a typical two-interface IPMP group today looks like:

\begin{alltt} {\footnotesize
{\bf{ce0}}: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
       inet 129.146.17.56 netmask ffffff00 broadcast 129.146.17.255
       groupname a
ce0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
       inet 129.146.17.55 netmask ffffff00 broadcast 129.146.17.255
{\bf{ce1}}: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
       inet 129.146.17.58 netmask ffffff00 broadcast 129.146.17.255
       groupname a
ce1:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
         inet 129.146.17.57 netmask ffffff00 broadcast 129.146.17.255
}\end{alltt}

The above output shows {\tt ce0} and {\tt ce1} with test addresses, and
{\tt ce0:1} and {\tt ce1:1} with data addresses\footnote{Where feasible,
  this configuration will be used for examples throughout the document.}.
If {\tt ce0} subsequently fails, the data address on {\tt ce0:1} will be
migrated to {\tt ce1:2}, but the test address will remain on {\tt ce0}
so that the interface can continue to be probed for repair.

In the future, this will instead look like:

\begin{alltt} {\footnotesize
ce0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
         inet 129.146.17.56 netmask ffffff00 broadcast 129.146.17.255
         groupname a
ce1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
         inet 129.146.17.58 netmask ffffff00 broadcast 129.146.17.255
         groupname a
{\bf{ipmp0}}: flags=801000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,{\bf{IPMP}}> mtu 1500 index 4
         inet 129.146.17.55 netmask ffffff00 broadcast 129.146.17.255
         groupname a
{\bf{ipmp0:1}}: flags=801000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,{\bf{IPMP}}> mtu 1500 index 4
         inet 129.146.17.57 netmask ffffff00 broadcast 129.146.17.255
}\end{alltt}

That is, all of the IP data addresses associated with the IPMP group will
instead be hosted on an IPMP IP interface, such as {\tt ipmp0}\footnote{In
  the new model, had the data addresses been hosted on the physical
  interfaces, then the test addresses would have appeared on {\tt ce0:1}
  and {\tt ce1:1}, with {\tt ce0} and {\tt ce1} containing {\tt 0.0.0.0}
  placeholder addresses.  On a configuration without test addresses, there
  would {\em only} be {\tt 0.0.0.0} placeholder addresses.}.  With this
new model, {\bf data addresses will no longer be associated with any
  specific physical underlying interface, but instead will belong to the
  IPMP group as a whole}.  As will become clear, this addresses many
outstanding problems and vastly simplifies the implementation.

There will be a one-to-one correspondence between IPMP groups and IPMP
interfaces.  That is, each IPMP group will have exactly one IPMP
interface.  By default, each IPMP interface will be named {\tt
  ipmp\emph{N}}, but administrators will be encouraged to specify
a name of their choosing, as described in section \ref{admipmpcreate}.
Since an IPMP interface's name will not be fixed, the system will set a
new {\tt IPMP} flag on all IPMP interfaces to indicate that the interface
has special properties and semantics, as detailed throughout this
document.

For clarity, this document uses the {\tt ipmp\emph{N}} naming convention
for IPMP interfaces unless another format is necessary to explain a given
topic.

\subsection{Outbound Load-Spreading}

Currently, the IP module load-spreads outbound traffic for each IP
destination address across any \emph{active} interface in the group, where
\emph{active} is defined as an interface that is not marked {\tt FAILED},
{\tt INACTIVE}, or {\tt OFFLINE}, and has at least one {\tt UP}
address\footnote{{\tt UP} is an address property rather than an interface
  property, but the IP module has always assumed that an interface must
  have at least one {\tt UP} address before it can send or receive
  packets.}.  This will be unchanged in the new model.

\subsection{Source Address Selection}
\label{basipaddrsel}
Currently, if the application does not bind to a specific source address,
the IP module will select a source address from the set of addresses in
the IPMP group.  The purpose of varying the source address is to affect
inbound load-spreading.  Specifically, in the current model, all addresses
hosted on a given interface are associated with that interface's hardware
address.  Thus, currently, if one connection uses a source address on
{\tt ce0}, and another a source address on {\tt ce1}, then each
connection will use a different interface for inbound traffic. It's
important to note that {\bf choice of outbound interface is independent from
choice of source address}.  Thus, it is quite common to have a
connection which sends packets through {\tt ce0}, using a source
address hosted on {\tt ce1}, and thus receives the response on
{\tt ce1} (this is necessary to spread outbound load when the set of
available source addresses is smaller than the set of available outbound
interfaces).

Since data addresses will now be hosted on the IPMP interface, this will
change slightly in the new model.  Specifically, the IP module itself will
have the responsibility of associating each available IP address in the
IPMP group with a specific active underlying interface, and ensuring that
the set of addresses remain evenly distributed over the active interfaces.
In particular, source addresses may need to be redistributed as available
addresses are added or removed from the group, or as a result of an
active interface becoming unusable or {\tt INACTIVE}.

As before, all packets to a specific destination will use the same source
address.  Both the IPv4 and IPv6 source address selection algorithms will
remain unchanged.  The IPv6 algorithm is described in {\tt inet(7P)}.
The IPv4 algorithm is undocumented, but will remain:

\begin{enumerate}
\addtolength{\itemsep}{-0.5\baselineskip}
\item Non-deprecated source addresses on the same subnet as the next-hop
  destination\footnote{{\it next-hop destination} is the next-hop router if
      the destination is off-link, or the destination itself if it is
      on-link.}.
\item Non-deprecated source addresses on a different subnet.
\item Deprecated source addresses on the same subnet as the next-hop destination.
\item Deprecated source addresses on a different subnet.
\end{enumerate}

Also as before, if after performing the algorithm more than one IP address
remains, the IP module will ``randomly''\footnote{We use the term loosely
  as the implementation need not actually use an RNG; anything that varies
  the IP source address will suffice.} pick from the remaining set of IP
addresses.

An implementation artifact currently allows test addresses to be
considered as source addresses.  However, since all IPMP groups are
required to have at least one data address, they will never be chosen by
the above algorithm (unless all data addresses are marked {\tt
  deprecated}, in which case the current code erroneously selects a test
address).  In the new model, the test addresses will \emph{never} be
considered because underlying interfaces will be ignored during source
address selection.  However, since the administrator may still choose to
mark some of the IPMP data addresses as deprecated, the source address
selection algorithm will continue to prefer non-deprecated addresses to
meet the guarantees stated in {\tt ifconfig(1M)}.

As before, only {\tt UP} non-zero addresses in the same zone will be
considered.

\subsection{Interface Failure and Repair}
\label{basfailrepair}
Currently, when {\tt in.mpathd} detects that an interface can no longer be
used to access the network, it marks it {\tt FAILED}, {\emph{and}}
triggers a failover operation to move the addresses to another functioning
interface in the group.  However, with the IPMP interface, the address
migration step will no longer be necessary.  For instance, if {\tt
  in.mpathd} detects that {\tt ce0} has failed, it will just need to mark
{\tt ce0} as {\tt FAILED}\footnote{If the interface was
  active, an {\tt INACTIVE} interface in the group may also be activated --
  see sections \ref{basstandby} and \ref{basfailbackno}.}:

\begin{alltt}{\footnotesize
ce0: flags=19040843<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,{\bf{FAILED}}> mtu 1500 index 2
         inet 129.146.17.56 netmask ffffff00 broadcast 129.146.17.255
         groupname a
ce1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
         inet 129.146.17.58 netmask ffffff00 broadcast 129.146.17.255
         groupname a
ipmp0: flags=801000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP> mtu 1500 index 4
         inet 129.146.17.55 netmask ffffff00 broadcast 129.146.17.255
         groupname a
ipmp0:1: flags=801000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP> mtu 1500 index 4
         inet 129.146.17.57 netmask ffffff00 broadcast 129.146.17.255
}\end{alltt}

If {\tt ce1} were to then also fail, it would also be marked {\tt FAILED}
(and of course no migration would occur).  In addition, the group itself
will be marked {\tt FAILED}, making it clear that the entire IPMP group is
unusable (note that the new {\tt ipmpstat} tool discussed in section
\ref{admipmpstat} will bring more appreciable administrative improvements,
including fine-grained IPMP group and interface state information using
established FMA terminology.)

\begin{alltt}{\footnotesize
ce0: flags=19040843<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,{\bf{FAILED}}> mtu 1500 index 2
         inet 129.146.17.56 netmask ffffff00 broadcast 129.146.17.255
         groupname a
ce1: flags=9040843<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,{\bf{FAILED}}> mtu 1500 index 3
         inet 129.146.17.58 netmask ffffff00 broadcast 129.146.17.255
         groupname a
ipmp0: flags=801000843<UP,BROADCAST,MULTICAST,IPv4,IPMP,{\bf{FAILED}}> mtu 1500 index 4
         inet 129.146.17.55 netmask ffffff00 broadcast 129.146.17.255
         groupname a
ipmp0:1 flags=801000843<UP,BROADCAST,MULTICAST,IPv4,IPMP,{\bf{FAILED}}> mtu 1500 index 4
         inet 129.146.17.57 netmask ffffff00 broadcast 129.146.17.255
}\end{alltt}

As one would suspect, once {\tt in.mpathd} detects that the interface can
be used again, it will only need to clear the appropriate {\tt FAILED}
flags -- as with interface failure, address migration will no longer be
needed.

\subsection{Interface Offline and Undo-Offline}

Currently, data addresses are also migrated to another functioning
interface as part of an administratively requested offline or undo-offline
operation (if no interface is functioning, the operation fails).
Following the migration, the {\tt OFFLINE} flag is set to indicate that
the interface is now unusable, and any test addresses are brought down to
complete the offline operation (the opposite is done on an undo-offline).

Again, since data addresses will no longer be hosted on the physical
interfaces, migration step will no longer be necessary.  Instead, an
offline operation will consist of setting the {\tt OFFLINE} flag and
bringing down the test addresses\footnote{As with interface failure,
an {\tt INACTIVE} interface in the group may also be activated -- see sections
\ref{basstandby} and \ref{basfailbackno}.}. For instance, if {\tt ce0} in
the previous example is offlined, the resulting configuration will be:

\begin{alltt}{\footnotesize
ce0: flags=9040843<BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,{\bf OFFLINE}> mtu 1500 index 2
       inet 129.146.17.56 netmask ffffff00 broadcast 129.146.17.255
       groupname a
ce1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
       inet 129.146.17.58 netmask ffffff00 broadcast 129.146.17.255
       groupname a
ipmp0: flags=801000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP> mtu 1500 index 4
       inet 129.146.17.55 netmask ffffff00 broadcast 129.146.17.255
       groupname a
ipmp0:1: flags=801000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP> mtu 1500 index 4
       inet 129.146.17.57 netmask ffffff00 broadcast 129.146.17.255
}\end{alltt}

\subsection{Dynamic Reconfiguration}
\label{basdr}
Dynamic Reconfiguration support is provided by the Solaris Reconfiguration
Coordination Manager (RCM) framework and built upon the aforementioned
offline and undo-offline operations.  Specifically, when removal of a
network card is requested, RCM first offlines the interface in the manner
previously described.  Next, any test addresses assigned to the interface
are removed, and finally the interface is unplumbed from the system.

A card insertion operation does the reverse: RCM replumbs the interface,
rebuilds the IP configuration of the interface from the system's {\tt
  /etc/hostname.{\it if}} file, and then finally performs an undo-offline
operation to restore the interface to its previous state.

While this sounds straightforward, the current implementation is
complicated significantly by the need to handle data address migration.
In particular, because the kernel uses interface indices to track which
data addresses belong to which interfaces, and because the interface index
of the inserted interface may be different than the removed interface, RCM
must interact with {\tt in.mpathd} and the kernel to change the
``original'' interface index of each data address to the inserted
interface's index.  In addition, the RCM code duplicates a significant
piece of the {\tt ifconfig} command-line interpreter in order to parse the
{\tt /etc/hostname.{\it if}} file.

This complexity will be eliminated in the new model.  In particular, since
the card being replaced will no longer host data addresses, there will be
no need to reassociate any addresses with the interface index of the
replacement card.  This will allow all of the interface index tracking
logic to be removed from the kernel and IPMP utilities, along with the
removal of the undocumented {\tt SIOCGLIFOINDEX} and {\tt SIOCSLIFOINDEX}
ioctls, and the private {\tt MI\_SETOINDEX} message interface between {\tt
  in.mpathd} and RCM.  In addition, the hefty parsing engine inside RCM
will be able to be replaced with a simple call to the existing {\tt
  ifparse(1M)} command\footnote{The {\tt ifparse} command post-dates the
  original RCM work, which is why it wasn't used originally.} to obtain
the IPMP-related information from {\tt /etc/hostname.{\it if}}.  Finally,
to allow DHCP-managed test addresses to work with DR, RCM will be enhanced
to check for {\tt /etc/dhcp.{\it if}}, and if found, use {\tt
  ifconfig} to start DHCP on the interface.  To mirror boot functionality,
the contents of {\tt /etc/dhcp.{\it if}} will be passed as command-line
  arguments to {\tt ifconfig}.

The RCM code also has special handling for undoing an offline operation
for a card that was never actually removed.  This code is usually
exercised to restore the system's configuration when a remove request
fails part-way through.  In this case, the ``current'' IP interface
configuration is restored from a cache inside RCM, rather than being
rebuilt from the persistent IP interface configuration in the {\tt
  /etc/hostname.{\it if}} file.  RCM builds creates its cache using the
undocumented {\tt ifconfig configinfo} subcommand.  While this is
questionable, this will not be changed since it is orthogonal to the IPMP
Rearchitecture and since customers may depend on the current behavior.

\subsection{{\tt STANDBY} Interaction}
\label{basstandby}
Currently, interfaces can optionally be marked {\tt STANDBY} by an
administrator.  This flag indicates that the interface must not be used
unless some other interface in the group fails.  Thus, when an interface
is marked {\tt STANDBY}, the kernel also sets {\tt INACTIVE} to indicate
that the interface must not be used for outbound traffic, and {\tt
  in.mpathd} moves all data addresses to another interface so that the
interface will not be used for inbound traffic.  If an active interface
subsequently fails, {\tt in.mpathd} will prefer to failover its addresses
to another {\tt INACTIVE STANDBY} interface in the group.  As part of the
failover operation, the IP module clears the {\tt INACTIVE} flag, allowing
the interface to be used.  When the failed interface is eventually
repaired, the IP module restores the {\tt INACTIVE} flag as part of the
failback operation.

Since the concept of address failover and failback will no longer exist,
the behavior of a {\tt STANDBY} interface will change slightly in the new
model.  Specifically, in the new model, {\tt in.mpathd} will be
responsible for updating the {\tt INACTIVE} flag to activate {\tt STANDBY}
interfaces as necessary.  Thus, if there is an {\tt INACTIVE STANDBY}
interface in the group when an {\it active} interface fails or is taken
offline, instead of triggering a failover to the {\tt INACTIVE STANDBY}, {\tt
  in.mpathd} will simply clear its {\tt INACTIVE} flag, allowing the {\tt
  STANDBY} to be used.

An interface repair or undo-online may allow an active {\tt STANDBY} to be
made {\tt INACTIVE} again.  Specifically, if an interface not marked {\tt
  STANDBY} is repaired or brought back online, and that interface causes
there to be more active interfaces than interfaces not marked {\tt
  STANDBY}, {\tt in.mpathd} will mark one of the active {\tt STANDBY}
interfaces {\tt INACTIVE} again.  Similarly, if a {\tt STANDBY} interface
is repaired or brought back online, and there are more active interfaces
than interfaces not marked {\tt STANDBY}, then the repaired interface will be
marked {\tt INACTIVE}.

\subsection{{\tt FAILBACK=no} Interaction}
\label{basfailbackno}
The {\tt FAILBACK /etc/default/mpathd} tunable gives administrators
another way to control IPMP's interface selection.  Specifically, as long
as there are other active interfaces in the group, setting {\tt
  FAILBACK=no} instructs the system to keep a previously-failed interface
unused even after repair.  However, if another interface subsequently
fails, then the unused interface will again be used.  This behavior is
desirable in certain environments where changing the binding of addresses
to incoming interfaces must be kept to a bare minimum -- for instance, in
order to avoid needless invalidation of other hosts' ARP caches.

The support for this functionality in the existing model has been a
challenge (see CRs \bug{5084072} and \bug{4796820} -- and the undocumented
{\tt SIOCSIPMPFAILBACK} ioctl).  In the new model, support for this will
be straightforward: on interface repair, {\tt in.mpathd} will mark it {\tt
  INACTIVE} as long as another interface in the group is also functioning.
Thus, in the common case, the repaired interface will remain unused for
data traffic.  If another active interface in the group subsequently
fails, and there are no interfaces marked {\tt INACTIVE STANDBY}, then
{\tt in.mpathd} will search for interfaces marked just {\tt INACTIVE}, and
clear the {\tt INACTIVE} flag on the first one found.  Since kernel
involvement will no longer be needed, the {\tt SIOCSIPMPFAILBACK} ioctl
will be removed.

Note that {\tt FAILBACK=no} has a subtle interaction with {\tt STANDBY}.
Specifically, if {\tt FAILBACK=no} causes a repaired interface to be
marked {\tt INACTIVE} (as described above), then by definition the number
of active interfaces in the group will remain unchanged.  Thus, the repair
will not cause an active {\tt STANDBY} interface to be marked {\tt INACTIVE}.

\subsection{IPMP Interface Flags}
\label{basipmpflags}
As is clear from the preceding discussion, a wide range of IPMP attributes
are currently managed through interface flags, visible through {\tt
  ifconfig(1M)}:

\begin{table}[h]
\begin{tabular}{|p{0.75in}|p{4.75in}|} \hline
\textbf{flag}    & \textbf{meaning}\\ \thline
{\tt STANDBY}    & Interface is administratively assigned to only be used if
another interface fails \\ \hline
{\tt INACTIVE}   & Interface is functioning, but not currently being used according
to administrative policy \\ \hline
{\tt OFFLINE}    & Interface is an unknown state and unavailable for use \\ \hline
{\tt FAILED}     & Interface is not functioning and must not be used \\ \hline
{\tt RUNNING}    & If not set, interface is not functioning and must not
be used \\ \dhline
{\tt NOFAILOVER} & Address is administratively assigned as a test address and
must not be used \\ \hline
{\tt DEPRECATED} & Address is administratively not preferred for source address
selection \\ \hline
{\tt UP}         & If not set, address is administratively unavailable for
source address selection \\ \hline
\end{tabular}
\end{table}

Although the semantics of all of these flags must be supported for
backward compatibility, administrative and architectural improvements will
greatly reduce the programmatic and administrative reliance on these
flags.  Most notably, {\tt ipmpstat -i} (section \ref{basipmpstati}) will
allow the administrator to easily see how each underlying interface will
be used by IPMP without having to consider the flags, and only the
following subset of the above flags will be visible on an IPMP interface:

\begin{table}[h]
\begin{tabular}{|p{0.75in}|p{4.75in}|} \hline
\textbf{flag}    & \textbf{meaning}\\ \thline
{\tt FAILED}     & Interface is not functioning and must not be used \\ \hline
{\tt RUNNING}    & If not set, interface is not functioning and must not be
used \\ \dhline
{\tt DEPRECATED} & Address is administratively not preferred for source
address selection \\ \hline
{\tt UP}         & If not set, address is administratively unavailable for source
address selection \\ \hline
\end{tabular}
\end{table}

Of these, {\tt FAILED} and {\tt RUNNING} are mutually exclusive on the
IPMP interface, and one will or the other will always be set (strictly
speaking, {\tt FAILED} is not necessary, but will be preserved for
symmetry with the way underlying interfaces are managed).  Further, {\tt
  DEPRECATED} will only be set if explicitly enabled on an address (e.g.,
through {\tt ifconfig ipmp0:1 deprecated}).  Thus, in practice, only the
{\tt RUNNING} and {\tt UP} flags will prove relevant to understanding the
behavior of an IPMP interface and its set of addresses.  Since both of
these flags have existed for decades and are well-understood, they will be
properly handled by administrators and applications alike.

While all of the flags have been preserved, a few backward-compatible
revisions to their semantics have been made for administrative simplicity
or for correctness.  Specifically, {\tt NOFAILOVER} will now always imply
{\tt DEPRECATED} (section \ref{bastestaddr}), {\tt FAILED} will now always
imply {\verb@~RUNNING@} on the IPMP interface (section
\ref{progipmpflags}), and {\tt INACTIVE} will be controlled by {\tt
  in.mpathd} (section \ref{basstandby}).

\subsection{IPMP Interface MTU}
\label{basipmpmtu}

Currently, each interface in an IPMP group maintains its own MTU, as does
each address on each interface.  When an address migrates between
interfaces, the MTU of the address is checked against the MTU of the
destination interface.  If the interface's MTU is smaller, the original
MTU of the address is saved, and the MTU of the address is lowered to
match the interface's MTU.  If the address later migrates again (either as
a result failback, or another failure), the original MTU is consulted, and
the MTU of the address is increased, if possible.

In the new model, the each interface in an IPMP group will still maintain
its own MTU, but that MTU will only directly affect probe packets sent
over the interface.  The MTU of data packets will be controlled by the
{\it IPMP interface's MTU}, which will be set to the smallest MTU supported
by {\it all} of the interfaces currently in the group.  Thus, if an IPMP
group is comprised of two interfaces, one with a 1400 byte MTU, and
another with a 1500 byte MTU, then the IPMP interface will have a 1400
byte MTU.  If the first interface is removed from the group, the MTU will
be automatically raised to 1500 bytes.  As a degenerate case, an IPMP
interface with no underlying interfaces will have the minimum MTU
supported by the IP module (currently 68 bytes for IPv4, and 1280 bytes
for IPv6).

Because applications cannot tie themselves to a specific interface in an
IPMP group, this change will not have any affect on applications.  That
is, because any packet could be sent over any interface in the group, an
application already had to assume that the MTU was the smallest supported
by all interfaces in the group.

\subsection{Failure and Repair Detection}

Currently, {\tt in.mpathd} can use {\it probe-based} failure
detection, {\it link-based} failure detection, or both forms of failure
detection to track the health of an interface.  {\it Probe-based} failure
detection actively probes the send and receive paths of each network
interface using a method similar to {\tt ping(1M)}.  Since performing
probe-based failure detection on an interface requires a test address to
be configured on it, the administrator can deactivate probe-based failure
detection by {\it not} configuring a test address.

In contrast, {\it link-based} failure detection passively monitors the
link status of the interface, as reported by the underlying driver.  This
requires no explicit configuration and provides instantaneous failure
detection, but requires support from each underlying network driver, and
only monitors the health of the link itself, rather than the entire send
and receive path of the network interface.  Unfortunately, it is not
currently possible to administratively tell if an interface supports
link-based failure detection, limiting the adoption of the feature.

Because each form of failure detection has its strengths and weaknesses,
both will remain.  However, the administrator will now be able to use
{\tt ipmpstat} to easily see which failure detection method is enabled
on a given interface, including whether the underlying interface supports
link-based failure detection.  To preserve compatibility, for interfaces
that support link-based failure detection, the {\tt RUNNING} flag will
be set when the link is up, and clear when the link is down.

\subsection{Responsibilities of {\tt in.mpathd} and the IP module}

Currently, the IPMP implementation is split between two
components: the {\tt in.mpathd} daemon, and the kernel's IP module.
This split will remain, but some of the responsibilities will change.

Specifically, {\tt in.mpathd} will remain responsible for failure and
repair detection, including the selection of probe targets, the sending
and receiving of ICMP probes, and the tracking of each interface's link
status.  Additionally, the kernel will remain responsible for outbound
load-spreading and source address selection.  However, {\tt in.mpathd}
will no longer need to perform address migration, simplifying its
implementation.  Since the inbound interface that will receive a packet is
dictated by the IP interface hosting the IP address the packet is sent to,
this change will have a subtle impact on inbound load-spreading.
Specifically, since {\tt in.mpathd} will no longer perform address
migration, it will no longer have any control over how the IP addresses
are distributed across the IP interfaces in the group, and the IP module
will assume complete responsibility for inbound load-spreading.  As part
of transitioning that responsibility, {\tt in.mpathd} will be responsible
for marking interfaces {\tt INACTIVE}, rather than the IP module.

\subsection {Multicast Interaction}
\label{basmulti}
Currently, because there is no concept of an IPMP interface, all multicast
membership information is associated with the underlying interfaces in a
group (or the addresses hosted on those interfaces).  This introduces a
significant amount of implementation complexity, since the original
multicast membership associations must be maintained across interface
failure and repair.  Moreover, nothing prevents a multicast group from
being joined on more than one underlying interface in the same IPMP group,
leading to duplicate traffic.  Further, established technologies such as
IGMP snooping have not been well-supported in an IPMP
environment\footnote{See bug \bug{6220619}.}.

In the future, multicast memberships will be associated with the IPMP
interface\footnote{There is one exception: to support neighbor discovery
  for IPv6 test addresses, the IP module will internally create
  solicited-node multicast memberships for each IPv6 test address on the
  underlying IP interface hosting each address.  Since test addresses
  remain tethered to their underlying IP interface, the implementation
  remains straightforward.}.  In turn, the IP module will nominate one
active interface to handle {\it all} multicast traffic for the IPMP group
(provided an active interface exists)\footnote{This also means that
  multicast traffic will not be load-spread.  If this becomes a problem,
  we could enhance the implementation to load-spread by multicast group --
  at the cost of code complexity.}.  Since the memberships themselves will
remain tethered to the IPMP interface, and since only one interface in the
group will handle all multicast traffic, the handling is simplified
considerably.  Specifically, if the nominated interface fails, then
another active interface will be nominated (if one exists), and
memberships will be transferred to it.  As part of the nomination process,
IGMP or MLD messages will be generated to alert any IGMP or MLD snoopers.
Since the memberships themselves are tied to the IPMP interface, no action
is needed upon interface repair (unless it is the first active interface
in the IPMP group).

Another problem concerns the handling of multicast ICMP echo requests.
Specifically, if the {\tt ip\_respond\_to\_echo\_multicast ndd} tunable
is enabled, then an {\tt all-hosts} multicast ICMP echo request will be
answered using the first {\tt UP} address on the interface, even if it is
a test address.  This is problematic because it allows other hosts to
accidentally discover and potentially use the test address, which is not
highly available.  In the future, {\tt all-hosts} multicast ICMP echo
requests will be answered using an arbitrary data address in the IPMP
group, fixing the problem.

\subsection{Broadcast Interaction}
\label{basbroad}
Currently, in order to avoid duplicate broadcast traffic, an active
interface in the group is nominated to receive all broadcast traffic for
the IPMP group.  If the interface subsequently fails, the nomination
process is repeated.  In the future, the interface nominated to handle
multicast traffic will also be used to receive broadcast traffic.
However, administratively, the two concepts will remain decoupled (see
section \ref{basipmpstati}).

As with multicast, broadcast ICMP echo requests are currently mishandled.
Specifically, if the {\tt ip\_respond\_to\_echo\_broadcast ndd} tunable is
enabled, then a broadcast ICMP echo request will be answered using the
first {\tt UP} address on the interface, even if it is a test address.  In
the future, broadcast ICMP echo requests will be answered using an
arbitrary data address in the IPMP group, fixing the problem.

\subsection{Data-link Interaction}
\label{basdl}
As the name implies, an IPMP group exists at the IP layer.  Below this
layer, an IPMP group is simply a collection of individual data-link
interfaces that have the same ``interface type'' as per the {\tt IFT\_*}
values in {\tt <net/if\_types.h>}.  That is, each data-link interface in
a group maintains its own identity, including its own DLPI state machine,
hardware address and link speed.  Because each data-link interface in an
IPMP group maintains its own identity, it is not semantically feasible to
provide data-link access to the IPMP interface as a collective whole, and
thus no such access will be provided.

While this may seem like a significant limitation, IP-layer support for
packet monitoring (section \ref{baspm}) tools, packet filtering (section
\ref{baspf}), and DHCP (section \ref{basdhcp}), will significantly lessen
its impact.  Moreover, data-link access to an IPMP group as a whole is not
currently provided, so this will not be a regression.  Further, this
rearchitecture will not add any new barriers to implementing such an
abstraction in the future.  Finally, although it not part of this
rearchitecture, restricted data-link access could be added on a
case-by-case basis to address specific limitations, such as packet
monitoring of the ``group'' at the data-link layer.

Of course, data-link access to each underlying physical interface in a group
will still be provided.

\subsection{Naming Considerations}
\label{basnaming}
Currently, each IPMP group has two sets of names: the name of the
group itself (e.g., {\tt a}), and the names of the interfaces in the
group (e.g., {\tt ce0} or {\tt hme0}).  With the IPMP
rearchitecture, a third naming abstraction will be introduced: the IPMP
interface name (e.g., {\tt ipmp0}).

Since each IPMP interface name represents an IPMP group, it would have
been ideal to use the IPMP group name itself as the interface name.
Unfortunately, this is not possible because the naming rules associated
with interface names are more strict than those associated with group
names (most notably, an interface name {\it must} end in a number -- but
there are other differences as well).  While it would be possible to
translate group names into a format acceptable as an interface name, we
decided it was simpler to implement parallel namespaces, but to provide an
administrative model that encouraged administrators to use the same name
in both (e.g., by defaulting to a group name that matched the chosen IPMP
interface name).

While both namespaces will continue to exist, the IPMP interface name will
become the core administrative naming abstraction for IPMP.  This is
necessary in order to allow IPMP to be smoothly integrated with other core
networking abstractions, such as the routing and ARP tables, which operate
exclusively in terms of IP interface names.  By encouraging administrators
to choose the same name for their IPMP interface name and IPMP group name,
we will shield them from this subtlety, and eventually permit group names
to be phased out as a separate namespace.

While the {\tt dladm} enhancements provided by
\htmladdnormallinkfoot{Clearview Vanity Naming}
{See PSARC/2006/499 and http://opensolaris.org/os/project/clearview/uv/}
allow one to rename the data-links associated with the underlying
interfaces in an IPMP group, the IPMP interface does not exist at the
data-link layer and thus cannot be renamed through that facility.  As
such, an IPMP-specific mechanism, described in section
\ref{admipmpcreate}, will be provided through {\tt ifconfig} to enable the
IPMP interface name to be set.  While the asymmetry of this is
unfortunate, it is unavoidable given that IPMP simply does not exist below
the IP layer.

\subsection{Sun Cluster Considerations}
\label{bascluster}
Sun Cluster makes use of IPMP to provide high-availability between nodes
of a cluster.  In addition to supporting the typical IPMP configurations,
Sun Cluster also uses a special configuration called {\it IPMP Singleton}.
In this configuration, only a single network interface is placed into an
IPMP group.  Because there is only one interface in the IPMP group, any
addresses on the interface cannot migrate to another interface, and thus
any test addresses can also serve as IPMP data addresses.  IPMP Singleton
is used by Sun Cluster to monitor interface failures and then trigger a
higher-level failover operation, such as transitioning tasks to another
node in the cluster, without requiring the explicit configuration of test
addresses.

Currently, addresses in an IPMP Singleton group that are not marked {\tt
  NOFAILOVER} will only be used as data addresses, whereas those marked
{\tt NOFAILOVER} will be used as both data and test addresses.  The
default configuration for Sun Cluster is to configure a single address on
each IP interface in a singleton group, and mark it {\tt NOFAILOVER} so
that it can be used for both data and test traffic.

Since the new model requires that all data addresses be on an IPMP
interface and all test addresses be on an underlying interface, and an
address can only be on one interface, allowing a single IP address to be
both a data address and a test address is incompatible with the model.
As such, this will no longer be supported, and an explicit test address
must be configured to perform probe-based failure detection.  However,
given that link-based failure detection is now widespread and sufficient
for many Sun Cluster configurations, and that a dedicated test address can
always be configured if necessary, we expect this limitation will have
minimal impact.

The Sun Cluster software also has internal Sun contracts to use the IP
interface flags {\tt IFF\_FAILED}, {\tt IFF\_INACTIVE}, {\tt IFF\_STANDBY}
and {\tt IFF\_NOFAILOVER}, and the IPMP ioctls {\tt SIOCGLIFGROUPNAME} and
{\tt SIOCSLIFGROUPNAME}.  For the most part, these interfaces
are unaffected by the IPMP Rearchitecture.  However, as per section
\ref{progioctl}, {\tt SIOCSLIFGROUPNAME} will enforce several new
restrictions which will complicate direct use by Sun Cluster.  As such, we
will work with the Sun Cluster team to have the impacted software use {\tt
  ifconfig} to change the group name\footnote{Unfortunately, all IP
  interface configuration logic is hardwired into {\tt ifconfig} rather
  than factored out into a library, precluding a C API.  Properly
  refactoring {\tt ifconfig} is outside the scope of this work.}.

\subsection{IPMP Anonymous Group Support}
\label{basanon}
Along similar lines, IPMP also supports an obscure {\it Anonymous Group}
feature.  This feature is disabled by default, but can be enabled by
setting {\tt TRACK\_INTERFACES\_ONLY\_WITH\_GROUPS} to {\tt no} in {\tt
  /etc/default/mpathd}.  When this feature is enabled, {\tt in.mpathd}
tracks the health of {\it all} network interfaces in the system,
regardless of whether they belong to an IPMP group.  Just like the IPMP
Singleton case, each interface tracked in this manner is alone in a group
and thus no migration is possible, allowing a data address on the
interface to also be used as a test address.  However, unlike IPMP
Singleton, these interfaces are never placed into an IPMP group, and thus
no restriction is needed on whether any of their addresses are marked {\tt
  NOFAILOVER}.  That is, from IPMP's perspective, these interfaces are not
part of {\it any} IPMP group and thus will be visible to applications
regardless of the setting of {\tt NOFAILOVER}.

\subsection{Interaction with Other Network Availability Technologies}

As previously mentioned, Solaris currently supports a range of network
availability technologies.  The proposed rearchitecture will slightly
modify their interaction and integration with IPMP, as summarized in the
following subsections.

\subsubsection{Interaction with 802.3ad and Sun Trunking}

As is currently the case, it will remain possible to layer IPMP on top of
802.3ad aggregations or Sun Trunking trunks.  However, just as IPMP is
unaware of the underlying speed of each underlying interface in an IPMP
group, it is also unaware of the capacities of each aggregation or trunk,
and thus is unable to optimize the load-spreading across them.  This will
remain the case in the new model, though programming interfaces can be
added in the future to allow each underlying link's capacity to be
obtained and monitored, if there is sufficient customer interest.

Since the IPMP interface will not exist at the DLPI layer, it will
not be possible to place an IPMP interface into an aggregation or trunk.

\subsubsection{Interaction with CGTP}
 
Currently, IPMP is mutually exclusive with CGTP: the CGTP virtual
interface cannot be placed into an IPMP group, and none of the physical
interfaces used by CGTP can belong to an IPMP group.  This behavior is
required because CGTP must guarantee that each packet sent using a
particular source address is always sent over a specific interface, which
is directly at odds with the outbound load-spreading feature of IPMP.  For
this reason, this exclusivity will be preserved, and attempts to create
CGTP routes to the IPMP interface will be rejected by the kernel.

\subsubsection{Interaction with SCTP}

As is currently the case, SCTP-based applications will be able to use
addresses that belong to an IPMP group.  Since all SCTP applications are
IP-based, they will transparently benefit from the proposed changes.

\subsubsection{Interaction with OSPF-MP}
\label{basospf}

Currently, the system does not allow OSPF-MP to be used with IPMP.
Specifically, because {\tt vni(7D)} is virtual and does not represent a
single link, it makes no sense to place it into an IPMP group.  Further,
while it is semantically reasonable to allow {\tt usesrc} to be applied
consistently to all interfaces in an IPMP group (see below), there is
currently no administrative way to enforce this usage.  That is, with the
current implementation of IPMP, it is not possible to ensure that the
value of {\tt usesrc} is identical for each interface in an IPMP group --
and indeed, it is quite likely {\tt usesrc} will be misused.

The introduction of the IPMP group interface will fix the usability
problems with {\tt usesrc}.  Specifically, by allowing {\tt usesrc} to be set
on an IPMP interface, we could ensure that each IPMP group had only one
{\tt usesrc} value, and that the value applied to all interfaces in the
group.  When set, this would override the group's source address selection
algorithm.  Thus, the addresses hosted on each IPMP group interface would
be used to route packets to the addresses hosted on the appropriate {\tt
  vni} interface. Further, the source address would always be an address
on the selected {\tt vni} interface.

Consider a configuration with two IPMP groups together in an OSPF-MP
group, where each IPMP group had two underlying interfaces.  This would
provide twice the network utilization versus placing all four interfaces
into an OSPF-MP group\footnote{This is because ECMP is not currently
available on Solaris.} (since both interfaces in the selected IPMP
group will be able to send and receive traffic) and require two fewer
subnets (since each pair of interfaces in an IPMP group would be on the
same subnet).  Further, this could provide end-to-end availability that
would not be available by simply placing all four interfaces into an IPMP
group.

While this is an intriguing opportunity afforded by the new architecture,
we would like to gauge customer need before implementing such a feature.
Moreover, once ECMP support is available, OSPF-MP will likely provide a
much simpler and more compelling story for end-to-end availability.  As
such, we will not yet allow {\tt usesrc} to be applied to an IPMP group.
However, this restriction may be lifted in the future.

\subsection{SMF Considerations}

Currently, IPMP is not tied into the SMF framework (though Sun Cluster
provides a restarter for {\tt in.mpathd} as an interim solution).
While integrating IPMP into the SMF is desirable, it is not necessary to
satisfy the core requirements specified in section \ref{req}.  Further,
there are other complicating factors:

\begin{itemize}
\item IPMP configuration is not cleanly separable from other IP interface
  configuration currently done via {\tt /etc/hostname.{\it if}}.  Thus,
  migration of IPMP configuration should be done as part of a broader
  effort to migrate {\tt /etc/hostname.{\it if}} information to the SMF.

\item The {\tt in.mpathd} daemon itself is an awkward fit for the SMF
  model.  In particular, it is usually started on-demand by {\tt ifconfig}
  when an IP interface is placed in a group (and never self-terminates).
  However, it needs to be started explicitly (typically by the boot
  scripts) if IPMP anonymous groups are being used and no named IPMP
  groups are configured.

\item Some third party applications, such as Symantec VCS, are known to
 explicitly start and stop {\tt in.mpathd}.  Since that will no longer
 work as expected once {\tt in.mpathd} is under SMF control, we will need
 to work with them to understand their needs and provide an alternative
 (likely simply a restart of the SMF service).

\end{itemize}

Therefore, this proposal will not integrate IPMP into the SMF.  However,
the introduction of the IPMP group interface will simplify the future
representation of IPMP into SMF, since a special construct will not be
required to represent IPMP groups.  For instance, many of the existing
tunables in {\tt /etc/defaut/mpathd} should be per-IPMP-group rather than
global; tying these to IP interface properties which could be set on each
IPMP group interface would provide more flexibility and improve
administrative consistency. Finally, this proposal is careful not to
introduce any new administrative commands or tunables which might
complicate or conflict with future integration of IPMP into the SMF.

\pagebreak

\section{IPMP Rearchitecture: Administrative Model}

\subsection{Configuration with {\tt ifconfig}}

\subsubsection{Adding and Removing Data Addresses}

Since the IPMP interface will become the core administrative abstraction
for IPMP, we will recommend that administrators directly configure
addresses on the IPMP interface.  For instance, to add an additional
address to {\tt ipmp0}, one will do:

\begin{alltt}
  # ifconfig ipmp0 addif 129.146.17.59 up
\end{alltt}

To maintain compatibility with previous releases of Solaris, data
addresses will also be able to be added to the underlying physical
interface:

\begin{alltt}
  # ifconfig ce0 addif 129.146.17.59 up
\end{alltt}

However, once the address is brought up, it will actually appear in the
next available logical interface  on the IPMP interface associated
with {\tt ce0} -- e.g., {\tt ipmp0:2}.  Note that since the address
will not usable prior to bringing it up, there will be no need to implement
complex address migration to support this operation -- the address will
simply be changed to be associated with the IPMP interface and brought up.

Once a data address has been associated with an IPMP interface, its
interface name binding (e.g., {\tt ipmp0:2}) will never change except
by explicit administrative action.  To remove a given data address from a
group, the administrator will be able to specify either the logical
interface or the address.  That is, the following will be equivalent:

\begin{alltt}
  # ifconfig ipmp0 removeif 129.146.17.59
  # ifconfig ipmp0:2 unplumb
\end{alltt}

Alternatively, marking an address {\tt down} will temporarily remove it
from system use:

\begin{alltt}
  # ifconfig ipmp0:2 down
\end{alltt}

\subsubsection{Configuring Data Addresses with DHCP}
\label{admifconfigdhcp}
As with physical network interfaces, it will be possible to use DHCP to obtain
addresses on the IPMP interface:

\begin{alltt}
  # ifconfig ipmp0:1 plumb
  # ifconfig ipmp0:1 dhcp start primary
\end{alltt}

A summary of the changes to DHCP is provided in section \ref{basdhcp}.
\subsubsection{Adding and Removing Test Addresses}
\label{bastestaddr}
Since test addresses remain associated with each underlying interface,
they will still be configured on the physical interface:

\begin{alltt}
  # ifconfig ce0 129.146.17.56 -failover deprecated up
\end{alltt}

However, since no test addresses should ever be used as a source address,
setting {\tt -failover} will now imply {\tt deprecated}.
Thus, this will also configure a test address:

\begin{alltt}
  # ifconfig ce0 129.146.17.56 -failover up
\end{alltt}

Test addresses will be prohibited on IPMP interfaces, and attempting to
set {\tt -failover} on an address hosted on an IPMP interface will fail.

As is currently the case, test addresses will be able to be removed using
the {\tt unplumb} or {\tt removeif ifconfig} subcommands.  One will also
be able to prevent the use of a test address by marking it {\tt down}.  It
will also be possible to convert the test address to a data address via
the {\tt failover} subcommand, though this will rarely be useful in
practice\footnote{As per section \ref{progipmpflags}, changing
  {\tt IFF\_NOFAILOVER} on an {\tt IFF\_UP} address will not be permitted.
  Accordingly, {\tt ifconfig} will bring the address {\tt down} first, and
  back {\tt up} afterwards; the {\tt up} will trigger migration.}.  Of
course, if {\tt failover} is issued on an {\tt UP} address, the address
will migrate from the physical interface to the appropriate IPMP
interface.

\subsubsection{Adding and Removing Underlying Interfaces}
\label{basgroupif}
As is currently the case, underlying interfaces will be able to be added
or removed from a group using the {\tt group} subcommand.  For instance,
to replace {\tt ce1} with {\tt ce2} in group {\tt a}:

\begin{alltt}
  # ifconfig ce2 group a
  # ifconfig ce1 group ""
\end{alltt}

While we currently document a number of requirements that need to be met
in order to place an interface into a group, none of these are currently
enforced by the system when an interface is placed into a group.  In the
future, several of these requirements will be enforced by the kernel,
resulting in diagnostic messages being printed by {\tt ifconfig}.
Specifically:

\begin{itemize}
\item The interface type must be the same as any other interfaces in
  the group (e.g., Ethernet).

\item The interface type must be broadcast-capable (e.g., no
  point-to-point interfaces)\footnote{In principle, point-to-point could
    be made to work, but only if the peer was also IPMP-aware.  Moreover,
    Multilink PPP already fills this need.}.

\item The interface must not have any data addresses that are controlled by
  other applications (currently {\tt dhcpagent(1M)} and {\tt in.ndpd(1M)}),
  as migration of those addresses may confuse those applications.  (Since
  there is no convenient way to control {\tt in.ndpd(1M)}, {\tt ifconfig}
  will bring down the link-local IPv6 address which {\tt in.ndpd(1M)} will
  use as a hint to tear down its addresses on that IPv6 interface.
  However, if {\tt in.ndpd(1M)} does not tear down its addresses within a
  short period of a time, a diagnostic message will still be printed.)
\end{itemize}

Note that as before, every active interface in a group must have a unique
hardware address.  However, since a hardware address may become a
duplicate {\it after} an interface has joined a group, this will not
become a requirement to join a group.  See section \ref{bashwaddr} for
details.

To meet the requirement that {\tt UP} addresses never migrate, the kernel
will fail requests to place interfaces with {\tt UP} addresses into a
group.  However, {\tt ifconfig} will automatically bring any {\tt UP}
addresses {\tt down} prior to placing an interface into a group, and bring
them back {\tt up} afterwards, triggering migration as
appropriate\footnote{Of course, test addresses will not migrate as part of
  being brought {\tt up}.  Nonetheless, the kernel will requires that even
  test addresses be brought down to ensure that an interface is entirely
  quiesced prior to joining a group.}.  As such, administrative
compatibility will be preserved.

\subsubsection{Creating IPMP Interfaces}
\label{admipmpcreate}
IPMP interfaces will be able to be created using either {\it implicit
  creation} or {\it explicit creation}.  Explicit creation will provide
optimal control over the configuration of the IPMP interface, at the
expense of requiring the administrator to learn a new administrative
procedure.  In contrast, implicit IPMP interface creation will require no
new administrative knowledge, but will limit the administrative control
over the name of the IPMP interface and interfaces it will be associated
with.  While our documentation (e.g., on {\tt docs.sun.com}) will describe
both, we will encourage administrators to use explicit creation both
because it provides optimal control and because it makes the semantic
split between the IPMP and underlying IP interfaces apparent.

{\bf Implicit IPMP Interface Creation}

Implicit creation will occur by placing a physical interface into an IPMP
group that does not currently exist.  In this case, the system will create
the lowest available {\tt ipmp{\it N}} interface and migrate any {\tt UP}
addresses from the physical interface to the IPMP interface.  For
instance, if {\tt ipmp0} already exists as group {\tt a}, then
{\tt ifconfig ce0 group b} will cause the system to create
{\tt ipmp1}.

Unfortunately, because data addresses are automatically migrated when
brought {\tt UP}, the following misconfiguration will now always lead to
unexpected results:

\begin{alltt}
 # ifconfig ce0 group b 10.0.0.1 up
 # ifconfig ce0 -failover
\end{alltt}

Specifically, since {\tt 10.0.0.1} will not be on {\tt ce0} by the time
the {\tt -failover} subcommand is issued, {\tt -failover} will not be
applied to it.  Note that the above sequence of commands has always been
incorrect, but has usually worked as long as {\tt ce0} has not failed.  To
reduce the risk of this being a call generator, a warning will be placed
in both the IPMP documentation and the release notes.

{\bf Explicit IPMP Interface Creation with {\tt ipmp}}

Explicit creation will enable the administrator to choose the name of the
interface, and optionally associate a specific group name with it.  IPMP
interfaces will be explicitly created using the new {\tt ipmp} subcommand.
This subcommand is analogous to {\tt plumb}, but will instruct the system to
create an IPMP group interface rather than a traditional IP interface.
For instance, to explicitly create an IPMP interface named {\tt ipmp0}:

\begin{alltt}
  # ifconfig ipmp0 ipmp
\end{alltt}

As with {\tt plumb}, by default {\tt ipmp} will create an IPv4 instance,
but an IPv6 instance will be able to be created instead by using {\tt
  inet6 ipmp}.  However, in practice, this will rarely be needed since
{\tt IPv6} will be instantiated on demand if IPv6 addresses are added to a
given IPMP group interface.

Attempting to use the {\tt ipmp} subcommand to create an IP interface that
already exists will fail.  Similarly, attempting to use the {\tt ipmp}
subcommand to create an IPMP group that already exists will fail.

By default, the group name of the interface will match its interface name.
Thus, above, {\tt ipmp0} will have a group name of {\tt ipmp0}.  We hope
that this behavior, coupled with recommendations in our documentation,
will convince administrators to use identical IPMP group and
interface names.

Because the {\tt ipmp} subcommand will allow {\tt ifconfig} to differentiate
the creation of an IPMP group from the plumbing of a physical interface,
the administrator will be able to associate a semantically meaningful name
with an IPMP group interface:

\begin{alltt}
  # ifconfig outside0 ipmp
\end{alltt}

As justified in section \ref{basnaming}, we will still support the ability
to associate an arbitrary group name with an IPMP interface.
Unsurprisingly, this will be done with the {\tt group} subcommand.  Thus,
to move {\tt outside0} from group {\tt outside0} to group {\tt b}:

\begin{alltt}
  # ifconfig outside0 group b
\end{alltt}

Of course, the two former commands will be able to be merged together:

\begin{alltt}
  # ifconfig outside0 ipmp group b
\end{alltt}

See section \ref{admgroupname} for restrictions associated with changing
the group name.

\subsubsection{Destroying IPMP Interfaces}

Like any other interface, the {\tt unplumb} subcommand will cause an IPMP
group interface to be destroyed.  However, destruction will only be
possible if there are no physical interfaces currently part of the
relevant IPMP group.  For instance:

\begin{alltt}
  # ifconfig ipmp1 ipmp group b
  # ifconfig ce0 group b
  # ifconfig ipmp1 unplumb
  ifconfig: ipmp1: cannot unplumb: IPMP group is not empty
  # ifconfig ce0 group ""
  # ifconfig ipmp1 unplumb
\end{alltt}

As with a traditional physical interface, any IP addresses associated with
the IPMP interface will be removed from the system as part of the
{\tt unplumb} operation.

\subsubsection{IPMP Interface Extent}
As discussed in section \ref{admipmpcreate}, the initial placement of a
physical interface into an IPMP group will create an IPMP interface for the
given group (if it does not yet exist), and move any {\tt UP} data
addresses to the IPMP interface.  To avoid having to preserve the
complicated address migration code, the move will instead be modeled as a
set of {\tt SIOCLIFREMOVEIF} operations (to remove the addresses from the
physical interface), followed by a set of {\tt SIOCLIFADDIF} operations (to
add the addresses to the IPMP interface).

At this point, the data addresses will be ``owned'' by the IPMP interface
-- {\bf removing the interface from the IPMP group will not cause the
  addresses to ``migrate back''}\footnote{Unfortunately, this also means
  that if an interface is placed into the wrong group, any migrated
  addresses must be manually ``migrated back''  via {\tt
    removeif} and {\tt addif}.}.  Similarly, placing the interface into a
different IPMP group will not trigger address migration because the
physical interface will have no data addresses prior to the group change
operation.

Even if all physical interfaces are removed from an IPMP group, the IPMP
interface associated with the group will continue to exist (this will be
necessary to support explicit IPMP group creation).  Thus, the IPMP group
can be explicitly destroyed using {\tt unplumb} if it is no longer
desired.

\subsubsection{Configuring IPMP Group Names}
\label{admgroupname}

As previously mentioned, the administrator will be able to use the {\tt
  group} subcommand to set the group name of an IPMP interface.  Any
interfaces in the group will be changed to have the new group name.  That
is:

\begin{alltt}
  # ifconfig ipmp0 ipmp
  # ifconfig ce0 group ipmp0
  # ifconfig ipmp0 group b
  # ifconfig ce0
  ce0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> ...
       inet 129.146.17.56 netmask ffffff00 broadcast 129.146.17.255
       {\bf{groupname b}}
\end{alltt}

Attempts to use an existing group name or to clear the group name will
fail:

\begin{alltt}
  # ifconfig ipmp0 ipmp
  # ifconfig ipmp1 ipmp
  # ifconfig ipmp1 group ipmp0
  ifconfig: ipmp1: SIOCSLIFGROUPNAME: already exists
  # ifconfig ipmp1 group ""
  ifconfig: ipmp1: SIOCSLIFGROUPNAME: invalid argument
\end{alltt}

\subsubsection{Configuring {\tt STANDBY} Interfaces}

Since {\tt STANDBY} is inherently tied to a given physical interface,
this attribute will still be configured on it using:

\begin{alltt}
  # ifconfig ce0 standby
\end{alltt}

Since it makes no semantic sense, attempting to set {\tt standby} on an
IPMP interface will fail.

\subsubsection{Configuring Hardware Addresses}
\label{bashwaddr}

IPMP requires that only one interface in a group may use a given hardware
address.  This is usually the case, unless the {\tt eeprom} variable {\tt
  local-mac-address?}  is set to {\tt false}\footnote{See CRs
  \bug{4868371} and \bug{6504375} for more on this unfortunate variable.},
or the underlying interface does not support a unique hardware address.
In the former case, {\tt local-mac-address?} can simply be set to {\tt
  true}.  In the latter case, the {\tt ether} subcommand must be used to
configure the interface with a unique hardware address.

This will remain unchanged in the new model.  However, in the new model,
{\tt in.mpathd} will actively enforce that only one interface in a group
may use a given hardware address.  Specifically, if an interface with a
duplicate hardware address is added to a group, or an existing interface
in a group is changed to have a duplicate hardware address, {\tt
  in.mpathd} will log a diagnostic message and take the interface
offline\footnote{As covered in section \ref{basipmpstati}, {\tt ipmpstat}
  will also report this misconfiguration.}.  If subsequent changes to the
group configuration cause the interface's hardware address to become
unique again, {\tt in.mpathd} will log another diagnostic message and
bring the interface back online.  Finally, {\tt in.mpathd} will fail
attempts to online an interface with a duplicate hardware address
(regardless of whether the interface was explicitly offlined by {\tt
  in.mpathd}, or merely became a duplicate while it was administratively
offline).

Since the IPMP interface has no hardware address, attempting to
use {\tt ether} on it will fail.

\subsubsection{Miscellaneous {\tt ifconfig} Operations}

Since the IPMP interface will not exist at the DLPI layer, the {\tt
  auto-revarp}\footnote{While we could change {\tt ifconfig} to perform
  RARP on an active underlying interface in the group, DHCP has mostly
  obsoleted RARP, making such a wart hard to justify.}, {\tt modlist}, {\tt
  modinsert} and {\tt modremove ifconfig} subcommands will fail with an
appropriate error.  Additionally, each IPMP interface has its own
interface metric (see section \ref{progifmetric}), which will be able to
be changed with the {\tt metric} subcommand.
\subsection{Observability with {\tt ipmpstat}}
\label{admipmpstat}
The IPMP rearchitecture will include a new command in {\tt /usr/sbin} called
{\tt ipmpstat}.  This tool will concisely display information about the
IPMP subsystem that is currently only available by sifting through the
output of several existing commands, or simply not available at all.

By sifting through open bug reports and posts to IPMP mailing lists over
the past several years, and drawing on our own internal experience with
IPMP, we generated a list of \htmladdnormallinkfoot{common questions
  regarding the IPMP
  subsystem}{http://opensolaris.org/os/project/clearview/ipmp/ipmp-obs-questions.txt}.
While the amount of IPMP-related information is surprisingly large, it can
be broken down into five subcategories:

\begin{itemize} \addtolength{\itemsep}{-0.5\baselineskip}
\item IPMP group information
\item IPMP address information
\item IPMP underlying interface information
\item IPMP target information
\item IPMP probe information
\end{itemize}

Accordingly, we propose five different output formats for the {\tt ipmpstat}
command, each selectable through a command-line flag.  The following
subsections explain each output format through a collection of sample output.
Please note that:
\begin{itemize}

\item By default, hostnames rather than numeric IP addresses will be output
  (provided a hostname exists).  As with {\tt netstat} and other existing
  commands, an {\tt -n} option will cause numeric IP addresses to be
  output instead.  The following examples use {\tt -n} where necessary
  to clarify the relationship between the proposed {\tt ipmpstat} output
  and the {\tt ifconfig} output shown earlier.
  
\item As implied by the examples, in general, no special privileges will
  be needed to invoke {\tt ipmpstat}.  One exception is {\tt ipmpstat -p},
  since limitations\footnote{See CR \bug{4791900}.} in the sysevent
  framework prevent unprivileged processes from receiving sysevents.

\item The primary purpose of {\tt ipmpstat} will be to interactively
  debug problems.  As such, the standard output format will be {\tt
    unstable}.  However, for each output format, an {\tt -o} option
  will allow the set of displayed fields to be controlled, and
  a {\tt -P} option will produce a machine-parsable format; see
  sections \ref{basipmpstato} and \ref{basipmpstatP} for details.

\item In order to properly showcase each of the formats, the sample
  configuration shown in the earlier examples has been expanded
  significantly.  Subsequent sections of this document will revert to the
  simpler configuration first shown in section \ref{basipmpif}.

\end{itemize}

\subsubsection{Observing IPMP groups with {\tt ipmpstat -g}}

The state of all IPMP groups on the system will be viewable using the
{\tt -g} flag to {\tt ipmpstat}:

\begin{alltt}
  \$ ipmpstat -g
  GROUP      GROUPNAME   STATE      FDT       INTERFACES
  ipmp0      outside     ok         10.00s    ce0 ce1
  ipmp1      inside      failed     --        [hme0 hme1]
  ipmp2      service     degraded   20.00s    qfe0 qfe3 (qfe2) [qfe1]
\end{alltt}

As can be seen above, for each IPMP group, {\tt ipmpstat} will show its
name, associated group interface, group state, failure detection time
({\tt --} if probe-based failure detection is disabled), and underlying
interfaces.  If IPMP anonymous group support is enabled, the anonymous
group will be shown with {\tt --} for its {\tt GROUP} and {\tt GROUPNAME}
fields.

The group state will be one of:
\begin{itemize}
 \addtolength{\itemsep}{-0.5\baselineskip}
\item {\tt ok}: All interfaces in the group are usable.
\item {\tt degraded}: Some -- but not all -- interfaces in the group are
 usable. 
\item {\tt failed}: No interfaces in the group are usable.
\end{itemize}

The terms {\tt degraded}\footnote{The concept of a ``degraded'' group is
new with the IPMP rearchitecture.}  and {\tt ok} were selected to
ease future integration between IPMP and the Solaris
\htmladdnormallinkfoot{Fault Management Architecture
(FMA)}{http://opensolaris.org/os/community/fm/}, which provides
compatible definitions for these terms.  Although FMA prefers {\tt
faulted} to indicate failure, we propose {\tt failed} in order to remain
consistent with existing IPMP terminology and exposed interfaces
such as the {\tt IFF\_FAILED} flag.

Finally, closer examination of the interface list will
allow experienced administrators to quickly tell which interfaces are
active, {\tt INACTIVE}, and unusable.  Specifically:

\begin{itemize}
\item Active interfaces will be listed first and not enclosed in any
  brackets or parenthesis.  Active interfaces are those currently being
  used by the system to send or receive data traffic.

\item {\tt INACTIVE} interfaces will be listed next and enclosed in
  parenthesis.  {\tt INACTIVE} interfaces are those that are functioning,
  but not currently being used according to administrative policy.

\item Unusable interfaces will be listed last and enclosed in brackets.
  Unusable interfaces are those unable to be used at all in their current
  configuration (e.g., {\tt FAILED} or {\tt OFFLINE}).

\end{itemize}

\subsubsection{Observing IPMP data addresses with {\tt ipmpstat -a}}

The state of all IPMP data addresses on the system will be viewable using
the {\tt -a} flag to {\tt ipmpstat}:

\begin{alltt}
  \$ ipmpstat -an
  ADDRESS                  STATE   GROUP      INBOUND    OUTBOUND
  129.146.17.55            up      ipmp0      ce0        ce0 ce1
  129.146.17.57            up      ipmp0      ce1        ce0 ce1
  192.0.0.100              up      ipmp1      --         --
  192.0.0.101              up      ipmp1      --         --
  128.0.0.100              up      ipmp2      qfe0       qfe0 qfe3
  128.0.0.101              up      ipmp2      qfe3       qfe0 qfe3
  128.0.0.102              down    ipmp2      --         --
\end{alltt}

For each data address, the administrator will be able to quickly see what
group it belongs to, whether it is available for use (as toggled with
{\tt ifconfig up}/{\tt down}), and what inbound and outbound
interfaces it can be used on.  Note that this will allow administrators to
finally see that a given data address is not tied to a specific outbound
interface, but rather may be used on {\it any} active interface in the
group.  Likewise, this output will make it explicit that each data address
can be associated with at most one inbound interface at a time, and allow
administrators to observe the current inbound interface binding chosen by
the system.

\subsubsection{Observing IPMP interfaces with {\tt ipmpstat -i}}
\label{basipmpstati}
The state of all interfaces using IPMP on the system will be viewable
using the {\tt -i} flag to {\tt ipmpstat}:

\begin{alltt}
  \$ ipmpstat -i
  INTERFACE   ACTIVE  GROUP       FLAGS     LINK      PROBE     STATE
  ce0         yes     ipmp0       --mb---   up        ok        ok
  ce1         yes     ipmp0       -------   up        disabled  ok
  hme0        no      ipmp1       -----d-   up        disabled  offline
  hme1        no      ipmp1       is-----   down      unknown   failed
  qfe0        yes     ipmp2       --mb---   unknown   ok        ok
  qfe1        no      ipmp2       i------   up        ok        ok
  qfe2        no      ipmp2       -------   up        failed    failed
  qfe3        yes     ipmp2       --mb---   up        ok        ok
\end{alltt}
For each interface, the administrator will be able to quickly tell whether
an interface is active and what group it belongs to ({\tt --} for
interfaces in the anonymous group).  The remaining information will allow
the administrator to ascertain the reason the interface is or is not
active:
\begin{itemize}

\item The {\tt link} field will provide the current link status, and will
  be one of {\tt up}, {\tt down}, or {\tt unknown}. The {\tt unknown}
  state will indicate that the driver does not support link up/down
  notification.

\item The {\tt probe} field will provide the current probe status, and
  will be one of {\tt ok}, {\tt failed}, {\tt unknown}, or {\tt disabled}.
  The {\tt unknown} state will indicate that no probe targets could be
  found.  The {\tt disabled} state will indicate that no IPMP test address
  is configured on the interface, disabling probe-based failure detection.

\item The {\tt state} field will provide the overall state of the
  interface, and will be one of {\tt ok}, {\tt failed}, {\tt offline},
  or {\tt unknown}.  If the interface has been offlined, then its state
  will be {\tt offline}.  For online interfaces, if the link status is
  {\tt down} or the probe status is {\tt failed}, then the interface state
  will be {\tt failed}.  Otherwise, if the probe status is {\tt unknown}
  (indicating that no probe targets could be found), the overall state
  will be {\tt unknown}.  If none of the above apply, it will be {\tt ok}.

\item The {\tt flags} field will concisely display whether the
  interface is:
\begin{itemize}
\item {\tt{i}}: {\tt INACTIVE}
\item {\tt{s}}: {\tt STANDBY}
\item {\tt{m}}: Nominated to send and receive IPv4 multicast traffic (see
  sections \ref{basmulti} and \ref{progmulti}).
\item {\tt{b}}: Nominated to receive broadcast traffic (see section
  \ref{basbroad}).
\item {\tt{M}}: Nominated to send and receive IPv6 multicast traffic (see
  sections \ref{basmulti} and \ref{progmulti}).
\item {\tt{d}}: Unusable due to being marked {\tt down}.
\item {\tt{h}}: Unusable due to having a duplicate hardware
  address (see section \ref{bashwaddr}).
\end{itemize}
Additional flags may be defined in the future, if needed.
\end{itemize}
\subsubsection{Observing IPMP probe targets with {\tt ipmpstat -t}}

The status of all probe targets associated with each interface using IPMP
will be viewable using the {\tt -t} flag to {\tt ipmpstat}:

\begin{alltt}
  \$ ipmpstat -tn
  INTERFACE  MODE      TESTADDR         TARGETS
  ce0        routes    129.146.17.56    129.146.17.1 129.146.17.3
  ce1        disabled  --               --
  hme0       disabled  --               --          
  hme1       routes    192.1.2.200      192.1.2.1
  qfe0       multicast 128.0.0.200      128.0.0.1 128.0.0.2
  qfe1       multicast 128.0.0.201      128.0.0.2 128.0.0.1
  qfe2       multicast 128.0.0.202      128.0.0.1 128.0.0.2
  qfe3       multicast 128.0.0.203      128.0.0.1 128.0.0.2
\end{alltt}

For each interface, the administrator will be able to see the method used
to obtain the probe targets, the test address used to send and receive
probes, and the current list of probe targets.  All IP interfaces
configured to use IPMP will be displayed, but those that have probe-based
failure detection disabled will show a target mode of {\tt disabled}, and
{\tt --} will be shown for the remaining fields.  Although unusual, the
administrator may choose to configure an IP interface with both IPv4 and
IPv6 test addresses, in which case two lines of output will be displayed
(for the IPv4 and IPv6 probe target configurations, respectively).

If probe-based failure detection is enabled, the target mode will be set
to either {\tt routes} or {\tt multicast}, depending on whether the system
routing table or a multicast probe was used to obtain the target list.  In
addition, the test address associated with the interface will be
displayed, along with a list of current probe targets, in firing order
(i.e., the first one listed will be the next target to be probed).  Note
that if multicast is used, all interfaces in a group will not necessarily
have the same ordering or set of probe targets.

\subsubsection{Observing IPMP probes with {\tt ipmpstat -p}}
\label{admipmpstatp}
All of the IPMP probes being sent on the system will be viewable using the
{\tt -p} flag to {\tt ipmpstat}:

\begin{alltt}
  \# ipmpstat -pn
  TIME      INTERFACE   PROBE   TARGET              NETRTT    RTT       RTTAVG
  0.11s     ce0         589     10.8.57.248         0.51ms    0.76ms    0.76ms 
  0.17s     hme1        612     192.1.2.1           --        --        --     
  0.25s     qfe0        602     128.0.0.1           0.61ms    1.10ms    1.10ms 
  0.26s     qfe1        602     128.0.0.2           --        --        --     
  0.25s     qfe2        601     128.0.0.1           0.62ms    1.20ms    1.00ms 
  0.26s     qfe3        603     128.0.0.1           0.79ms    1.11ms    1.10ms 
  1.64s     ce0         590     10.8.57.248         0.49ms    0.50ms    0.73ms 
  1.66s     hme1        613     192.1.2.1           --        --        --     
  1.70s     qfe0        603     128.0.0.1           0.63ms    1.10ms    1.10ms 
  ^C
  \#
\end{alltt}

Unlike other {\tt ipmpstat} command-line flags, {\tt -p} will run until
explicitly terminated by the administrator.  While running, {\tt ipmpstat}
will retrieve information from the IPMP subsystem regarding acknowledged
or lost probes, and output any new probe activity.

For each probe, {\tt ipmpstat -p} will be able to report the following
information:

\begin{itemize}
\item An identifier representing the probe. The identifier will start at
  zero and will monotonically increment for each probe sent by {\tt
    in.mpathd} over a given IP interface.  To facilitate more detailed
  analysis by packet monitoring tools, this identifier will match the {\tt
    icmp\_seq} field of the ICMP probe itself.

\item The interface the probe was sent on.  Administrators interested in
  probes only from specific interfaces can post-process the output based
  on this field.

\item The time the probe was sent, relative to when {\tt ipmpstat} was
  invoked. If the displayed probe was sent prior to starting
  {\tt ipmpstat}, the time will be shown as negative, relative to when
  {\tt ipmpstat} was invoked.

\item The target the probe was sent to.  Administrators interested in
  probes only to specific targets can post-process the output based on
  this field.

\item The network round-trip-time for the probe, in milliseconds.  If {\tt
    in.mpathd} has concluded that the probe has been lost, {\tt --} will
  be displayed instead. This is the time between when the IP module sends
  the probe and when the IP module receives the ack.

\item The total round-trip-time for the probe, in milliseconds.If {\tt
    in.mpathd} has concluded that the probe has been lost, {\tt --} will
  be displayed instead.  This is the time between when {\tt in.mpathd}
  starts executing the code to send the probe, and when {\tt in.mpathd}
  completes processing the ack.  Spikes in the total round-trip time that
  are not present in the network round-trip time indicate that the system
  itself is overloaded.

\item The current average and standard deviation for the round-trip-time
  to the specified target on the specified interface.  This will allow
  easy identification of slow or jittery targets.  As implied by the above
  example, the standard deviation ({\tt RTTDEV}) will not be reported by
  default because of insufficient screen width.
\end{itemize}
\subsubsection{Controlling Output with {\tt ipmpstat -o}}
\label{basipmpstato}

Any of the aforementioned output formats will be able to be combined with
{\tt -o} to produce a customized and deterministic output format.  The
{\tt -o} option will take a comma-separated list of field names to
display.  For example:

\begin{alltt}
  \$ ipmpstat -g -o group,fdt,interfaces
  GROUP      FDT       INTERFACES
  ipmp0      10.00s    ce0 ce1
  ipmp1      --        [hme0 hme1]
  ipmp2      20.00s    qfe0 qfe3 (qfe2) [qfe1]
\end{alltt}

The special field name {\tt all} will cause all fields to be output,
rather than the default behavior of displaying the fields that will fit in
80 columns.  This distinction is only important for output formats that
have more fields than will fit in 80 columns (currently only {\tt ipmpstat
  -p}).  However, since new fields may be added that cause an output
format to overflow 80 columns, {\tt all} will ensure that all fields are
always output even as {\tt ipmpstat} is enhanced.

\subsubsection{Machine-Parsable Output with {\tt ipmpstat -P} }
\label{basipmpstatP}

Any of the aforementioned output formats will be able to be combined with
{\tt -P} to produce a machine-parsable format.  The format will be
based on {\tt dladm}'s parsable format, and thus will differ from
the normal {\tt ipmpstat} output format in the following ways:
\begin{itemize}
\item The header will be omitted.
\item Fields will be separated by {\tt :} rather than whitespace.
\item If multiple fields are requested, and a literal {\tt :} or {\tt
    $\backslash$} occur in a field's value, they will be escaped by
  prefixing them with {\tt $\backslash$}.
\item Fields with empty values will yield no output, rather than {\tt --}.
\end{itemize}

Thus, adding {\tt -P} to the previous example:
\begin{alltt}
  \$ ipmpstat -g -P -o group,fdt,interfaces
  ipmp0:10.00s:ce0 ce1
  ipmp1::[hme0 hme1]
  ipmp2:20.00s:qfe0 qfe3 (qfe2) [qfe1]
\end{alltt}

As with {\tt dladm}, this format is intended to be easy to use from shell
scripts.  For instance, to get an FDT for a specific group:

\begin{alltt}
  getfdt() \{
        ipmpstat -gP -o group,fdt | while IFS=: read group fdt; do
                if [[ "$group" = "$1" ]]; then
                        echo "$fdt"
                        return
                fi
        done
  \}
\end{alltt}	

To ensure that the output remains predictable as the {\tt ipmpstat}
feature set evolves, the command will fail if an output field list is not
specified, or if {\tt -o all} is specified:

\begin{alltt}
  \$ ipmpstat -gP
  ipmpstat: output field list (-o) required in parsable output mode
  \$ ipmpstat -gP -o all
  ipmpstat: "all" not allowed in parsable output mode
\end{alltt}

\subsection{System Boot}

\subsubsection{IPMP Configuration at Boot}
\label{admboot}
Administratively, the persistent IPMP configuration of each IP interface
is stored in a corresponding {\tt /etc/hostname.{\it if}} file.  As part
of system boot, the contents of these files are passed to {\tt ifconfig}
to configure the interface's IPMP-related attributes.  In addition, {\tt
  in.mpathd} is started to begin probing interface health and perform any
necessary address migrations.  At startup, {\tt in.mpathd} reads {\tt
  /etc/default/mpathd}, and adjusts its behavior accordingly.

Because the IPMP rearchitecture maintains {\tt ifconfig} compatibility,
existing {\tt /etc/hostname.{\it if}} files will continue to work, and no
changes to {\tt /etc/hostname.{\it if}} files will be required as part of
system upgrade.  However, by analogy with {\tt ifconfig}, our
documentation (e.g, on {\tt docs.sun.com}) will encourage administrators
to put all IPMP data addresses in new {\tt/etc/hostname.{\it ipmpif}}
files, rather than in the underlying interface's {\tt /etc/hostname.{\it
    if}} file.  In addition, administrators who wish to associate a
particular IPMP interface with a particular groupname will need to create
a corresponding {\tt /etc/hostname.{\it ipmpif}} file containing {\tt ipmp
  group {\it groupname}}.

For instance, the current way to configure the configuration used
throughout this document is:
\begin{alltt}
  /etc/hostname.ce0:
     group a
     129.146.17.55 netmask + broadcast + up
     addif 129.146.17.56 deprecated -failover netmask + broadcast + up
  /etc/hostname.ce1:
     group a
     129.146.17.57 netmask + broadcast + up
     addif 129.146.17.58 deprecated -failover netmask + broadcast + up
\end{alltt}

That will continue to work, but this will be preferred:
\begin{alltt}
  /etc/hostname.net17:
    ipmp group a
    129.146.17.55 netmask + broadcast + up
    addif 129.146.17.57 netmask + broadcast + up
  /etc/hostname.ce0:
    group a 129.146.17.56 -failover
  /etc/hostname.ce1:
    group a 129.146.17.58 -failover
\end{alltt}

\subsubsection{Interface Creation Order at System Boot}

For the above example to work, {\tt net17} must configured at boot prior
to the configuration of {\tt ce0} and {\tt ce1} -- otherwise, the system
would implicitly create an {\tt ipmp0} interface and assign {\em it}
(rather than {\tt net17}) to group {\tt a}.  To address this, any
{\tt/etc/hostname.{\it ipmpif}} files (easily identified since their first
token will be {\tt ipmp}) will be configured prior to configuring any
underlying interfaces.

\subsubsection{Missing Interfaces at System Boot}

While boot appears straightforward, there is currently an edge case that
significantly complicates matters.  In particular, suppose an
{\tt /etc/hostname.{\it if}} file exists for an interface that is part
of an IPMP group, but the interface cannot be plumbed.  In this case, the
IP addresses in that {\tt /etc/hostname.{\it if}} file {\it must} be
brought up on another interface in the group, so that the system remains
completely available.  Unfortunately, since the interface cannot be
plumbed, {\tt in.mpathd} is unaware of the ``failed'' interface, and thus
the system boot scripts must implement failover on {\tt in.mpathd}'s
behalf.  Because selecting the optimal interface to failover to is
complicated, this logic significantly complicates system boot.

Since the concept of address failover no longer exists, this logic will be
able to be simplified considerably.  Specifically, in the future, the boot
scripts will still try to plumb each interface, and will still build a
list of interfaces that could not be plumbed.  However, once the list of
failed interfaces has been constructed, the boot scripts will only need to
determine what IPMP group each interface was supposed to be in, and add
its data addresses directly to the appropriate IPMP group interface
(potentially creating IPMP groups along the way).

A pleasing side effect of the new model is that a system will be able to
boot properly even when {\em all} of the group's underlying interfaces
cannot be plumbed, and will automatically make use of any underlying
interfaces that are subsequently dynamically reconfigured into the system.

\subsubsection{Boot Environment  Changes}
As part of handling missing interfaces, the boot scripts must check
whether a given IPMP group already exists.  This can easily be done using
{\tt ipmpstat}, but will require that {\tt ipmpstat} be in {\tt /sbin}
since {\tt /usr} may not be available yet.  An {\tt ipmpstat} symlink will
be provided in {\tt /usr/sbin} for convenience and consistency with other
administrative programs used during boot, such as {\tt mount(1M)}.  Since
{\tt ipmpstat} makes use of {\tt libipmp} (see section \ref{proglibipmp}),
{\tt libipmp} will be moved from {\tt /usr/lib} to {\tt /lib}.

Since {\tt ipmpstat} queries live state within {\tt in.mpathd}, it must
also be running and thus must also be in {\tt /sbin}.  Long ago, {\tt
  in.mpathd} resided in {\tt /sbin} but was moved to {\tt /usr/lib/inet}
to make use of {\tt libsysvent.so} back before dynamic linking was
possible in {\tt /sbin}.  However, programs in {\tt /sbin} are now
dynamically linked and thus {\tt in.mpathd} can be moved back to {\tt
  /sbin}, and a compatibility symlink put in {\tt /usr/lib/inet}.  In
addition, the undocumented {\tt SUNW\_NO\_MPATHD} environment variable --
which prevented {\tt in.mpathd} from starting until {\tt /usr} was mounted
-- will be removed.  However, to ensure that {\tt in.mpathd} is started on
systems only using the IPMP anonymous group, the {\tt net-init} script
will continue to invoke {\tt in.mpathd} with the undocumented {\tt -a}
(adopt) flag if it is not already running.  As is currently the case, {\tt
  in.mpathd} will automatically exit if there are no IP interfaces to
track.

\subsection{Dynamic Reconfiguration}

Since dynamic reconfiguration is performed at the hardware-level, the
administrative model for DR will be unchanged.  That is, the administrator
will continue to refer to specific physical devices when using
{\tt if\_mpadm}, and specific attachment points when using
{\tt cfgadm}.  Since it is not associated with any specific piece of
hardware, it will not be possible to offline or remove an IPMP interface.

\subsection{Routing Table Administration}

Understanding and manipulating the routing table will become significantly
simpler with the new model.  Specifically, the core issues will be fixed
since a routing table entry will now be able to refer to the IPMP group as
a whole.  This is semantically identical to the current behavior, but
suffers from none of the existing corner cases, such as what to do when
the interface associated with a given route is removed from the IPMP
group.  It also makes it clear to the administrator that the route is
associated with the group as a whole, rather than a specific interface in
a group.

For instance, the routing table for the configuration used throughout
currently looks like:

\begin{alltt}
  129.146.17.0         129.146.17.56        U         1      0  ce0
  129.146.17.0         129.146.17.58        U         1      0  ce1
  129.146.17.0         129.146.17.55        U         1   6582  {\bf{ce0:1}}
  129.146.17.0         129.146.17.57        U         1    191  {\bf{ce1:1}}
  default              129.146.17.1         UG        1  15273  {\bf{ce0}}
\end{alltt}

Specifically, we see interface routes for the data addresses on {\tt ce0}
and {\tt ce1}, and that {\tt ce0} has been arbitrarily picked to represent
the IPMP group for the {\tt default} route.  With the IPMP interface, this
will look like:

\begin{alltt}
  129.146.17.0         129.146.17.56        U         1      0  ce0
  129.146.17.0         129.146.17.58        U         1      0  ce1
  129.146.17.0         129.146.17.55        U         1   6582  {\bf{ipmp0}}
  129.146.17.0         129.146.17.57        U         1    191  {\bf{ipmp0:1}}
  default              129.146.17.1         UG        1  15273  {\bf{ipmp0}}
\end{alltt}

As one would expect, routes will be added using the IPMP interface -- for
instance, to add another default route:
\begin{alltt}
  # route add -net default 129.146.17.2 -ifp ipmp0
\end{alltt}
For backward compatibility, this will also be supported:
\begin{alltt}
  # route add -net default 129.146.17.2 -ifp ce0
\end{alltt}

Specifically, the kernel will automatically convert this to a request to
add a route on the corresponding IPMP group interface; see section
\ref{progrtadm} for details.  (The first two routes in the preceding
routing table examples are interface routes installed by the kernel when
the test addresses are brought up and are used by {\tt in.mpathd} for
probe traffic.)

\subsection{IPv6 Configuration}

\subsubsection{IPv6 Link-Local Test Addresses}
\label{admv6ll}
Currently, when an interface is plumbed for IPv6, an IPv6 link-local
address is automatically generated and brought up.  Thus, each physical
interface in an IPMP group has its own IPv6 link-local address.  While
these addresses do not fail over as part of interface failure, they will
only be used as test addresses by {\tt in.mpathd} if explicitly marked
{\tt IFF\_NOFAILOVER} by the administrator.

To continue to provide failure and repair detection for IPv6 hosts, the
existing semantics of IPv6 link-local addresses on underlying interfaces
will be preserved.  In particular, IPv6 link-local addresses will still
never fail over, and will stillonly be used as test addresses by {\tt
  in.mpathd} if explicitly marked {\tt IFF\_NOFAILOVER} by the
administrator\footnote{We considered having the IP module automatically
  mark all link-locals on underlying IP interfaces as {\tt
    IFF\_NOFAILOVER}, but this introduces additional complexity -- e.g.,
  the IP module would need to track whether the administrator or IP module
  set {\tt IFF\_NOFAILOVER} so that the IP module could restore the old
  value if the IP interface later leaves the group.}.

\subsubsection{IPv6 Link-Local Data Addresses}

Since link-local test addresses cannot fail over on interface failure,
they are unsuitable for application use.  To fill this need, when an IPv6
IPMP interface is created, the IP module will automatically configure an
IPv6 ``link-local'' data address on it.  As described in RFC 2373, IPv6
link-local addresses are constructed from a 64-bit {\it Interface ID},
which is in turn typically generated using the associated link's hardware
address.  Since the IPMP interface does not have a dedicated hardware
address, a different approach must be used to generate the Interface ID.

Specifically, we propose a variant of the algorithm specified in section
3.2.1 of RFC 3041 (Privacy Extensions for Stateless Address
Autoconfiguration) which will compute the MD5 hash using the IPMP
interface name, the zone name, and the host identifier (as reported by
{\tt hostid(1)}), and will map that MD5 hash to an Interface ID as per RFC
3041.  This will ensure that a given IPMP interface will consistently
generate the same Interface ID (and thus the same link-local address)
while minimizing risk of duplicates.  Note that as with other IPv6
interfaces, the administrator can also override the generated Interface ID
by using the {\tt ifconfig token} subcommand.

Duplicate address detection for the link-local address will be performed
on one of the active underlying interfaces (provided one exists) to ensure
that no other on-link hosts are currently using it.  If there are no
active underlying interfaces in the IPMP group, then the link-local
address will be unreachable by other hosts.  Otherwise, it will be
associated with the hardware address of one of the active interfaces in
the IPMP group.  If the link-local address must be reassociated with a
different underlying interface's hardware address (e.g., as part of
interface failure), and an unsolicited Neighbor Advertisement message will
be sent to update other hosts on the link.

The Neighbor Solicitation and Neighbor Advertisement messages associated
with the IPMP interface's IPv6 link-local address will use the
currently-associated hardware address for the link-layer address fields.
However, note that the messages themselves may be sent over any active
interface in the group, which may have a different hardware address.

In total, the IPv6 link-local test and data address configuration might
look like:

\begin{alltt}
  ce0: flags=a000841<UP,RUNNING,MULTICAST,IPv6,NOFAILOVER> mtu 1500 index 2
         inet6 {\bf fe80::203:baff:fe16:51bd/10}
         groupname a
  ce1: flags=a000841<UP,RUNNING,MULTICAST,IPv6,NOFAILOVER> mtu 1500 index 3
         inet6 {\bf fe80::a00:20ff:feec:f42f/10}
         groupname a
  ipmp0: flags=b000843<UP,RUNNING,MULTICAST,IPv6,IPMP> mtu 1500 index 4
         inet6 {\bf fe80::45c0:9dc2:2f82:89c3/10}
         groupname a
\end{alltt}

\subsubsection{IPv6 Stateless Address Autoconfiguration}
\label{admautoconf}
Currently, when an interface is plumbed for IPv6, stateless address
autoconfiguration is attempted.  Thus, if autoconfiguration is successful,
each physical interface in an IPMP group has its own global address, and
the IPMP group will collectively have as many global addresses as
interfaces.

However, this approach will not work with IPMP interface, because no data
addresses will be hosted on the physical interfaces.  While we could
automatically migrate these data addresses to the IPMP interface, or
update {\tt in.ndpd} to automatically populate these addresses on the IPMP
interface, there is a more fundamental problem.  Specifically, the global
addresses are tied to the hardware addresses of the underlying interfaces
and are really owned by them.  Thus, if one unplumbed IPv6 on an
interface, one would expect the addresses to disappear from the IPMP
interface.  This conflicts with our proposed model, which states that once
an address has become part of an IPMP interface, it will remain there
until explicitly removed by the administrator.  Moreover, it would mean
that the set of global IPv6 addresses would change across a DR operation,
causing any applications using the changed address(es) to fail.

Thus, we will instead perform stateless address autoconfiguration on the
IPMP interface itself, as part of configuring IPv6 on it.  Specifically, as
with any other IPv6 interface, {\tt in.ndpd} will combine the IPMP
interface's Interface ID (whose generation was discussed in section
\ref{admv6ll}) with the global prefixes learned via router advertisements
to generate and configure IPv6 global addresses.  Note that only one
address per prefix will be configured -- and thus in the common case of a
single prefix, inbound global unicast IPv6 traffic will be received on a
single underlying interface.  This limitation will be documented, and
impacted sites will be advised to configure additional global IPv6
addresses, either statically or through DHCPv6.

\subsection{{\tt dladm}}

Because IPMP groups do not exist at the data-link layer (section
\ref{basdl}), they cannot be administered with {\tt dladm}. However, the
links used by the underlying interfaces in a group will be individually
displayed with {\tt dladm show-link}, and the devices which are used by
those links will be individually displayed with {\tt dladm show-dev}.

However, {\tt ipmpstat} (section \ref{admipmpstat}) will allow the
administrator to check the status of the IPMP interfaces in the system.
Because this tool will be IPMP-specific, its output will quickly allow the
administrator to answer specific questions about the health of their IPMP
interfaces, mitigating the lack of visibility from {\tt dladm}.

Sadly, aside from {\tt ifconfig}, there is no tool available to get a
broad view of the health and configuration of the IP layer of the system.
While adding such a tool is beyond the scope of this work, the addition of
the IPMP interface will make IPMP significantly easier to represent in
such a tool.  Further, the absence of this tool is not sufficient
justification to violate layering and provide support for IPMP in {\tt
dladm}.

Since IPMP interfaces will not be visible to {\tt dladm}, it will not be
used to provide administratively-chosen (vanity) names.  However, as
previously discussed, IPMP interfaces will be able to be given
administratively-chosen names when explicitly created through {\tt
  ifconfig}.  See section \ref{basnaming} for a summary of naming issues
associated with IPMP interfaces.

\subsection{{\tt kstat}}
\label{admkstat}

The IPMP interface will also have kstats which will be displayed with {\tt
  kstat(1M)}, and consumed by utilities such as {\tt netstat}.  The
following hardware-independent kstats (described in {\tt gld(7D)}) will be
supported:
\begin{alltt}
  obytes          opackets         ipackets         multircv         link_up
  obytes64        opackets64       ipackets64       multixmt
  rbytes          oerrors          ierrors          brdcstrcv         
  rbytes64                                          brdcstxmt         
\end{alltt}

Note that the counters listed above will represent the number of packets
or bytes that have flowed through the given IPMP interface since it was
created, which may not be the same as the number of packets that have
flowed through the underlying interfaces.  For instance, if an interface
is removed from an IPMP group, the packets that were sent through that
interface will still be counted in the IPMP interface's {\tt opackets}
kstat.  The {\tt link\_up} kstat will be non-zero if at least one of the
underlying interface links is up (i.e., the IPMP interface has {\tt
  IFF\_RUNNING} set).

\subsection{{\tt netstat}}
\label{admnetstat}
Like any IP interface, basic statistics about each IPMP interface will be
available through {\tt netstat}:
\begin{alltt}
  \$ netstat -i
  Name  Mtu  Net/Dest     Address        Ipkts  Ierrs Opkts  Oerrs Collis Queue
  lo0   8232 loopback     localhost      165180  0    16280  0     0      0
  ce0   1500 monkeys      129.146.17.56  263828  0    329580 0     0      0
  ce1   1500 monkeys      129.146.17.58  481035  0    104774 0     0      0
  ipmp0 1500 monkeys      129.146.17.55  744863  0    434354 0     0      0
\end{alltt}

The interface statistics for the underlying interfaces in the group will
count the number of packets actually sent over that interface, whereas the
interface statistics for the group represent all the packets sent to and
received by the group as a whole.

In addition, {\tt netstat -p} can be used to display the IPv4 ARP
table, and {\tt netstat -p -f inet6} can be used to display the IPv6
Neighbor Discovery table.  For instance, invoking {\tt netstat -p}
might currently produce output such as:
\begin{alltt}
  ce0    129.146.17.55         255.255.255.255 SP    08:00:20:ed:79:05
  ce1    129.146.17.56         255.255.255.255 SP    08:00:20:ed:79:01
  ce0    129.146.17.58         255.255.255.255 SP    08:00:20:ed:79:05
  ce1    129.146.17.57         255.255.255.255 SP    08:00:20:ed:79:01
  ce0    224.0.0.0             240.0.0.0       SM    01:00:5e:00:00:00
  ce1    224.0.0.0             240.0.0.0       SM    01:00:5e:00:00:00
  ce1    huon.SFBay.Sun.COM    255.255.255.255       08:00:20:89:d4:17
  ce0    zion-sp.SFBay.Sun.COM 255.255.255.255       00:09:3d:00:16:17
\end{alltt}
In the future, this same ARP table will look like:
\begin{alltt}
  ipmp0  129.146.17.55         255.255.255.255 SP    08:00:20:ed:79:05
  ce0    129.146.17.56         255.255.255.255 SP    08:00:20:ed:79:01
  ce1    129.146.17.58         255.255.255.255 SP    08:00:20:ed:79:05
  ipmp0  129.146.17.57         255.255.255.255 SP    08:00:20:ed:79:01
  ce0    224.0.0.0             240.0.0.0       SM    01:00:5e:00:00:00
  ce1    224.0.0.0             240.0.0.0       SM    01:00:5e:00:00:00
  ipmp0  huon.SFBay.Sun.COM    255.255.255.255       08:00:20:89:d4:17
  ipmp0  zion-sp.SFBay.Sun.COM 255.255.255.255       00:09:3d:00:16:17
\end{alltt}

That is, published ARP entries associated with the IPMP test addresses and
multicast mappings will remain tied to a specific underlying interface (as
they must for correctness).  However, ARP cache entries associated with
other hosts (such as {\tt huon} and {\tt zion-sp}) will be associated with
the group's IPMP interface that they can be used by {\em any} interface in
the group.  Similarly, the IPMP group's data addresses will be associated
with the IPMP group interface.

Semantically identical changes will be made to the output of the Neighbor
Discovery table entries displayed through {\tt netstat -p -f inet6}.
\subsection{{\tt arp}}

The {\tt arp} command does not currently operate on interface names, and
is thus unaffected by this proposal.  However, {\tt arp} with no arguments
invokes {\tt netstat -p}, which is discussed in section \ref{admnetstat}.

\subsection{Packet Monitoring}
\label{baspm}
We propose to to use Clearview's \htmladdnormallinkfoot{IP-level
  observability component}{See PSARC/2006/475 and
  http://opensolaris.org/os/project/clearview/ipnet/} in order to provide
packet monitoring of an IPMP group as a whole at the IP layer.
Specifically, the aforementioned configuration would provide the following
nodes in {\tt /dev/ipnet}:
\begin{alltt}
  /dev/ipnet/lo0
             ce0
             ce1
             ipmp0
\end{alltt}
Packets sent to and received from {\tt ce0} and {\tt ce1} will be visible
by monitoring the corresponding {\tt ipnet} node; monitoring {\tt ipmp0}
will allow one to see all IP packets sent to and received from {\it all}
underlying interfaces in the IPMP group.  If promiscuous-mode is not
enabled, only IP packets destined to IP addresses hosted on a particular
interface (along with broadcast and multicast IP packets) will be visible.
Since only test addresses will be hosted on {\tt ce0} and {\tt ce1}, IP
promiscuous mode must be enabled to see the non-test traffic flowing over
those interfaces.  However, promiscuous mode is the default behavior for
most packet monitoring programs, including {\tt snoop}, so this should not
matter in practice.

Because IPMP is an IP abstraction, it will not be possible to
monitor an IPMP group at the data-link layer.  However, as is currently
the case, each underlying interface will be able to be monitored at the
data-link layer.  Additionally, {\tt ipmpstat -p} (section
\ref{admipmpstatp}) may be used by itself to  track down simple
IPMP probe-related problems, or in concert with packet monitoring tools to
root-cause more subtle issues (using the probe number for data
correlation).
\subsection{Packet Filtering}
\label{baspf}
Since the integration of \htmladdnormallinkfoot{Packet Filtering
  Hooks}{http://opensolaris.org/os/community/networking/files/pfhooks-design-2006-03-09.pdf}
into Solaris, packet filtering is no longer performed by inserting STREAMS
modules below IP, but by registering packet filtering routines to be
called back by IP itself on a per-IP-interface basis.  Thus, the
introduction of the IPMP group interface will enable packet filtering
rules to be applied to the group as a whole.  As such, administrators will
be able to specify rules that apply to IPMP group interfaces.  To ensure
that the filtering behavior of the group is predictable, once an interface
joins a group, packets flowing over it will only be subject to the group
interface's rules.  If an interface is removed from a group, the
underlying interface's rules (if any) will again be applied.

As part of this work, the {\tt ipmp\_hook\_emulation ndd} tunable -- which
provided an expedient way for IPMP and IP Filter to work together -- will
be rendered unnecessary and thus removed.  Subsequent work is planned to
make Packet Filtering Hooks available for public use, which will enable
third-party packet filtering software to work with IPMP as well.

\subsection{DHCP Interaction}
\label{basdhcp}
As shown in section \ref{admifconfigdhcp}, DHCP can be used on an IPMP
interface just as it can on a physical interface.  This makes
administrative sense because data addresses no longer move, and thus the
lease's handle (e.g., {\tt ipmp0:1}) will remain constant, allowing core
attributes to be specified in {\tt /etc/default/dhcpagent} and enabled at
boot using {\tt /etc/dhcp.{\it if}} files.  The DHCP client can already
automatically generate client identifiers for logical interfaces, and we
will extend this to also support physical interfaces that do not have a
hardware address.  Thus, explicit configuration of the {\tt CLIENT\_ID}
parameter in {\tt /etc/default/dhcpagent} will {\it not} be necessary.
Because the semantics of {\tt IFF\_RUNNING} on an IPMP interface mirror
the semantics on a traditional IP interface, the DHCP client will
automatically refresh any data address leases across a group failure.  As
with traditional IP interfaces, the MTU of an IPMP interface must be
sufficient to support DHCP.  As a degenerate case, an IPMP interface with
no interfaces (see section \ref{basipmpmtu}) does not have a large enough
MTU to support DHCP, and thus DHCP lease acquisition will fail.

Test addresses will also be able to be configured using DHCP.  Although a
test address can be configured on any logical interface, given that the
test address will typically be the only address on an underlying
interface, we will recommend that it be configured on the physical
interface.  Thus:

\begin{alltt}
  # ifconfig ce0 plumb
  # ifconfig ce0 group a
  # ifconfig ce0 dhcp start
  #
\end{alltt}

Note in particular that {\tt -failover} need not be specified, as the DHCP
client will automatically set {\tt IFF\_NOFAILOVER} prior to starting
lease acquisition\footnote{As per section \ref{progipmpflags}, attempting
  to clear {\tt IFF\_NOFAILOVER} on an address managed by DHCP will
  fail.}.  Although the DHCP client will use any active interface in the
group to maintain the address lease, infinite leases will be strongly
recommended for test addresses so that a group failure will not trigger
spurious repairs.  For example, if the DHCP client is unable to renew a
test address lease (e.g., because of a group failure), then it must
unconfigure the test address, which will cause {\tt in.mpathd} to disable
probe-based failure detection on the interface, possibly triggering a
spurious repair.

Although it is not documented, the boot scripts currently process the
contents of {\tt /etc/hostname.{\it if}} files even for interfaces using
DHCP.  So, to make the above configuration persistent:
\begin{alltt}
  # > /etc/dhcp.ce0
  # echo group a > /etc/hostname.ce0
\end{alltt}

This will be explicitly documented in the future.

One oddity is that if the {\tt REQUEST\_HOSTNAME} tunable is enabled, {\tt
  dhcpagent} will read each line of the {\tt /etc/hostname.{\it if}} file,
and if the a line is of the form {\tt inet {\it hostname}}, then it will
pass {\tt {\it hostname}} as a hint to the DHCP server (see {\tt
  dhcpagent(1M)}).  While awkward, this feature can be used on either an
underlying interface or an IPMP interface.  For example, to have {\tt
  dhcpagent} request hostname {\tt frobnitz} for IPMP interface {\tt ipmp0}:

\begin{alltt}
  # echo REQUEST_HOSTNAME=yes >> /etc/default/dhcpagent
  # > /etc/dhcp.ipmp0
  # echo ipmp group b > /etc/hostname.ipmp0
  # echo inet frobnitz >> /etc/hostname.ipmp0
\end{alltt}

Ostensibly to avoid interfering with static address configuration, the
DHCP client has traditionally monitored the {\tt IFF\_UP} flag on
DHCP-managed addresses and abandoned the lease if a change is detected.
However, the process of offlining an interface brings down its test
addresses, which would cause the current DHCP client to erroneously
sabandon test address leases.  As such, this monitoring will be removed
and the DHCP client will instead rely on normal lease expiration.  As
before, the DHCP client {\it will} abandon a lease if its address is
changed, thus ensuring that conflicts with static address configuration
will still be avoided.  Although {\tt IFF\_UP} monitoring is (briefly)
documented, this is to ease problem diagnosis rather than to provide a
programmatic interface (and further, programmatic interfaces already exist
to drop or release a lease), and thus the change is low-risk.

To allow DHCP-managed test addresses to work with DR, RCM will be enhanced
to also restore DHCP configuration across a DR operation; see section
\ref{basdr} for details.

\subsection{Zones Interaction}
\subsubsection{Shared IP Stack Zones}
An IPMP interface's addresses can be assigned to a shared-stack zone in
the same manner as a traditional physical IP interface.  Specifically,
since the IPMP interface {\it is} an IP interface, it (and any addresses)
can be specified using the {\tt zonecfg(1M)} {\tt physical} property --
e.g.,:
\begin{alltt}
     # zonecfg -z a
     a: No such zone configured
     Use 'create' to begin configuring a new zone.
     zonecfg:a> create
     zonecfg:a> set zonepath=/export/home/a
     zonecfg:a> set autoboot=true
     zonecfg:a> add net
     zonecfg:a:net> set address=192.168.0.1/24
     zonecfg:a:net>{\bf set physical=ipmp0}
     zonecfg:a:net> end
     zonecfg:a> exit\
\end{alltt}

Legacy configurations will also continue to work.  For instance, if {\tt
  ce0} is in the IPMP group associated with {\tt ipmp0}, and {\tt set
  physical=ce0} is instead specified above, then when {\tt zoneadmd}
brings up the address during zone boot, the address will be automatically
migrated to {\tt ipmp0}.  Since the address (rather than the interface) is
assigned to a zone, a subsequent {\tt ifconfig -a} in the booted zone will
show the address on a logical interface of {\tt ipmp0}.  Regardless, since
the migration at zone boot is potentially confusing and the configuration
suggests the wrong administrative model for IPMP, this will not be a
recommended configuration.  As such, {\tt zoneadm verify} will be updated
to warn if a {\tt physical} property is set to an underlying interface.

Since {\tt in.mpathd} requires privileges that are not granted to a
shared-stack zone by default, and moreover, manages the health of IP
interfaces that are often in use by multiple zones, {\tt in.mpathd} will
continue to run from the global zone. Because {\tt ipmpstat} must
communicate with {\tt in.mpathd} to retrieve its information, and
inter-zone communication facilities would noticeably complicate the
implementation, {\tt ipmpstat} will not operate in a shared-stack
non-global zone.  If this becomes a significant limitation, {\tt ipmpstat}
support could be implemented, but only to show the IPMP state associated
with that zone.  Specifically, {\tt ipmpstat} output would be restricted
to showing addresses assigned to the zone, along with the IP interfaces
associated with those addresses and the IPMP groups associated with those
IP interfaces.

\subsubsection{Exclusive IP Stack Zones}
Since IPMP is an IP-level construct, and since the resources exported to
exclusive IP-stack zones are data-links, the introduction of the IPMP
interface will not affect the configuration of exclusive-stack zones.  For
instance, the following will still configure a zone with two data-links,
{\tt ce1000} and {\tt ce1001}, across which IPMP could be configured:
\begin{alltt}
     # zonecfg -z a
     a: No such zone configured
     Use 'create' to begin configuring a new zone.
     zonecfg:a> create
     zonecfg:a> set zonepath=/export/home/a
     zonecfg:a> set autoboot=true
     zonecfg:a>{\bf set ip-type=exclusive}
     zonecfg:a> add net
     zonecfg:a:net>{\bf set physical=ce1000}
     zonecfg:a:net> end
     zonecfg:a> add net
     zonecfg:a:net>{\bf set physical=ce1001}
     zonecfg:a:net> end
     zonecfg:a> exit\
\end{alltt}

The local zone could then configure its {\tt /etc/hostname.{\it if}} files
to use IPMP using one of the approaches discussed in section
\ref{admboot}.  As with all exclusive IP-stack zones, the resulting IP
configuration would be local to the zone and invisible to the global zone.
Therefore, as is currently the case, any IP interfaces using IPMP in that
zone will be exclusively monitored by a zone-local {\tt in.mpathd} daemon.
Similarly, running {\tt ipmpstat} in that zone will show only the IPMP
state associated with the IP interfaces in the zone, and those interfaces
will be invisible to {\tt ipmpstat} in the global zone.  One caveat is
that sysevents do not work yet in non-global zones, so {\tt ipmpstat -p}
will not work in non-global zones until that restriction is lifted.

\pagebreak
\section{IPMP Rearchitecture: Programming Impact}

The majority of networking programming is done using abstractions that are
above the IP interface layer, and thus most software will be unaffected by
this rearchitecture. Indeed, it was this thought that motivated the
original IPMP model -- though hindsight has shown that the exceptions
constitute a set of core programming interfaces that are used by a number
of critical applications, such as routing daemons.

To address this problem, a variety of ``solutions'' have been explored,
including avoiding certain IPMP configurations, ``porting'' applications
to be IPMP-aware, or simply prohibiting use of the application on systems
running IPMP.  All of these approaches limit widespread adoption of IPMP
on Solaris and are therefore inadequate.

As will become clear, the introduction of the IPMP interface will resolve
a wide variety of existing programming issues concerning IPMP.  In
particular, the new model will {\it finally} allow IP-based networking
applications to operate without modification on systems using IPMP.  It
will provide facilities whereby applications that would benefit from being
IPMP-aware (e.g., network management software) can opt-in and thus provide
first-class IPMP support.  The API changes required to support the new
model are outlined in this section.

\subsection{Socket Interface Discovery {\tt ioctl} Operations}
\label{progifdisc}
Networking applications discover the set of configured network interfaces
and IP addresses on the system through four socket {\tt ioctl} operations:
{\tt SIOCGIFNUM}, {\tt SIOCGLIFNUM}, {\tt SIOCGIFCONF} and {\tt
  SIOCGLIFCONF}.  Currently, these operations return all interfaces and IP
addresses, regardless of whether the interface or IP address is
appropriate for use by the application.  As a result, applications often
end up using IPMP test addresses or attempting to send packets out a
network interface that is {\tt INACTIVE}, {\tt FAILED}, or {\tt OFFLINE}.
Thus, currently any applications which perform these operations must be
``ported'' to be IPMP-aware and recompiled in order to work properly on
IPMP systems\footnote{One prominent example is SNMP.}.

With the new model, this will no longer be necessary.  Specifically, by
default, {\bf any physical network interface in an IPMP group will be
  invisible to these {\tt ioctl} operations, preventing them from being
  accidentally used by IPMP-unaware applications.} However, any IPMP
interfaces and any addresses hosted on them {\it will} be visible to
applications.

For administrative applications such as {\tt ifconfig}, {\bf a new
{\tt LIFC\_UNDER\_IPMP} flag will be provided}. Specifically, if this flag
is set in the {\tt lifc\_flags} member of the {\tt struct lifreq}
when using {\tt SIOCGLIFNUM} or {\tt SIOCGLIFCONF}, then physical
network interfaces in IPMP groups will be returned as well.  While this
flag will be documented, we expect that only a handful of unbundled
administrative applications will be revised to set it.  It is important to
note that administrative applications that are not updated will still
operate properly, but will simply be unaware of the underlying network
interfaces in an IPMP group.  Further, these applications will likely
require other changes, e.g., in order to indicate the relationship between
an IPMP interface and the physical interfaces that comprise the group.

Table \ref{tabsockdisc} summarizes these changes by showing the results of
the various {\tt ioctls} on the configuration we have been using throughout.
(Since {\tt lo0} is always configured on any functioning machine, it is
included in the results.)
\pagebreak
\begin{table}[h]
\begin{tabular}{|l||r|r|} \hline
{\tt ioctl} & current & proposed \\ \thline
{\tt SIOCGIFNUM} & 5  &  3 \\ \hline
{\tt SIOCGLIFNUM} &  5  &  3 \\ \hline
{\tt SIOCGLIFNUM LIFC\_UNDER\_IPMP} & {\it N/A} & 5 \\ \hline
{\tt SIOCGIFCONF} & {\tt ce0 ce0:1 ce1 ce1:1 lo0} & {\tt ipmp0 ipmp0:1 lo0} \\ \hline
{\tt SIOCGLIFCONF} & {\tt ce0 ce0:1 ce1 ce1:1 lo0} & {\tt ipmp0 ipmp0:1 lo0} \\ \hline
{\tt SIOCGLIFCONF LIFC\_UNDER\_IPMP} & {\it N/A} & {\tt ce0 ce1 ipmp0 ipmp0:1 lo0} \\ \hline
\end{tabular}
\caption{Proposed Socket Discovery ioctl Changes}
\label{tabsockdisc}
\end{table}

\subsection{Examination and Manipulation of Socket Interface Flags}
\label{progsockflags}			
Table \ref{tabifflags} summarizes the flags that will be able to be set
through {\tt SIOCSLIFFLAGS}, and may be seen as set with {\tt
  SIOCGLIFFLAGS}, on both an IPMP interface and the underlying interfaces
in an IPMP group.  Aside from the asterisked entries, the semantics of
issuing {\tt SIOCSLIFFLAGS} on the underlying interface will be unchanged
from the current behavior.  Interesting cells of the table are shown in
bold and discussed below:
\begin{enumerate}

\item Bringing an address on an underlying interface {\tt IFF\_UP}
  is still permitted, but unless the address is marked {\tt
    IFF\_NOFAILOVER}, the kernel will immediately move it to the
  corresponding IPMP interface.

\item Per section \ref{basipmpflags}, {\tt IFF\_STANDBY},
  {\tt IFF\_OFFLINE}, {\tt IFF\_INACTIVE}, and {\tt IFF\_NOFAILOVER}
  will be prohibited on the IPMP interface.  However, so that
  {\tt in.mpathd} can implement {\tt STANDBY} (section \ref{basstandby}) and
  {\tt FAILBACK=no} (section \ref{basfailbackno}) support, {\tt
    IFF\_INACTIVE} will now be modifiable on underlying interfaces.
  Section \ref{progipmpflags} details how {\tt IFF\_INACTIVE}
  interacts with {\tt IFF\_FAILED} and {\tt IFF\_STANDBY}.

\item Per section \ref{basipmpif}, the new {\tt IFF\_IPMP}
  flag will be automatically set on all IPMP interfaces.

\item Although the IPMP interface is not associated with any specific
  physical hardware, it is not marked {\tt IFF\_VIRTUAL} since it can
  still be used to send and receive IP traffic.  ({\tt IFF\_VIRTUAL} means
  that an interface has {\bf no} associated physical hardware at all, thus
  e.g., routing daemons will never forward ``through'' it.)

\item {\tt IFF\_COS\_ENABLED} will only be set if {\it all} of the
  interfaces in the IPMP group have it set.

\item Per section \ref{basgroupif}, interfaces with {\tt
      IFF\_POINTOPOINT} addresses will not be allowed into a group.
    Since all {\tt IFF\_UNNUMBERED} addresses are {\tt IFF\_POINTOPOINT},
    {\tt IFF\_UNNUMBERED} will never be set.

\item Per section \ref{basipmpflags}, setting {\tt
    IFF\_NOFAILOVER} will now also set {\tt IFF\_DEPRECATED}.  Per
  section \ref{progipmpflags}, clearing {\tt IFF\_NOFAILOVER} on an {\tt
    IFF\_UP} or application-managed address will be prohibited.

\item Since setting {\tt IFF\_ROUTER} on only part of an IPMP group makes
  no sense, attempting to set or clear it on an IPMP interface or any
  interface in the group will set or clear it on {\it all} interfaces
  in the IPMP group.  Additionally, any interfaces added to a group will
  have their {\tt IFF\_ROUTER} flags updated to match the current group
  value.

\item Setting {\tt IFF\_NOARP}, {\tt IFF\_NONUD}, or
  {\tt IFF\_XRESOLV} on an underlying interface will affect only the
  packets associated with applications directly bound to those underlying
  interfaces (e.g., {\tt in.mpathd}'s probes).  As IPMP is tightly
  integrated with ARP and IPv6 Neighbor Discovery, attempting to set {\tt
  IFF\_NOARP}, {\tt IFF\_NONUD}, or {\tt IFF\_XRESOLV} will fail on an
  IPMP interface.

\item Though the behavior of {\tt IFF\_FIXEDMTU} will be unchanged, it remains special
    since it can be set by an application, but not directly through {\tt
    SIOCSLIFFLAGS}.  Instead, {\tt SIOCS[L]IFMTU} must be used.

\end{enumerate}


\begin{table}[h!]
\begin{tabular}{|l||c|c|c|c|} \hline
{\tt ioctl} & \multicolumn{2}{|c|}{{\bf Modifiable?}}
 & \multicolumn{2}{|c|}{{\bf Possibly Set?}} \\ \hline
& {\bf IPMP Interface} & {\bf Underlying} & {\bf IPMP Interface} & {\bf Underlying}
\\ \thline
{\tt IFF\_UP}		&	Yes& {\bf Yes*}&	Yes&	Yes \\ \hline
{\tt IFF\_RUNNING}	&	 No& No        &	Yes&	Yes \\ \hline
{\tt IFF\_FAILED}	&        No& Yes       &        Yes&	Yes \\ \hline
{\tt IFF\_STANDBY}	&  {\bf No}& Yes       &   {\bf No}&	Yes \\ \hline
{\tt IFF\_INACTIVE}	&  {\bf No}& {\bf Yes*}&   {\bf No}&	Yes \\ \hline
{\tt IFF\_OFFLINE}	&  {\bf No}& Yes       &   {\bf No}&	Yes \\ \hline
{\tt IFF\_VIRTUAL}	& 	 No& No        &   {\bf No}&	No  \\ \hline
{\tt IFF\_IPMP{\bf *}}  &  {\bf No}& {\bf No}  &  {\bf Yes}& {\bf No} \\ \hline
{\tt IFF\_LOOPBACK}	&	 No& No	       &	 No&	No  \\ \hline
{\tt IFF\_POINTOPOINT}  &	 No& No	       &   {\bf No}& {\bf No*} \\ \hline
{\tt IFF\_UNNUMBERED}	& 	 No& No	       &   {\bf No}& {\bf No*} \\ \hline
{\tt IFF\_NOFAILOVER}	&  {\bf No}& {\bf Yes*}&   {\bf No}&	Yes \\ \hline
{\tt IFF\_DUPLICATE}    &	No&	No&		Yes&	Yes \\ \hline
{\tt IFF\_DHCPRUNNING}  &	Yes&    Yes&            Yes&	Yes \\ \hline
{\tt IFF\_PRIVATE}      &	Yes&	Yes&		Yes&	Yes \\ \hline
{\tt IFF\_PREFERRED}    &	Yes&	Yes&		Yes&	Yes \\ \hline
{\tt IFF\_TEMPORARY}    &	Yes&	Yes&		Yes&	Yes \\ \hline
{\tt IFF\_DEPRECATED}   &	Yes&	Yes&		Yes&	Yes \\ \hline
{\tt IFF\_ADDRCONF}     &	Yes&	Yes&		Yes&	Yes \\ \hline
{\tt IFF\_ANYCAST}      &	Yes&	Yes&		Yes&	Yes \\ \hline
{\tt IFF\_NOXMIT}       &	Yes&	Yes&		Yes&	Yes \\ \hline
{\tt IFF\_NOLOCAL}      &	Yes&	Yes&		Yes&	Yes \\ \hline
{\tt IFF\_NOARP}        &  {\bf	No}&	{\bf Yes*}&	 No&	Yes \\ \hline
{\tt IFF\_NONUD}        &  {\bf	No}&	{\bf Yes*}&	 No&	Yes \\ \hline
{\tt IFF\_XRESOLV}      &  {\bf	No}&	{\bf Yes*}&	 No&	Yes \\ \hline
{\tt IFF\_COS\_ENABLED} &	No&	No&		{\bf Yes}&	Yes \\ \hline
{\tt IFF\_BROADCAST}    &	No&	No&		Yes&	Yes \\ \hline
{\tt IFF\_MULTICAST}    & 	No&	No&		Yes&	Yes \\ \hline
{\tt IFF\_MULTI\_BCAST} &	No&	No&		No& 	Yes \\ \hline
{\tt IFF\_ROUTER}       &	Yes&	{\bf Yes*}&	Yes&	Yes \\ \hline
{\tt IFF\_IPV4}         &	No&	No&		Yes&	Yes \\ \hline
{\tt IFF\_IPV6}         &	No&	No&		Yes&	Yes \\ \hline
{\tt IFF\_FIXEDMTU}     &{\bf Yes}&	{\bf Yes*}& Yes&	Yes \\ \hline
\end{tabular}
\caption{Proposed Interface Flag Behavior 
  (``{\bf*}'' denotes changed semantics)}
\label{tabifflags}
\end{table}

\pagebreak

\subsection{IPMP-Specific Interface Flag Considerations}
\label{progipmpflags}
As just discussed, unlike the underlying physical interfaces, the
following IPMP-specific flags will {\it never} be set on an IPMP group
interface:

\begin{table}[h!]
\begin{tabular}{|l|l|} \hline
\textbf{flag}         & \textbf{meaning}\\ \thline
{\tt IFF\_STANDBY}    & Interface is administratively assigned to only be
used if another interface fails \\ 
{\tt IFF\_INACTIVE}   & Interface is functioning, but currently inactive
and must not be used \\
{\tt IFF\_OFFLINE}    & Interface is an unknown state and unavailable for use \\ \dhline
{\tt IFF\_NOFAILOVER} & Address is administratively assigned as a test
address and thus must not be used \\ \hline
\end{tabular}
\end{table}

Thus, applications will no longer encounter interfaces with these flags
set, obviating the need to understand and respect the flags.

Although {\tt IFF\_NOFAILOVER} prevents an address from failing over, it
does not currently prevent it from being used as a source address.  As
such, one must also set {\tt IFF\_DEPRECATED} to ensure that the source
address selection algorithm will not prefer the address.  While preserving
the separation between {\tt IFF\_NOFAILOVER} and {\tt IFF\_DEPRECATED} is
necessary (e.g., so that IPv6 data addresses can be deprecated as per
the RFC's), an {\tt IFF\_NOFAILOVER} address should always be {\tt
  IFF\_DEPRECATED}.  As such, setting {\tt IFF\_NOFAILOVER} will now
  automatically set {\tt IFF\_DEPRECATED}.

Because clearing {\tt IFF\_NOFAILOVER} on an {\tt IFF\_UP} address would
require migration of an {\tt IFF\_UP} address, it will not be permitted.
Since the only application that manipulates the {\tt IFF\_NOFAILOVER} flag
is {\tt ifconfig}, it will instead be changed to clear {\tt IFF\_UP} prior
to setting {\tt IFF\_NOFAILOVER} and to restore {\tt IFF\_UP} afterwards,
preserving administrative compatibility.  To avoid confusing applications
that may be managing an {\tt IFF\_NOFAILOVER} address, clearing {\tt
  IFF\_NOFAILOVER} on an address with {\tt IFF\_DHCPRUNNING}
or {\tt IFF\_ADDRCONF} set will not be permitted.

The current IPMP design has an additional IPMP-specific flag, {\tt
  IFF\_FAILED}, which indicates that the interface has failed and thus can
no longer be used.  This flag is similar to the well-known and
well-established {\tt IFF\_RUNNING} flag, but differs in that an external
entity (in this case, {\tt in.mpathd}) has determined that the interface
is no longer fit for use.  However, since {\tt IFF\_FAILED} is peculiar to
Solaris, most applications do not consult it, defeating its purpose.

While both flags will remain, the IP module {\bf will be updated to clear
  {\tt IFF\_RUNNING} whenever {\tt IFF\_FAILED} is set on the IPMP
interface.}  Since IPMP-unware applications will interact exclusively with
IPMP interfaces, this change will allow them to work correctly without
modification. As before, {\tt IFF\_RUNNING} and {\tt IFF\_FAILED} will
remain independent on underlying interfaces, which will enable IPMP-aware
applications (and administrators with established habits) to determine an
underlying interface's link state without having to query DLPI.

The management of the IPMP interface's {\tt IFF\_RUNNING} and {\tt
  IFF\_FAILED} flags will be handled by the IP module, and will not be
able to be manipulated by applications.  In particular, the IP module will
set {\tt IFF\_FAILED} and clear {\tt IFF\_RUNNING} when all of the
underlying interfaces have {\tt IFF\_FAILED} set.  As a degenerate case,
since an IPMP interface with no underlying interfaces will not be able to
send or receive data, it too will have {\tt IFF\_RUNNING} cleared and {\tt
  IFF\_FAILED} set.

As discussed in section \ref{basstandby}, {\tt IFF\_INACTIVE} will now be
managed by {\tt in.mpathd}.  To preserve existing semantics that an {\tt
  INACTIVE} interface is {\it always} functioning, setting {\tt
  IFF\_INACTIVE} on an {\tt IFF\_FAILED} interface or vice-versa will
fail.  Further, to preserve the existing relationship between {\tt
  IFF\_INACTIVE} and {\tt IFF\_STANDBY}, setting {\tt IFF\_STANDBY} on a
functioning interface will set {\tt IFF\_INACTIVE}, and clearing
{\tt IFF\_STANDBY} will clear {\tt IFF\_INACTIVE} if set.  In
addition, to ensure that the {\tt STANDBY} and {\tt FAILBACK=no} {\tt
  INACTIVE} semantics remain distinct, setting {\tt IFF\_STANDBY} will
fail if {\tt IFF\_INACTIVE} is already set.

As previously discussed, one additional IPMP-specific flag,
{\tt IFF\_IPMP}, will be set on all IPMP group interfaces to indicate
that the interface has the special properties and semantics described
throughout this document.

\subsection{Routing Socket Interaction}

\subsubsection{Using Routing Sockets on the IPMP Group Interface}

As mentioned previously, applications using routing sockets currently get
an incomplete and confusing picture of the system's networking
configuration.  The IPMP group interface abstraction will fix
this, since applications will now be able to monitor a routing socket for
changes to the IPMP interface, rather than changes to the collection of
underlying interfaces in the group.  Specifically:
\begin{itemize}
\item The addition and removal of {\tt UP} data addresses from the group
  will trigger {\tt RTM\_NEWADDR} and {\tt RTM\_DELADDR} events associated
  with the IPMP group.

\item State changes associated with the group will trigger {\tt
    RTM\_IFINFO} events associated with the IPMP group.
\end{itemize}

However, applications will be insulated from irrelevant interface
failures, repairs, and DR operations since:
\begin{itemize}
\item Addresses never migrate on the IPMP interface.
\item State changes on the underlying interfaces will not cause
{\tt RTM\_IFINFO} messages to be generated unless they affect the IPMP
group interface as a whole (e.g., the last interface in the group has
failed).
\end{itemize}
\subsubsection{Using Routing Sockets on the Underlying Physical Interfaces}

For consistency with our proposed changes to the socket discovery {\tt
  ioctl} operations (section \ref{progifdisc}), {\bf by default, routing
  sockets will not report events on interfaces in IPMP groups}, and
messages sent to the kernel that refer to the underlying interfaces will
fail with {\tt EINVAL}.  Further, when an interface is placed into an IPMP
group, {\tt RTM\_DELADDR} messages will be generated for each of the {\tt
  IFF\_NOFAILOVER} addresses that are left behind, and an {\tt
  RTM\_IFINFO} message will be sent indicating that the interface is now
down.  Similarly, when an underlying interface is removed from an IPMP
group, an {\tt RTM\_IFINFO} message will be sent indicating that the
interface is now up, and {\tt RTM\_NEWADDR} messages will be generated for
each {\tt IFF\_UP} address found on the interface.  Note that since only
{\tt IFF\_NOFAILOVER} addresses are permitted on interfaces in an IPMP
group, all {\tt RTM\_NEWADDR} messages will be for {\tt IFF\_NOFAILOVER}
addresses.

As with {\tt SIOCGLIFCONF}, it is expected that most applications will
wish to remain unaware of the underlying interfaces in an IPMP group.
However, administrative applications which need to be aware of changes to
the underlying interfaces will be able to set the new {\tt RT\_AWARE}
socket option to the value {\tt RTAW\_UNDER\_IPMP}.  This will cause
routing socket messages to:
\begin{itemize}
\item Be received for each change to an underlying interface in an IPMP group.
\item Be able to be sent that refer to underlying interfaces in an IPMP group.  However, since routes must be associated with the IPMP group interface, operations other than {\tt RTM\_GET} will fail with {\tt EINVAL}.
\item {\it Not} be generated when an interface is placed into or removed from an IPMP group (as described earlier).
\end{itemize}
As with {\tt LIFC\_UNDER\_IPMP}, the {\tt RT\_AWARE} socket option will be
documented, along with the values {\tt RTAW\_UNDER\_IPMP} and {\tt
  RTAW\_DEFAULT} (aka ``unaware'').  Note that additional {\tt RT\_AWARE}
values can be added in the future to further to align routing sockets with
existing {\tt SIOCGLIFCONF} behavior -- e.g., {\tt RTAW\_ALLZONES} could
mirror the existing {\tt LIFC\_ALLZONES} flag.  Finally, since the {\tt
  RT\_AWARE} socket option will not be a generic {\tt SOL\_SOCKET} option,
nor will it be tied to a specific IP protocol, a new {\tt SOL\_ROUTE}
socket option level will be introduced and documented.

\subsection{Routing Table Manipulation Operations}
\label{progrtadm}
Solaris provides two interfaces for manipulating the routing table: the
legacy BSD {\tt SIOCADDRT} / {\tt SIOCDELRT} {\tt ioctl} operations, and
routing sockets.

The legacy operations do not operate on interface names and thus are
unaffected.  However, as described in the last subsection, the routing
socket interfaces {\it are} affected.  For backward compatibility, an {\tt
  RTM\_ADD} request issued on an underlying interface will be
automatically mapped by the IP module to an {\tt RTM\_ADD} request on the
corresponding IPMP interface.  Nonetheless, performing {\tt RTM\_ADD}
requests directly on the IPMP interface will be strongly preferred.  Other
routing socket operations that manipulate the routing table ({\tt
  RTM\_DELETE}, and {\tt RTM\_CHANGE}) {\it must} be performed on the IPMP
interface associated with the group. Attempting to issue these operations
on an underlying interface will fail with {\tt EINVAL}.

While this is a new restriction, it should have little impact in practice
since the other changes described in this section effectively hide the
underlying interfaces in a group from applications.  Thus, the only way
such an interface should become known to an application is if it is
explicitly provided due to misconfiguration by an administrator.  If
future experience shows this restriction to be too strict, we can
implement mapping functionality for all operations.

\subsection{Interface Index Considerations}

Currently, since an IPMP group is not modeled as a single interface, APIs
that require an interface index are overloaded: they usually refer to {\it
  any} interface in the group, but may sometimes refer to a specific
interface in the group.  To help the kernel make the choice, several
undocumented socket options ({\tt IP\_DONTFAILOVER\_IF}, {\tt
  IPV6\_DONTFAILOVER\_IF}, and {\tt IPV6\_BOUND\_PIF}) have been
introduced over the years.

With the introduction of the IPMP interface, each IPMP group will have its
own interface index, retrieved with {\tt SIOCG[L]IFINDEX} and set with
{\tt SIOCS[L]IFINDEX}.  Thus, if an application enables the {\tt
  IP\_RECVIF}, {\tt IP\_RECVPKTINFO}, or {\tt IPV6\_RECVPKTINFO} socket
options, then the ancillary data associated with packets sent to the data
addresses in the group will have the index of the appropriate IPMP
interface.  Similarly, applications such as routing protocols can bypass
the routing table and send a packet over an active interface in the group
by specifying the group's IPMP interface index to the {\tt IP\_PKTINFO},
{\tt IPV6\_PKTINFO}, {\tt IP\_BOUND\_IF}, or {\tt IPV6\_BOUND\_IF} socket
options.

Because the IPMP group interface and all IP interfaces under it will have
their own interface indices, the standard socket options will enable
applications to send or receive on any or all IP interfaces in a
group\footnote{Since IPMP-unaware applications will not discover
  underlying interfaces, socket options will generally apply to the IPMP
  group interfaces.}. As such, the aforementioned undocumented socket
options will be rendered unnecessary and thus will be removed\footnote{
{\tt in.mpathd} will use {\tt
    IP\{,V6\}\_BOUND\_IF} in place of {\tt
    IP\{,V6\}\_DONTFAILOVER\_IF} to send and receive probes.}.

\subsection{Interface Metric Considerations}
\label{progifmetric}

Each IPMP interface will also have its own interface metric, retrieved
with {\tt SIOCGLIFMETRIC} and set with {\tt SIOCSLIFMETRIC}.  Since the
IPMP interface metric is an administratively-assigned property that is not
directly associated with any underlying interface, it will be set to zero
when the IPMP interface is created.  Because the routing metrics of underlying
interfaces in an IPMP group will never be adjusted by routing daemons,
issuing {\tt SIOCSLIFMETRIC} on an underlying interface will fail with
{\tt EINVAL}.

\subsection{Logical Interface Configuration}

Logical interfaces will be able to be added or removed from an IPMP group
interface using the {\tt SIOCLIFADDIF} and {\tt SIOCLIFREMOVEIF
ioctl} operations.  Once a new logical interface has been added to an
IPMP group interface, it will be configurable using the other socket
{\tt ioctl} operations discussed throughout this section.  Removing a
logical interface from an IPMP group interface will cause the system to
stop using any IP addresses that had been configured on it.

Logical interfaces will still be able to be created on or removed from the
underlying interfaces in an IPMP group as well.  This is primarily useful
for adding or removing IPMP test addresses.  However, if the {\tt IFF\_UP}
flag is later set, any logical interfaces that are not marked {\tt
  IFF\_NOFAILOVER} will be brought up on the appropriate IPMP group
interface instead (see section \ref{progsockflags}).

\subsection{Address Manipulation Operations}

As with any IP interface, IPv4 and IPv6 addresses will be able to be
configured on an IPMP group interface through the following {\tt ioctl}
operations:

\begin{table}[h]
\begin{tabular}{|p{1.75in}|p{3.75in}|} \hline
{\bf {\tt ioctl}}	   &	     \textbf{meaning}\\ \thline
{\tt SIOC[GS][L]IFADDR}    & 	     Get or set IPv4 or IPv6 address \\
{\tt SIOC[GS][L]IFNETMASK} &	     Get or set IPv4 netmask \\
{\tt SIOC[GS][L]IFBRDADDR} &	     Get or set IPv4 broadcast address \\
{\tt SIOC[GS]LIFTOKEN}     &	     Get or set IPv6 address token \\
{\tt SIOC[GS]LIFSUBNET}    &	     Get or set IPv6 subnet prefix \\ \hline
\end{tabular}
\end{table}

These operations will still be able to be issued on the underlying
interfaces in a IPMP group.  However, if the {\tt IFF\_UP} flag is later
set on the address and {\tt IFF\_NOFAILOVER} is not set, the address will
be brought up on the appropriate IPMP group interface instead (see section
\ref{progsockflags}).
\subsection{Interface Parameter Operations}

All of the documented {\tt ioctl} operations to get and set interface
parameters will be usable on an IPMP group interface:
\begin{table}[h]
\begin{tabular}{|p{1.75in}|p{3.75in}|} \hline
{\bf {\tt ioctl}}        & \textbf{meaning}\\ \thline
{\tt SIOC[GS]LIFLNKINFO} & Get or set interface link information \\
{\tt SIOC[GS]LIFMTU}     & Get or set interface MTU \\
{\tt SIOC[GS]LIFMUXID}   & Get or set interface mux ID \\ \hline
\end{tabular}
\end{table}

These parameters will be used by IP when using the addresses configured on
a given IPMP group.  These parameters will also be able to be set or
retrieved on the underlying interfaces, but their values will only apply
to the test addresses configured on those interfaces. 

\subsection{ARP Table Configuration}

Solaris provides two sets of interfaces to delete, get, and set (add)
entries in the ARP table: {\tt SIOC[DGS]ARP} and {\tt SIOC[DGS]XARP}.  The
first set only allows an IP address and a hardware address to be
specified; the second set also allows an IP interface to be specified.

\subsubsection{ARP for On-Link Hosts}

The current handling of {\tt SIOC[DGS]ARP} and {\tt SIOC[DGS]XARP} for
on-link hosts is incorrect because each operation only applies to a
specific interface in an IPMP group, rather than to {\it all} interfaces
in the group.  That is, if an ARP entry is established on one interface in
an IPMP group, it cannot be used by another interface in the same group.

To fix this, in the future, on-link hosts reachable through an interface
in an IPMP group will always be associated with the interface's
corresponding IPMP interface.  That is, {\tt SIOC[DS]ARP} (or {\tt
  SIOC[DS]XARP} with no specified interface) will delete or add an ARP
entry for use by all interfaces in an IPMP group.  Further, attempts to
manipulate ARP entries on underlying interfaces via {\tt SIOC[DS]XARP}
will fail with {\tt EPERM}.  Finally, {\tt SIOCG[X]ARP} will retrieve a
given ARP entry; for {\tt SIOCGXARP}, the IPMP group interface associated
with the ARP entry will also be returned.

\subsubsection{ARP for IPMP Data Addresses}

As previously discussed, although IPMP data addresses will no longer be
hosted on physical interfaces, the IP module will ensure that each usable
IP data address in the IPMP group is associated with an underlying
physical interface in the group (section \ref{basipaddrsel}).  Thus, {\tt
  SIOCG[X]ARP} will return an arbitrary hardware address of a physical
interface in the group, and {\tt SIOCGXARP} will return the IPMP group
interface associated with the ARP entry.  This behavior will not be a
regression because the current behavior is not actually deterministic:
nothing prevents the IPMP data address from migrating to another physical
interface after the lookup operation.

Because the ARP entries associated with IPMP data addresses are
system-managed, attempting to add or delete ARP entries for IPMP data
addresses via {\tt SIOC[DS][X]ARP} will fail with {\tt EPERM}.

\subsubsection{ARP for IPMP Test Addresses}

Since IPMP test addresses remain associated with specific underlying
interfaces, an {\tt SIOCG[X]ARP} for an IPMP test address will always
return the corresponding interface's hardware address; for {\tt
  SIOCGXARP}, the associated underlying interface will also be returned.
Because the ARP entries associated with IPMP test addresses are
system-managed, attempting to add or delete ARP entries for IPMP test
addresses via {\tt SIOC[DS][X]ARP} will fail with {\tt EPERM}.

\subsubsection{Proxy ARP}

Some sites make use of proxy ARP in order to allow a host to ``pretend to
be'' another host -- making it appear to be directly reachable from the
perspective of other on-link hosts.  Currently, the fact that ARP entries
must be associated with underlying interfaces precludes proxy ARP from
working with IPMP.

In the future, proxy ARP will work transparently with IPMP.  Specifically,
when {\tt SIOCS[X]ARP} is performed, the system will check if the entry
being added has a hardware address corresponding to an interface that is
in an IPMP group, but has an IP address that does not correspond to any of
the system's usable IP addresses.  If so, the system knows that a proxy
ARP entry is being established for an interface using IPMP, and checks to
see if the interface associated with the hardware address is active.  If
so, or if there are no active interfaces in the IPMP group, the ARP entry
will be added verbatim.  Otherwise, the hardware address will be changed
to that of an active interface in the IPMP group.  From this point on, the
proxy ARP entry will be treated like an ARP entry for an IPMP data
address, and the system will adjust the entry's hardware address in order
to keep the IP address reachable (provided there is an active interface in
the group).

As with ARP entries for on-link hosts, proxy ARP entries can be deleted
with {\tt SIOCD[X]ARP} and retrieved with {\tt SIOCG[X]ARP}; for {\tt
  SIOCGXARP} the IPMP group interface  associated with the ARP entry
will also be returned.  Attempts to manipulate ARP entries on underlying
interfaces via {\tt SIOC[DS]XARP} will fail with {\tt EPERM}.

\subsection{Neighbor Discovery Configuration}

Not surprisingly, configuration of the IPv6 neighbor discovery cache
mirrors configuration of the ARP table.  Specifically,
{\tt SIOCLIFGETND}, {\tt SIOCLIFSETND}, and {\tt SIOCLIFDELND}
are used to get, set, and delete entries from the neighbor discovery
cache.  Like {\tt SIOC[GS]XARP}, all three of these {\tt ioctls}
include a network interface name to associate the entry with, and also
like ARP, the current architecture incorrectly forces operations to be
applied to a specific interface in an IPMP group.

The new handling of these operations will be analogous to {\tt
  SIOC[SG]XARP}: {\tt SIOCLIFGETND} will always return the name of the
IPMP group interface hosting the hardware address (e.g., {\tt ipmp0}), and
{\tt SIOCLIF\{DEL,SET\}ND} requests associated with underlying interfaces
will fail with {\tt EPERM}.  Similarly, because IPMP data addresses are
system-managed, {\tt SIOCLIF\{DEL,SET\}ND} requests for IPMP data
addresses will fail with {\tt EPERM}.

\subsection{Zone Configuration}

For a shared-stack zone, the {\tt SIOCGLIFZONE} and {\tt SIOCSLIFZONE}
operations will still be used to get or set the zone associated with an
address on an IPMP interface or an underlying interface in the group.
Note that the zone only controls whether the given address can be used by
the zone; as is true today, other interfaces in the group may be used to
send or receive packets.

For an exclusive-stack zone, the datalinks themselves will still be
assigned to the zone (through the private {\tt DLDIOC\_SETZID} ioctl), thus
ensuring that any IP interfaces configured atop of them (and any resulting
IPMP groups) are also in the same zone.  Additionally, each exclusive
stack will continue to have its own instance of {\tt in.mpathd}, which
will perform probe-based failure detection atop any addresses marked {\tt
  IFF\_NOFAILOVER}.

\subsection{{\tt USESRC} Manipulation}

As discussed in section \ref{basospf}, OSPF-MP and IPMP will remain
mutually exclusive for now.  As such, one will not be able to use either
an IPMP group interface or an underlying IPMP interface as the donor or
recipient interface in {\tt SIOCSLIFUSESRC} operations.  Similarly, if an
interface has been configured with {\tt SIOCSLIFUSESRC}, attempting to
place it into an IPMP group will fail.  Since OSPF-MP and IPMP are
mutually exclusive, the {\tt SIOCGLIFUSESRC} and {\tt SIOCGLIFSRCOF}
operations will trivially succeed when used with IPMP group interfaces or
underlying IPMP interfaces.  Note that if support for OSPF-MP on IPMP
group interfaces is added in the future, the restrictions on the IPMP
group interface will be removed.

\subsection{Multicast Interaction}
\label{progmulti}
From an application's perspective, as with unicast, all multicast traffic
will be associated with the IPMP interface rather than its underlying
physical interfaces.  Specifically, since all IP data addresses will be
hosted on IPMP group interfaces, the address provided in an
{\tt IP\_ADD\_MEMBERSHIP}, {\tt IP\_DROP\_MEMBERSHIP}, or
{\tt IP\_MULTICAST\_IF} socket option will be mapped to the IPMP group interface.
Similarly, the application will specify the interface index associated
with the appropriate IPMP interface when issuing the
{\tt IPV6\_JOIN\_GROUP}, {\tt IPV6\_LEAVE\_GROUP}, and
{\tt IPV6\_MULTICAST\_IF} socket options.  Additionally, the application
will specify the interface index associated with the appropriate IPMP
interface in the {\tt group\_req} and {\tt group\_source\_req}
structures associated with the {\tt MCAST\_JOIN\_GROUP},
{\tt MCAST\_LEAVE\_GROUP}, {\tt MCAST\_JOIN\_SOURCE\_GROUP},
{\tt MCAST\_LEAVE\_SOURCE\_GROUP}, {\tt MCAST\_BLOCK\_SOURCE}, and
{\tt MCAST\_UNBLOCK\_SOURCE} socket options.

When an IP interface is placed into an IPMP group, any existing multicast
memberships on that interface are torn down.  This may seem surprising,
but it is necessary to ensure consistency with the application's view that
the IP interface which they had joined on ``no longer exists''.  Moreover,
long-running applications will track the functioning set of IP interfaces
(either via routing sockets or by polling {\tt SIOCGLIFCONF}), and thus
will simply re-join their multicast group on the new IPMP interface (if
need be).  Since multicast memberships must be consistent across the
group, attempts to establish new multicast memberships on specific
underlying interfaces will fail with {\tt EINVAL}.

As per section \ref{basmulti}, the IP module will nominate one active
interface in the IPMP group to send and receive multicast traffic, and to
apply any needed multicast filtering.  If that interface subsequently
fails, the multicast membership and filtering will be moved to another
active interface in the group, if one exists.  When sending multicast
packets, the nominated interface's multicast mapping (visible in section
\ref{admnetstat}) is used to determine the destination link-layer
multicast address.

\subsection{Routing Table Bypass Socket Options}

The {\tt IP\_NEXTHOP}, {\tt IPV6\_NEXTHOP}, and {\tt SO\_DONTROUTE} socket
options all enable the routing table to be bypassed when sending a packet.
In a non-IPMP configuration, the on-link destination specified when
enabling either {\tt IP\_NEXTHOP} or {\tt IPV6\_NEXTHOP} -- or when
sending a packet after enabling {\tt SO\_DONTROUTE} -- also indirectly
specifies an outgoing interface.  In the new model, this will also
indirectly specify an outgoing interface, though that interface may be an
IPMP group interface, which may use any active underlying interface to
actually transmit on.  Since no applications other than {\tt in.mpathd}
should be generating IPMP test traffic, {\tt IP\_NEXTHOP}, {\tt
  IPV6\_NEXTHOP}, and {\tt SO\_DONTROUTE} will not necessarily be honored
if the specified address is on a network that is only accessible through
the system's IPMP test addresses.

\subsection{IPMP {\tt ioctl} Operations}
\label{progioctl}
To enable applications such as {\tt in.mpathd} and {\tt ifconfig} to set
or get the kernel's IPMP state, there are a collection of undocumented
IPMP-related {\tt ioctl} operations.  Specifically, the following private
IPMP-related {\tt ioctl} operations are currently supported:

\begin{table}[h]
\begin{tabular}{|p{1.75in}|p{3.75in}|} \hline
{\bf {\tt ioctl}}        & \textbf{meaning}\\ \thline
{\tt SIOCLIFFAILOVER}   & Perform an interface failover \\
{\tt SIOCLIFFAILBACK}	& Perform an interface failback \\
{\tt SIOCSIPMPFAILBACK} & Enable or disable {\tt FAILBACK=no} \\
{\tt SIOC[GS]LIFOINDEX}	& Get or set interface's original index \\
{\tt SIOC[GS]LIFGROUPNAME} & Get or set interface's IPMP group \\ \hline
\end{tabular}
\end{table}

In the new model, most of these ioctls are now unnecessary.  Specifically,
because the kernel now manages address distribution, the {\tt
  SIOCLIFFAILOVER} and {\tt SIOCLIFFAILBACK} operations no longer make
sense and will be removed.  Further, as discussed in section
\ref{basfailbackno}, {\tt SIOCSIPMPFAILBACK} will no longer be necessary
and will be removed (along with the {\tt lifr\_movetoindex} member of the
{\tt struct lifreq}).  Similarly, as discussed in section \ref{basdr}, the
{\tt SIOCGLIFOINDEX} and {\tt SIOCSLIFOINDEX} operations will also be
rendered unnecessary and will be removed.

{\tt SIOCGLIFGROUPNAME} and {\tt SIOCSLIFGROUPNAME} will remain.  However,
a number of new requirements will need to be met for {\tt
  SIOCSLIFGROUPNAME} to successfully place an interface in a new group
(e.g., it must not be in a group yet, and the IPMP interface for the group
must exist), which will simplify a number of IPMP codepaths in the kernel.
Since {\tt SIOCSLIFGROUPNAME} is private to the IPMP subsystem, these
changes pose no compatibility problems\footnote{With the exception of the
  Sun Cluster use covered in section \ref{bascluster}.}.

The new model will also necessitate two new private IPMP {\tt ioctl}
operations:

\begin{table}[h]
\begin{tabular}{|p{1.75in}|p{3.75in}|} \hline
{\bf {\tt ioctl}}        & \textbf{meaning}\\ \thline
{\tt SIOCGLIFBINDING}   & Get inbound interface binding for address \\
{\tt SIOCGLIFGROUPINFO}  & Get IPMP group information \\ \hline
\end{tabular}
\end{table}

The {\tt SIOCGLIFBINDING} operation will be needed for {\tt ipmpstat -a}
to report the inbound interface binding for a given address (since those
bindings will be under kernel control).  It will expect a {\tt struct
  lifreq} with the {\tt lifr\_name} set to the logical interface of the
address to look up, and return the result in a new {\tt lifr\_binding}
member, which will be part of the existing {\tt union lifr\_lifru}.

The {\tt SIOCGLIFGROUPINFO} operation will be needed by {\tt in.mpathd}
and {\tt ifconfig} for a variety of tasks, including quickly mapping an
IPMP group to its IPMP interface, obtaining the nominated multicast and
broadcast interfaces for a group, and determining whether a given IPMP
group is currently in-use.  It will expect a {\tt struct lifgroupinfo},
which will be a new structure with contents private to the IPMP subsystem.

\subsection{MIB II Interaction}

Network interface names and indices are also exposed through several
tables in the Networking MIB.  While access to the Networking MIB is not
currently documented, numerous popular third-party applications have
discovered how to access it in order to obtain information about the
networking stack.  In addition, internal utilities such as {\tt netstat}
make extensive use of it to display information such as the routing and
ARP tables.

The following MIB II structures have fields which expose network interface
names or indices:

\begin{table}[h!]
\begin{tabular}{|p{1.75in}|p{3.75in}|} \hline
{\bf MIB structure name} & \textbf{structure purpose}\\ \thline
{\tt mib2\_ipAddrEntry}&          IPv4 address information \\
{\tt mib2\_ipv6AddrEntry}&        IPv6 address information \\ \hline
{\tt mib2\_ipRouteEntry}&         IPv4 routing table entry \\ 
{\tt mib2\_ipv6RouteEntry}&       IPv6 routing table entry \\ \hline
{\tt mib2\_ipNetToMediaEntry}&    ARP table entry \\
{\tt mib2\_ipv6NetToMediaEntry}&  ND table entry \\ \hline
{\tt mib2\_ipv6IFStatsEntry}&     IPv6 interface statistics \\
{\tt mib2\_ipv6IfIcmpEntry}&      IPv6 ICMPv6 interface statistics \\ \hline
{\tt mib2\_tcp6ConnEntry}&        TCP over IPv6 connection information \\
{\tt mib2\_udp6Entry}&            UDP over IPv6 listener information \\ \hline
{\tt ip\_member}&                 IPv4 multicast membership \\
{\tt ipv6\_member}&               IPv6 multicast membership \\ \hline
{\tt ip\_grpsrc}&                 IPv4 multicast source filtering \\
{\tt ipv6\_grpsrc}&               IPv6 multicast source filtering \\ \hline
\end{tabular}
\end{table}

In almost all of the above cases, the network interface names and indices
returned through these structures will refer to the IPMP group interface,
rather than the underlying interface.  The only exceptions will be for
addresses, routing table entries and hardware address mappings that are
explicitly tied to an {\tt IFF\_NOFAILOVER} address, which will use the
interface name or index of the underlying interface the address is hosted
on.  To prevent IPMP-unaware applications from tripping over routing table
entries only used for test traffic, such entries will {\it not} be
returned unless an application specifies a new synthetic MIB II level
({\tt EXPER\_IP\_AND\_TESTHIDDEN}) when issuing the report request.  Since
such information is chiefly useful only to {\tt in.mpathd}, we do not
anticipate unbundled applications will need to make use of this synthetic
level.

\subsection{Kstats}

As previously discussed, the IPMP interface will have kstats, which can be
retrieved using the documented {\tt kstat(3KSTAT)} interfaces.  However,
many networking kstats are hardware-specific and thus do not apply to the
IPMP group interface.  See section \ref{admkstat} for a list of the kstats
that will be supported.

\subsection{DLPI Interaction}

As discussed in section \ref{basdl}, because each data-link in an IPMP
group maintains its own identity, it is not semantically feasible to
provide data-link (DLPI) access to the IPMP interface as a whole.  In
particular, one would need to make significant changes to the core DLPI
model to allow each data-link to be individually accessed (e.g., by
allowing a DLPI consumer to attach to a particular data-link, or to send a
packet out a particular data-link using a particular hardware address), or
make incompatible changes to the IPMP model to make all interfaces in a
group appear homogeneous (e.g., a single hardware address a la 802.3ad).

As such, {\bf DLPI access to an IPMP group is not planned}.

However, the Solaris IP stack expects all IP interfaces (other than
loopback) to terminate in a DLPI device, on top of which the IP kernel
module is pushed.  Thus, in order to satisfy this architecture, a stub
driver called {\tt dlpistub} will be implemented and be visible in the
filesystem as {\tt /dev/ipmpstub}.  This driver will be an enhanced
version of the current {\tt vni(7D)} stub driver that is used for OSPF-MP
support, implementing a few core STREAMS and DLPI messages that IP
expects.  As such, {\tt dlpistub} will also be able to export itself as
{\tt /dev/vni} for {\tt vni(7D)} support\footnote{{\tt /dev/vni} and {\tt
    /dev/ipmpstub} will be identical except for the DLPI type they
  export.}, and the existing {\tt vni} DLPI stub driver will be able to be
removed.  Note that {\tt /dev/ipmpstub} will be provided strictly for
implementation convenience.  It {\bf will not be able to send or receive
  packets, nor will it be documented for public use}.  The name {\tt
  /dev/ipmp} was intentionally avoided to prevent administrators from
being misled into believing that {\tt snoop -d ipmp0} will work.

\subsection{{\tt libipmp}}
\label{proglibipmp}
The {\tt libipmp} library currently provides an undocumented {\it query}
API used to snapshot and query the IPMP subsystem, as originally described
in PSARC/2002/615.  It was initially created for use by Sun Cluster and
the IPMP test suite, but was never used by Sun Cluster.  In the new
architecture, the query API provides an elegant foundation for
implementing {\tt ipmpstat}, though the API will need to be revised to
accommodate the proposed changes to the IPMP subsystem.  Because it is
currently only used by the IPMP test suite (which will itself undergo
extensive changes), when necessary, elegance of the resulting API will be
prioritized over backward compatibility.

In addition, {\tt libipmp} provides a logical place to house an additional
{\it administrative} API that can be used to control the IPMP subsystem.
This API will be used by {\tt ifconfig}, {\tt if\_mpadm}, and {\tt
  ip\_rcm}, which will allow large chunks of duplicated code to be removed
from those applications and modules, and will ensure that all
administrative IPMP operations are handled consistently.  With the
introduction of the administrative API, all communication with {\tt
  in.mpathd} will be performed through {\tt libipmp}, which enables the
previously-exposed IPC mechanism used between {\tt in.mpathd} and {\tt
  libipmp} to become private to the implementation.

A separate specification details the complete {\tt libipmp} API; this
document only covers the high-level changes and enhancements.

\subsubsection{{\tt libipmp} Query API}

The existing set of functions (declared in {\tt <ipmp\_query.h>}) will
remain unchanged:

\begin{table}[h]
\begin{tabular}{|p{1.75in}|p{3.75in}|} \hline
{\bf {\tt routine}}        & \textbf{purpose}  \\ \thline
{\tt ipmp\_setqcontext()}    & Set the IPMP query context (live or snapshot) \\
{\tt ipmp\_getgrouplist()}   & Get list of interfaces monitored by {\tt in.mpathd} \\
{\tt ipmp\_freegrouplist()}  & Free list returned from {\tt
  ipmp\_getgrouplist()} \\
{\tt ipmp\_getgroupinfo()}   & Get IPMP group information \\
{\tt ipmp\_freegroupinfo()}  & Free information returned from {\tt ipmp\_getgroupinfo()} \\
{\tt ipmp\_getifinfo()}      & Get IPMP-related interface information \\
{\tt ipmp\_freeifinfo()}     & Free information returned from {\tt ipmp\_getifinfo()} \\ \hline
\end{tabular}
\end{table}

However, the {\tt ipmp\_groupinfo\_t} returned by {\tt
  ipmp\_getgroupinfo()} will be extended to also include the group's
failure detection time, IPMP group interface name, data addresses, and
nominated broadcast and multicast interface names.  Similarly, the {\tt
  ipmp\_ifinfo\_t} structure returned by {\tt ipmp\_getifinfo()} will be
extended to also include the interface's test address (if any), probe
target mode, probe targets (if any), and probe- and link-based failure
detection status.

In addition, a new {\tt ipmp\_getaddrinfo()} function will be added to
provide IPMP data address information via a new {\tt ipmp\_addrinfo\_t}
structure.  This structure will contain the address's IPMP group, state
(either up or down), and inbound underlying interface binding (if any).
As with the existing query functions, {\tt ipmp\_getaddrinfo()} will
return either {\tt IPMP\_SUCCESS}, or an IPMP error code on failure.  A
new {\tt IPMP\_EUNKADDR} error code will be returned if the requested
address is not an IPMP data address.  Also mirroring the existing
functions, a new {\tt ipmp\_freeaddrinfo()} function will free an {\tt
  ipmp\_addrinfo\_t} structure.

\begin{table}[h]
\begin{tabular}{|p{1.75in}|p{3.75in}|} \hline
{\bf {\tt routine}}        & \textbf{purpose}\\ \thline
{\tt ipmp\_getaddrinfo()}   & Get IPMP data address information \\
{\tt ipmp\_freeaddrinfo()}  & Free information returned from {\tt ipmp\_getaddrinfo()} \\ \hline
\end{tabular}
\end{table}

\subsubsection{{\tt libipmp}  Administrative API}

The proposed administrative API will initially consist of just three
undocumented functions, which will be declared in a new {\tt
  <ipmp\_admin.h>} header file:

\begin{table}[h]
\begin{tabular}{|p{1.75in}|p{3.75in}|} \hline
{\bf {\tt routine}}         & \textbf{purpose}\\ \thline
{\tt ipmp\_ping\_daemon()}  & Check if {\tt in.mpathd} is operational \\
{\tt ipmp\_offline()}       & Offline an underlying interface \\
{\tt ipmp\_undo\_offline()} & Undo a previous offline of an underlying
interface \\  \hline
\end{tabular}
\end{table}

The {\tt ipmp\_ping\_daemon()} routine will enable an application to
easily check if {\tt in.mpathd} is running.  The {\tt ipmp\_offline()}
routine will attempt to offline the specified IP interface; the caller
must also specify the minimum number of usable interfaces in the group
that may remain after the offline operation.  If the interface is not
under IPMP control, or if the offline would lower the number usable
interfaces below the minimum, the operation will fail.  Finally, the {\tt
  ipmp\_undo\_offline()} routine allows a previously-offlined IP interface
to be brought back online.  If the interface was not previously offlined,
or if the interface cannot be brought back online (e.g., because its
hardware address is not unique across the group), the operation will fail.

\subsection{ IPMP Asynchronous Events Enhancements}
\label{progasync}
In addition to the query API described above, there's also an undocumented
set of sysevents that enable IPMP subsystem state changes to be observed,
as originally described in PSARC/2002/137.  As with the query API, it was
created for use by Sun Cluster and the IPMP test suite, but was never used
by Sun Cluster.

With the rearchitecture, the existing set of IPMP sysevents will be
enhanced slightly to account for new states (such as the ``degraded''
group state) and new configurations (such as an empty group), but will be
otherwise unchanged.  In addition, one new IPMP sysevent, {\tt
  ESC\_IPMP\_PROBE\_STATE}, will be introduced to provide state change
notifications for each IPMP probe packet, which will provide the
underlying mechanism for {\tt ipmpstat -p}.  To ensure that {\tt
  syseventd} and other sysevent consumers do not become saturated with
{\tt ESC\_IPMP\_PROBE\_STATE} sysevents on large IPMP configurations, {\tt
  in.mpathd} will switch from using the legacy sysevent mechanism to the
``General-Purpose Event Channels'' (GPEC) sysevent mechanism\footnote{In
  particular, the GPEC API provides fine-grained event channels which
  do not flow through {\tt syseventd}; see PSARC/2002/321 for details.}.

In particular, a new {\tt com.sun:ipmp:events} event channel will be
introduced to carry IPMP sysevents.  Because of these changes, the
existing {\tt IPMP\_EVENT\_VERSION} field of each IPMP sysevent will be
increased to the value {\tt 2}.

\subsection{Miscellaneous Library API Enhancements}

\subsubsection{{\tt if\_nametoindex(3SOCKET)} and {\tt if\_indextoname(3SOCKET)}}

Since the IPMP group interface is an IP interface, the existing {\tt
  if\_nametoindex(3SOCKET)} and {\tt if\_indextoname(3SOCKET)} routines
will be able to map an IPMP group interface name to its interface index,
and vice-versa.  More subtly, the APIs will also continue to work for
underlying interfaces in an IPMP group (rather than fail with {\tt
  ENXIO}), on the grounds that the caller has already discovered the
interface name or index (e.g., through an {\tt SIOCGLIFCONF} with {\tt
  LIFC\_UNDER\_IPMP}).

\subsubsection{{\tt if\_nameindex(3SOCKET)}}

The rarely-used and oddly-named {\tt if\_nameindex(3SOCKET)} routine
(which actually retrieves the system's IP interface names and indices)
will retrieve IPMP group interfaces, but to avoid confusing IPMP-unaware
applications, it will not retrieve underlying interfaces in an IPMP group.

\subsubsection{{\tt ifaddrlist()}}

The undocumented {\tt libinetutil} library currently has an ``enhanced''
(and incompatible) version of BSD's {\tt ifaddrlist()} routine, which
retrieves a list of IPv4 or IPv6 addresses currently configured on the
system, along with their associated IP interface name, flags, and index.
Currently, {\tt ifaddrlist()} builds the list of addresses by calling {\tt
  SIOCGLIFCONF} with {\tt lifc\_flags} set to zero, which means that {\tt
  IFF\_NOXMIT}, {\tt IFF\_TEMPORARY}, and all-zones addresses are never
retrieved.  Similarly, without modification, interfaces under IPMP would
never be retrieved in the future.

To address this limitation, {\tt ifaddrlist()} will be further enhanced to
take an additional {\tt lifc\_flags} argument.  Passing zero for this
argument will match current behavior; passing other values (e.g.,
{\tt LIFC\_UNDER\_IPMP}) will allow addresses tied to various ``hidden''
IP interfaces to be retrieved.

\subsubsection{{\tt ifaddrlistx()} and {\tt ifaddrlistx\_free()}}

An additional undocumented {\tt ifaddrlistx()} convenience function will
be added to {\tt libinetutil}.  This routine will be similar to {\tt
  ifaddrlist()}, but will only retrieve addresses for a specific interface
name that have a specific set of address flags set and clear.  In
addition, it will return a linked-list of {\tt ifaddrlistx\_t} structures
(rather than an array) so that the {\tt ifaddrlistx\_t} can grow in the
future without affecting binary compatibility.  An accompanying {\tt
  ifaddrlistx\_free()} function will be provided to free the linked-list.
As with {\tt if\_nametoindex()}, {\tt ifaddrlistx()} will succeed for
underlying interfaces in an IPMP group on the grounds that the caller has
already discovered the interface name.  A separate specification covers
the details of these interfaces; they are mentioned here only for
completeness.

\subsubsection{{\tt sockaddrcmp()}}

An additional undocumented {\tt sockaddrcmp()} convenience function will
be added to {\tt libinetutil}.  This function takes two pointers to {\tt
  struct sockaddr\_storage}, and returns {\tt B\_TRUE} if they contain the
same IPv4 or IPv6 address.

\pagebreak
\section{Summary}

In closing, we briefly revisit our original set of requirements, and
verify that the proposed model satisfies them. Taking the list from the
top:
\begin{enumerate}
\item IP addresses in an IPMP group must never migrate as part of failure, repair, offline, and undo-offline operations.
\item IP addresses in an IPMP group must be administered using the same tools and in the same manner as IP addresses not placed in an IPMP group. As a corollary, address autoconfiguration through DHCP must work.
\setcounter{enumii}{\value{enumi}}
\end{enumerate}

Since all data address will be hosted on IPMP interfaces, address
migration never occurs as a result of failure, repair, offline, and
undo-offline operations.  Further, because the data addresses exist on
stable logical interfaces, it is possible to administer them just like any
other IP address.  Enhancements made to {\tt dhcpagent} and {\tt in.ndpd}
will allow address autoconfiguration to work even though the addresses are
not associated with any specific hardware.

\begin{enumerate}
\setcounter{enumi}{\value{enumii}}
\item The IPMP group's core configuration and current state must be able
  to be easily and concisely obtained.
\setcounter{enumii}{\value{enumi}}
\end{enumerate}

The new {\tt ipmpstat} utility will allow the administrator to examine the
state and configuration of the IPMP subsystem on a variety of levels.
Specifically, IPMP groups, IPMP data addresses, IPMP interfaces, IPMP
probe targets, and IPMP-related probes can all be observed in real-time,
providing insight into a previously-opaque subsystem.

\begin{enumerate}
\setcounter{enumi}{\value{enumii}}
\item The IPMP group as a whole must be observable through packet
  monitoring tools such as {\tt snoop(1M)}.  It must also still be
  possible to observe individual interfaces as well.
\item The IPMP group as a whole must be able to administered as a whole --
  e.g., specified directly to {\tt route(1M)}, and visible directly
  through {\tt netstat(1M)}.  The Clearview network interface
  requirements must also be met.
\setcounter{enumii}{\value{enumi}}
\end{enumerate}

By modeling IPMP groups as IP interfaces, Clearview's IP Observability
component can be leveraged to provide packet monitoring of the IPMP group
as a whole.  Further, representing IPMP groups as interfaces will enable
IPMP groups to be specified directly to {\tt route(1M)}, and to be visible
through {\tt netstat(1M)}, along with a variety of other tools.  In order
to meet Clearview's network interface requirements, a new {\tt ipmp
  ifconfig} subcommand will allow the administrator to name an IPMP group
interface at creation time.

\begin{enumerate} 
\setcounter{enumi}{\value{enumii}}
\item  IPMP test addresses must not be visible to applications unless explicitly requested.
\setcounter{enumii}{\value{enumi}}
\end{enumerate}

By making a collection of changes to {\tt SIOCGLIFCONF} and routing
sockets semantics, networking applications will be shielded from test
addresses, unless they explicitly request to see them.  Since the revised
semantics will become the default, all applications will automatically
benefit with no coding changes.

\begin{enumerate}
\setcounter{enumi}{\value{enumii}}
\item If possible, address migration must be obviated (to prevent complex migration code in the kernel).
\setcounter{enumii}{\value{enumi}}
\end{enumerate}

Where necessary, migration of an address will be modeled as the removal of the address on one interface followed by the insertion of the address on another interface, allowing the current complex migration code to be removed.

\begin{enumerate}
\setcounter{enumi}{\value{enumii}}
\item The documented IPMP administrative commands ({\tt if\_mpadm},
  {\tt ifconfig}, and {\tt cfgadm}) must continue work as before.
\item The documented IPMP semantics, e.g., for failure detection, repair
  detection, outbound interface selection and source address selection
  must continue to operate as before.
\item The documented IPMP configuration files
  ({\tt /etc/hostname.{\it if}} files) must continue to operate as
  before.
\setcounter{enumii}{\value{enumi}}
\end{enumerate}

As outlined throughout this proposal, the documented behavior of all
administrative commands, semantics, and configuration files will be
preserved.  However, a new administrative model centered around the IPMP
group interface is also proposed, and administrators will be encouraged to
move to the new administrative model so that they can take full advantage
of the new IPMP architecture.

\pagebreak
\setcounter{secnumdepth}{-1}
\section{Glossary of IPMP Terminology}

\begin{description}

\item[Active Interface:]

  An underlying interface that is currently being used by the system to
  send or receive data traffic.  An interface is active if it has at least
  one {\tt UP} address, and the {\tt FAILED}, {\tt INACTIVE}, and {\tt
    OFFLINE} flags are {\it not} set.  Compare: usable interface,
  functioning interface, {\tt INACTIVE} interface.

\item[Address Migration:]

  The act of moving an address from one network interface to another
  network interface.  Currently, address migration frequently occurs, most
  often as part of failover or failback.  In the new model, migration of
  an {\tt UP} address will never occur, drastically simplifying the
  implementation.

\item[Data Address:]

  An IP address which can be used as the source or destination address for
  data.  Data addresses are part of an IPMP group and can be used to send
  and receive traffic on {\it any} interface in the group.  Moreover, the
  set of data addresses in an IPMP group can continue to be used as long
  as one interface in the group is active.  Currently, data addresses
  are hosted on the underlying interfaces of an IPMP group.  In the new
  model, data addresses will be hosted on the IPMP interface.  Compare:
  Test Address.

\item[{\tt DEPRECATED} Address:]

  An IP address which should not be used as the source address for data.
  Typically, IPMP Test Addresses are {\tt DEPRECATED}, but any addres
  may be marked {\tt DEPRECATED} to discourage it from being used as a
  source address.

\item[Explicit IPMP Interface Creation:]

  In the new model, the act of explicitly defining an IPMP interface by
  using the {\tt ipmp} subcommand of {\tt ifconfig}.  Explicit IPMP
  interface creation is preferred since it allows the IPMP interface name
  and IPMP group name to be set by the administrator.  Compare: Implicit
  IPMP Interface Creation.

\item[Failback:]

  Currently, the act of undoing a previous failover operation, once the
  previously-failed interface has been repaired.  The concept of failback
  will not exist in the new model.

\item[{\tt FAILBACK=no} Mode:]

  A mode which minimizes rebinding of incoming addresses to interfaces by
  avoiding redistribution during interface repair.  This mode is
  currently broken, but will be fixed in the new model.  Specifically,
  when an interface repair is detected, {\tt FAILED} will be cleared, but
  if another interface in the group is functioning,
  {\tt INACTIVE} will be set, preventing its use.  However, if another
  interface in the IPMP group fails, the {\tt INACTIVE} interface will be
  eligible to take over for the failed interface.  While the concept of
  failback will not exist in the new model, the name of this mode will be
  preserved for administrative compatibility.

\item[{\tt FAILED} Interface:]

  An interface which {\tt in.mpathd} has determined to be malfunctioning
  The {\tt FAILED} flag will be set on any {\tt FAILED} interface.
  Opposite: functioning interface.

\item[Failover:]

  Currently, the act of migrating the data addresses on a failed interface
  to another functioning interface in an IPMP group.  In the new model, all
  data addresses will be hosted on the IPMP interface, and thus be
  unaffected by interface failure.  Thus, the concept of failover will not
  exist in the new model.

\item[Failure Detection:]

  The act of detecting that a physical interface in the group has failed.
  Two forms of failure detection are implemented: link-based failure
  detection, and probe-based failure detection.

\item[Functioning Interface:]

  An interface which {\tt in.mpathd} has determined to be healthy.  This
  includes both active interfaces and {\tt INACTIVE} interfaces.  All
  functioning interfaces will have the {\tt FAILED} flag clear.  Opposite:
  {\tt FAILED} interface.

\item[{\tt INACTIVE} Interface:]

  A functioning interface which must not be used according to
  administrative policy.  All {\tt INACTIVE} interfaces will be have the
  {\tt INACTIVE} flag set.  Compare: active interface, usable interface.
  
\item[Implicit IPMP Interface Creation:]

  In the new model, the act of implicitly defining an IPMP interface by
  placing an underlying interface in an IPMP group that does not yet exist
  using {\tt ifconfig}.  Implicit IPMP interface creation is provided
  for backward compatibility, and as such does not provide the ability to
  set the IPMP interface name or IPMP group name. Compare: Explicit IPMP
  Interface Creation.

\item[Inbound Load-Spreading:]

  The act of spreading inbound traffic across the set of interfaces in an
  IPMP group.  Inbound load-spreading cannot be controlled directly with
  IPMP, but is indirectly manipulated by the source address selection
  algorithm.

\item[IPMP Anonymous Group Support:]

  An IPMP feature enabled by setting \\ {\tt
    TRACK\_INTERFACES\_ONLY\_WITH\_GROUPS} to {\tt no} in {\tt
    /etc/default/mpathd}. When this feature is enabled, {\tt in.mpathd}
  tracks the health of all network interfaces in the system, regardless of
  whether they belong to an IPMP group.  However, since the interfaces are
  not actually in an IPMP group, the addresses on these interfaces will
  not be available upon interface failure.

\item[IPMP Group:]

  A set of network interfaces that are treated as interchangeable by the
  system in order to improve network availability and utilization.  Each
  IPMP Group has a set of data addresses that the system may associate
  with any set of active interfaces in the group in order to maintain
  network availability and improve network utilization.  The administrator
  can select which interfaces to place into an IPMP group, but all
  interfaces in the same group must share a common set of properties, such
  as being attached to the same link and configured with the same set of
  protocols (e.g., IPv4 and IPv6).

\item[IPMP Group Interface:]

  See: IPMP Interface.

\item[IPMP Group Name:]

  The name of an IPMP group, as assigned with the {\tt ifconfig group}
  subcommand.  All underlying interfaces with the same IPMP group name are
  defined to be part of the same IPMP group.  In the new model, IPMP group
  names will be deemphasized in favor of IPMP interface names, and
  administrators will be encouraged to use the same name for both.

\item[IPMP Interface:]

  In the new model, the  IP interface that represents a given IPMP
  group, any or all of its underlying interfaces, and all of its data
  addresses.  The IPMP interface will become the core administrative
  abstraction for an IPMP group, and will be used in routing tables, ARP
  tables, firewall rules, and so forth.

\item[IPMP Interface Name:]

  In the new model, the name of an IPMP interface.  By convention, this
  document uses the naming convention of {\tt ipmp\emph{N}}, which will
  also be used by the system via implicit IPMP interface creation.
  However, the administrator will be able to choose any name by using
  explicit IPMP interface creation.

\item[IPMP Singleton:]

  An IPMP group containing only a single IP interface.  IPMP Singleton is
  typically used by Sun Cluster, which in turn provides higher level
  failover. Traditionally, IPMP Singleton has also allowed a data address
  to also act as a test address, but this incompatible with the new model
  and will no longer be supported.

\item[Link-Based Failure Detection:]

  A passive form of failure detection, in which the link status of the
  network card (as reported by the network card's driver) is monitored in
  order to determine an interface's health.  Link-based failure detection
  only tests whether the link is up, and is not supported by all network
  card drivers.  However, it requires no explicit configuration and
  provides instantaneous detection of link failures.  Compare: Probe-Based
  Failure Detection.

\item[Load-Spreading:]

  The act of spreading inbound or outbound traffic over a set of
  interfaces.  Unlike load balancing, there is no guarantee that the load
  will be {\it evenly} distributed.

\item[{\tt NOFAILOVER} Address:]

  An address that is associated with an underlying interface and thus will
  not remain available if the underlying interface fails.  All
  {\tt NOFAILOVER} addresses will have the {\tt NOFAILOVER} flag set
  on them.  IPMP test addresses {\it must} be {\tt NOFAILOVER}, and
  IPMP data addresses {\it must never} be {\tt NOFAILOVER}.  Although the
  concept of failover will not exist in the new model, the name {\tt
    NOFAILOVER} will remain for administrative compatibility.

\item[{\tt OFFLINE} Interface:]

  An interface which is unavailable for system use, usually in preparation
  for being removed from the running system.  All {\tt OFFLINE} interfaces
  will have the {\tt OFFLINE} flag set.

\item[Offline Operation:]

  The act of making an interface unavailable for system use, usually in
  preparation for removing it from the running system. The {\tt
    if\_mpadm(1M)} command can be used to offline an interface.

\item[Outbound Load-Spreading:]

  The act of spreading outbound traffic across the set of interfaces in an
  IPMP group.  Outbound load-spreading is performed on a per-destination
  basis by the IP module, and adjusted as necessary based on the health
  and members of the interfaces in the IPMP group.

\item[Physical Interface:]

  See: Underlying Interface.

\item[Probe:]

  An ICMP packet, similar to the ones used by {\tt ping(1M)}, used to test
  the send and receive paths of a given interface.  Probe packets will be
  sent by {\tt in.mpathd} if probe-based failure detection is enabled.  A
  probe packet will use an IPMP test address as its source address.

\item[Probe-Based Failure Detection:]

  An active form of failure detection, in which probes are exchanged with
  probe targets in order to determine an interface's health.  Probe-based
  failure detection tests the entire send and receive path of each
  interface it is enabled on, but requires the administrator to explicitly
  configure each interface with a test address.  Compare: Link-Based
  Failure Detection.

\item[Probe Target:]

  A system on the same link as an interface in an IPMP group, selected by
  {\tt in.mpathd} to help test the health of a given interface using
  probe-based failure detection.  The probe target may be {\it any} host
  on the link capable of sending and receiving ICMP probes.  Probe targets
  are usually routers, and several probe targets are usually used in order
  to insulate the failure detection logic from failures of the probe
  targets themselves.

\item[Source Address Selection:]

  The act of selecting a data address in the IPMP group as the source
  address for a particular packet.  Source address selection is performed
  by the system whenever the application has not specifically selected a
  source address to use.  Since each data address is associated with only
  one hardware address, source address selection indirectly controls
  inbound load-spreading.

\item[{\tt STANDBY} Interface:]

  An interface which has been administratively configured to be used only
  when another interface in the group has failed.  All {\tt STANDBY}
  interfaces will have the {\tt STANDBY} flag set.

\item[Test Address:]

  An IP address which {\it must} be used as the source or destination
  address for probes, and {\it must not} be used as a source or
  destination address for data traffic.  Test addresses are associated
  with an underlying interface, and are marked {\tt NOFAILOVER} so that
  they remain on the underlying interface even if it fails, in order to
  facilitate repair detection.  Since test addresses will not be available
  upon interface failure, all test addresses {\it must} be marked
  {\tt DEPRECATED} to keep the system from using them as a source
  addresses for data packets.
  
\item[Underlying Interface:]

  An IP interface that is part of an IPMP group and directly associated
  with an actual network device.  For instance, if {\tt ce0} and
  {\tt ce1} are placed into IPMP group {\tt ipmp0}, then they
  comprise the underlying interfaces of {\tt ipmp0}.  Currently, IPMP
  groups consist solely of underlying interfaces.  However, in the new
  model, these interfaces will {\it underly} the IPMP interface
  (e.g., {\tt ipmp0}) that represents the group, hence the name.

\item[Undo-Offline Operation:]

  The act of enabling a previously-offlined interface, so that it is again
  available for system use. The {\tt if\_mpadm(1M)} command can be used to
  perform an undo-offline operation.

\item[Usable Interface:]

  In the new model, an underlying interface which could be used to send or
  receive data traffic.  While all active interfaces are usable, not all
  usable interfaces are active (e.g., {\tt INACTIVE} interfaces).  An
  interface is usable if it has at least one {\tt UP} address, and has
  does not have {\tt FAILED} or {\tt OFFLINE} set.  Compare: active
  interface, {\tt INACTIVE} interface.  Opposite: unusable interface.

\item[Unusable Interface:]

  An interface is unusable if it has no {\tt UP} addresses, or has {\tt
    FAILED} or {\tt OFFLINE} set.  Opposite: usable interface.

\item[{\tt UP} Address:]

  An address which has been made administratively available to the system
  by setting its {\tt UP} flag.  An address which is not {\tt UP}
  will be treated as not belonging to the system, and thus will never be
  considered during source address selection.

\end{description}

\end{document}
