SCM Build Management Scenarios

The purpose of these scenarios is to provide a way for different build systems and SCM systems to interoperate. This interoperation goes both ways; we want an SCM system to be able to initiate a build by invoking a service provided by a 'foreign' build tool, and we want that build tool to be able to exchange information with the SCM system, and control some actions in that SCM system. Specifically, we want an application to be able to:

• Allow a user to select some SCM configuration
• Produce a file system image (view, work area, projection, etc.), of that configuration
• Select a build tool or service and initiate a build of the selected configuration

SamitMehta The above sounds more like a scenario where the service would have to be provided by the Build tool. Is this also in scope?

• Monitor the build progress

SamitMehta The above sounds more like a scenario where the service would have to be provided by the Build tool. Is this also in scope?

• See the build results

SamitMehta The above sounds more like a scenario where the service would have to be provided by the Build tool. Is this also in scope?

• Check the build results into SCM
• Produce a baseline, apply a tag or label, etc., to the build results

These requirements seem to indicate the need for at least the following services:

Services offered by the SCM system

• Discover available SCM services
• Select a configuration
  _NickCrossley This needs a lot of elaboration; the list of possible configurations might be very large, so the user needs ways to query for a configuration, or to select from a subset - for example, the current configurations on a branch, the configurations at some point in time, etc._
  _SamitMehta Also, not all SCM systems provide similar set of configurations. Would be helpful to take a look at the most commonly asked configurations for builds across various tools.
• Project a given configuration to the file system
  _NickCrossley Again, this needs elaboration. What options and limitations are there in file system visibility, cross-platform support, method of file system projection (actual copied vs. links, etc.), how are component dependencies represented, etc. Note that some options might be determined by the combined capabilities of both the SCM and build systems, not just one or the other - for example, some SCM systems might prefer to represent subcomponent dependencies using symbolic links on UNIX or Linux, but some build systems might not support symbolic links._
  _SamitMehta Nick, I believe that the issue you are raising is that once this service is called, the resulting "projection" onto the file system may not be consistently implemented across the various SCM systems. One possible outcome could be that the response to the service can provide clues to REST client tool about the "projection" - e.g. what dependencies exist between the components.
  _SamitMehta Something else that may be challenging is that most SCM tools provide client tools to "project" a configuration to the file system. Some of the SCM tools control these "projections" - e.g. permissions/ownership/dates, storage, track changes even in the projection etc. Some tools have built-in capabilities to optimize how these "projections" are updated with changes. Would these tools need to provide a different/new mechanism in support of this service? Or, does the service just invoke the built-in capabilities and wouldn't the REST API then have to be supported by the SCM client tool?
• Report on differences between two configurations
  NickCrossley Or two versions of a configuration, or the state of a configuration at two different points in time, etc. - this needs defining in more depth. The results would indicate new, modified, and removed files or versions thereof.
  SamitMehta The usual goals of such reports in organization is to show what's changed since last build of the same stream of development, and sometimes a complete report on what's been done since the beginning of the stream of development. It would be helpful if the service provides support for the semantic meanings of previous build configuration and beginning of the stream of development. The service provider's implementation would need to handle the different representations within the tools.
  SamitMehta Also, in some cases, the results could be hierarchical: changesets->files and folders in each changesets; and also a list of associated change requests
• Report on the differences between two files
  NickCrossley Or two versions of a file, or the versions of a file in two configurations, etc. - this needs defining in more depth. What is the expected result of such a service - a context or unified diff or some form, or a GUI comparison, or something else? Is this just a compare service, or a merge service? If a merge, is the result to be checked in? Should the service allow control over some options for the comparison, such as the treatment of white space and new lines, the file encodings, whether or ignore byte order marks, etc.?
• Check in modified or new files
  NickCrossley It is quite likely that a user would want to check in only some of the new or modified files produced by a build - for example, many users do not put intermediate .o or .obj files into SCM. And although getting a list of new and modified files is listed as an optional build service, in many cases the build system is not aware of all such files; it might be aware of only the ones specifically marked as build targets. So, the SCM system itself almost certainly needs a way to find the list of new and modified files, and a way to select a user-chosen subset for checking in.
  SamitMehta I would suggest modifying this service or adding an additional one that allows the Bill-of-Materials from the build to be "checked in".
  SamitMehta In addition, I would suggest that since some SCM tools may be able to provide a different way to store / track / version "build artifacts" than source code, etc, we should provide a way to specify whether the file should be versioned normally or as a build artifact.
• Produce a baseline, apply a tag or label
  NickCrossley Some SCM systems might have a way to generate the name for this new baseline, label, or tag; others might require the user to supply it; still others might have a combination of both.
  NickCrossley See also the ScmWorkItemIntegrationStory for a service to show the changes made between two baselines, tags, or labels.

Services offered by the build system

Note that the build system might be part of the SCM system, or might be provided by third parties.

NickCrossley Are the build system services in scope for the OSLC SCM team? If not, how can we make this build management story work?

SamitMehta My opinion is that the SCM services that are listed above can support certain scenarios: one-way integration from Build tool to SCM tool. That should be separated from the services described below that would allow an integration in the other direction. I would suggest that the initial set of SCM services should focus on build scenarios where the SCM tool is the provider and the Build tool is consumer.

• Discover available build services
• Start a build in some given file system directory
  NickCrossley In general, builds could take a long time, so we should not assume the initiating client is necessarily around to receive a response giving the final build result.
• Get current build progress
  NickCrossley The intent here is that an application could monitor a build that was in progress. That implies the monitoring client needs either to know some identity of the build it is interested in (possibly returned from the call that invoked it), or that there is a way to get a list of current builds in progress - or both.
• Get build status and results
  NickCrossley This is to get the status and results of a build that has completed. The results might include either copies of or references to output logs, possibly a list of built targets, etc.
• Get a list of new and modified files
  NickCrossley In some cases, a build system knows exactly which files were modified or created by a build; in many cases the full set is not known. Since the SCM system has to be able to provide a (potentially slow) way to determine a set of files to check in after a build, is it worth defining a method for obtaining this list in the comparatively rare case that the build systems knows it?.

Nick, I've embedded my comments directly in the above doc. Feel free to remove them as we finish discussing them.

-- SamitMehta - 14 Oct 2009