[Openroad-users] Subversion to track component changes

[Durwin Wright]

The OpenROAD Source Management definitely needs improving.  This is one
of the things that we hope to achieve by opening the OpenROAD source.
We need to be able to interface with any external source management
system that a user desires.

 

The OpenROAD development team faces the same challenges as our
customers.  I would like to share the technique that is used internally
by the OpenROAD team to interface with the internal source management
system used by Ingres Corporation.  

 

We use an internally developed source management system that is very
similar to Perforce.  I refer to the SCC used by Ingres and the OpenROAD
as simply the "Ingres Source Management System".  It is used to manage
not only 4GL source but the 3GL source for OpenROAD, Ingres, Enterprise
Access and a number of internal systems.   We can reconstruct any
version of OpenROAD (both 3GL and 4GL) for practically any release,
patch or service pack.

 

I am responsible for both the daily and release builds of the OpenROAD
Windows 3GL and the 4GL images (used by the Windows and Unix versions of
OpenROAD).  One of the things that I always do for the 4GL build is to
build the product from the source management archives.  The first thing
that I do is to destroy the existing 4GL database and build all
components from their OpenROAD component exports stored on disk that has
been staged from our source management system.  This takes about 45
minutes for the entire OpenROAD set of 4GL applications.

 

By sharing this information on this technique, I hope that you will get
a sense of what we are thinking about for this area.  It is our hope
that we can engage a number of our customers to help us define the
requirements assist in the implementation of a solution that meets all
of our requirements.

 

 

4GL Source Management Procedures Used by the OpenROAD Development Team

 

Abstract

 

This document describes the procedures we follow to maintain the source
for our 4GL applications.  The key point to note is that the component
exports in the Ingres Source Management System are now the real 4GL
source, and take precedence over anything in development databases.

 

 

The Ingres Source Management System and 4GL

 

The Ingres Source Management System is a powerful source management
tool, and we want to be able to utilize the features it offers when we
develop and maintain our 4GL applications.  But in order to use the
Ingres Source Management System we must represent our 4GL source as
files that can be submitted to the Ingres Source Management System.

 

But what kind of files?  To answer that question effectively we need to
consider the way the Ingres Source Management System saves its files
internally.  The Ingres Source Management System archives revisions by
saving only the differences between the prior revision and the next
revision.  In most cases, this technique saves a lot of space.  But it
also means that every time you check in a file, the differences between
the prior revision and the revision you are checking in must be
calculated.  If a file is large and has many differences, this
processing can be very expensive.

 

 

Application Exports

 

One way to convert 4GL into files that can be submitted to the Ingres
Source Management System is to simply export the application.  This is
convenient because an application export file encapsulates everything
needed to reconstitute the entire application in a development database
or a build database.

 

But this approach has several problems:

 

*         With non-trivial applications, the files tend to be very
large.  For example, an application export of WorkBench creates a 9MB
file.

*         The files are expensive to diff.  Large files are inherently
more difficult to diff, but the way OpenROAD does its encoding tends to
create a lot of difference sections even when very little of substance
has actually changed.

*         The diffs are generally not readable.  With diff output so
voluminous, and not easily associated with a particular component, it is
almost impossible to look at the diffs and get any sense of what was
actually changed.

*         The granularity is too large for meaningful change tracking.
As far as the Ingres Source Management System is concerned, every change
touches the whole application.  We don't get to use the Ingres Source
Management System's revision history to track changes at the component
level.

*         Note that the "expensive diff" problem is not a theoretical
issue.  We have found from bitter experience that checking in a large
application export can sometimes take hours because of the diff
calculations involved.  A workaround for this is to check in an
intermediate zero-length revision before submitting the real revision.
This wastes space, but it does allow the diff calculations to run
quickly.

 

 

Component Exports

 

Another way to convert 4GL into files that can be submitted to the
Ingres Source Management System is to export individual components into
separate files.  

 

This approach has several advantages:

 

*         The export files are smaller.  For example, the largest
Workbench component export (except for the 1MB splashscreen image) is
only 500KB.

*         The files are easier to diff.  Although there is still a lot
of noise generated by the way OpenROAD does its encoding, the smaller
files make the diffs much more manageable.

*         The diffs are often somewhat readable.  Because the diffs are
associated with a particular component, it is easier to look at them and
recognize which script is involved, and get some sense of how the text
was changed.

*         The smaller granularity provides more meaningful change
tracking.  We can utilize the Ingres Source Management System's
bookkeeping to track changes to individual components.

 

The main problem with component exports is that there is no component
that contains the application properties.  So there is no way to
completely reconstitute an application from component exports alone.

 

 

Application Descriptors

 

To enable us to reconstitute an entire application from component export
files, we have created the concept of an "application descriptor",
represented by an export file with a ".dsc" extension.  This contains
the application properties that are not contained in any components.

 

This is actually just an application export.  But it is an export of the
application in an empty state, stripped of all its components.  It
provides the empty shell of an application, into which all of its
components can then be imported.

 

(Because the new version of Workbench prevents you, quite reasonably,
from deleting the starting component of your application, an application
descriptor may not be entirely empty of components.  It is allowed to
contain the starting component.  However, this is just a placeholder.
The "real" copy of the starting component will be maintained in its own
component export file just like all the other components.)

 

The application descriptor file, combined with all the component export
files for that application, allows us to completely reconstitute an
entire application from files contained in the Ingres Source Management
System.  This provides a complete representation of 4GL application
source that can be preserved and tracked in the Ingres Source Management
System.

 

The Component Exports are now the REAL 4GL Source.  The application
descriptors and component exports in the Ingres Source Management System
should now be considered the "real" primary copy of the 4GL source for
all our 4GL applications.  The OpenROAD Development database is just a
workspace.

 

The OpenROAD Development database is still our primary workspace and
should always be kept in sync with what is in the Ingres Source
Management System.  But remember, if there is a disagreement, the Ingres
Source Management System wins!  Just to be safe, we may periodically
refresh the database from the component exports in the Ingres Source
Management System.

 

The Ingres Source Management System Conventions

 

In order to manage our 4GL source in the Ingres Source Management System
effectively, we must adhere to the following conventions:

 

1.    All directory names and all file names should be lowercase in the
Ingres Source Management System.  Two file names that differ only in
case may map to the same physical file on platforms with
case-insensitive file systems.  Using a strictly lowercase convention
helps us avoid that problem.  (But please leave any violations of this
convention grandfathered!!!  Never, never, never attempt to rename a
file in the Ingres Source Management System by changing only the case of
the name.  That will essentially make the entire directory unusable on
case-insensitive systems.)  Note that you may see some UNIX-specific
makefiles in the Ingres Source Management System with uppercase or
mixed-case names.  But all other files should be lowercase.

2.    Component and application names used in filenames must be
truncated at 28 characters.  Only the first 28 characters of the
component or application name can be used because the Ingres Source
Management System has a 32-character filename limit, and the ".exp" or
".dsc" extension takes up four of those.

3.    Application descriptor filenames should be created using (the
first 28 characters of) the application name in lowercase, followed by
the ".dsc" extension.

4.    Component export filenames should be created using (the first 28
characters of) the component name in lowercase, followed by the ".exp"
extension

5.    4GL source directory names should be created using the application
name in lowercase.  They can be located under any source path, but the
build script that contains the list of 4GL applications that will be
built must be updated to know where to find them during a build.

6.    Mixed case is fine for actual application names and component
names.  Mixed case is also allowed for image filenames. The build
scripts will restore the mixed case to an application name when it is
imported, and when it is imaged.   Only the Ingres Source Management
System source files and directories are required to be lowercase.

7.    Only one application can be contained in a source directory. It
should contain one "<applicationname>.dsc" descriptor file, and any
number of "<componentname>.exp" component export files.

 

 

Creating and Maintaining Application Descriptors

 

Creating the initial application descriptor export for a large
application can be tedious because you have to delete all the
components.  But at least it's not something that has to be done
frequently.  Open the application in Workbench, delete all the
components (except, optionally, the starting component), and then export
the (now "empty") application as "<applicationname>.dsc" into the
appropriate source directory.

 

To reconstitute an application from its component exports, you must
first import the "<applicationname>.dsc" file to establish the empty
application shell in the database.  Then the components can be imported
into that empty application.  (When importing a component, don't specify
the "-c" flag.  That way, the name, including the original case mixture,
will be read from the export file.)

 

If you ever change the included apps list, or any of the application
properties for an application, you must update the corresponding ".dsc"
file to reflect those changes.  Unfortunately, you can't simply export
the changed application from your work database, because all the
components will come with it.  And you probably don't want to delete the
components in your work database.  So the usual technique for updating
the ".dsc" file is to import it into an empty database (or one that
doesn't already have that application name in it), manually update the
included apps list or the application properties, and re-export the
empty application as a ".dsc" file.

 

 

Creating and Maintaining Component Exports

 

Do not store generated SCP components in the Ingres Source Management
System.  Those should always be regenerated during the build process.
SCPs were introduced in OpenROAD 4.1 with ASOLib.  See the product
documentation for more details.)

 

Before working on any 4GL component, first reserve its corresponding
component export file in the Ingres Source Management System.  This
ensures that nobody else is working on that component. 

 

Also, be sure that your work database is in sync with the Ingres Source
Management System source.  If you have any doubt, re-import the
components from the Ingres Source Management System component export
files before you start editing.

 

If you change several components and the changes are related, please
submit the changes as a single the Ingres Source Management System
change.  Use separate Ingres Source Management System submissions for
unrelated changes.

 

When you need to change some components, follow these steps:

 

1.    Reserve the Ingres Source Management System files.

2.    Obtain a list of the files you have reserved:

3.    Make your changes.  Until you have tested the changes, save
private copies of the components you change (File->Save Private).  When
you save a private copy, other users will not see your changes.

4.    Test the changes.  You can often test 4GL changes by running the
Workbench under the 4GL debugger.  Sometimes, however, it may be easier
(or necessary) to make a new image and use it to test the changes.

5.    When you have tested the changes and are sure that they are
correct, save the modified components with the File->Save command.

6.    Export each modified component to the appropriate directory.  The
Workbench export operation will automatically create the correct file
names unless the component name needs to be truncated.

7.    At this point it is useful to make sure that you have remembered
to export all the modified components.  

8.    Check the changes into the Ingres Source Management System.

 

The Ingres Source Management System provides some nice cross-integration
tools that utilize a 3-way-text-diff process to propagate changes
between related codelines.  Unfortunately, this technique cannot be used
with 4GL export files.  Export files must always be generated from a
database, and cannot be edited directly as text. 

 

 

Durwin Wright | Sr. Architect | Durwin.Wright at ingres.com | Ingres | 500
Arguello Street | Suite 200 | Redwood City | CA | 94063 | USA  +1
650-587-5523 | fax: +1 650-587-5550 

 

-----Original Message-----
From: openroad-users-bounces at peerlessit.com
[mailto:openroad-users-bounces at peerlessit.com] On Behalf Of Simon Lovell
Sent: Thursday, May 17, 2007 4:37 PM
To: openroad-users at peerlessit.com; PRabjohns at uk.atex.com
Subject: Re: [Openroad-users] Subversion to track component changes

 

One source of annoyance when I was using CVS was that you had to be
consistent whether you do the export/import set on UNIX or on Windows.
Otherwise you get/remove Control-M at the end of every line, and the
diff output is useless.

 

But better than diffing the export files is diffing the docapp.  By
doing it this way you get rid of most of the garbage and can get more
meaningful output even for form changes, although your changes would
have to be minor to be worth trawling through.  Have your checkin
program run a doc app before the export and save the output in a
directory somewhere.  You can store that in your source control too, if
you like, it's just that you can't re-import the output.

 

At a point in the export file is the number of characters in the script,
a colon, then the script.  I'm sure modifying export files is not
supported, but, if you were really keen, you could write an OpenROAD
program to do it.  Would be pretty tricky for form changes :).

 

 

 

 

-----Original Message-----

From: "Pete Rabjohns" [PRabjohns at uk.atex.com]

Date: 17/05/2007 08:54 PM

To: "International OpenROAD Users" <openroad-users at peerlessit.com>

Subject: [Openroad-users] Subversion to track component changes

 

Note: Original message sent as attachment

 

_______________________________________________

Openroad-users mailing list  Openroad-users at peerlessit.com

 

To unsubscribe please click on this link

mailto:openroad-users-unsubscribe at peerlessit.com&subject=unsubscribe

 

To subscribe please click on this link

mailto:openroad-users-subscribe at peerlessit.com&subject=subscribe 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://peerlessit.com/pipermail/openroad-users/attachments/20070517/05cd3572/attachment.html