Thursday, March 22, 2012

High Level Best Practices: SharePoint Source Code Management and the Related Deployment Process

By Errin O'Connor, Founder & CEO at EPC March 22, 2012

AIIM.ORG Article Link:

50,000 Foot Overview

This article describes the source code management and deployment process for SharePoint 2010 initaitives and provides the current-state of SharePoint 2007 deployment analysis as well as listing the various options and associated best practices for SharePoint 2010. This is a high-level EPC Group process we put in place for our clients but in follow-up articles I will take this to a much granular level.


The purpose of this article is to:
• Identify the different tools for source code management in SharePoint
• Provide best practices for source code management in SharePoint
• List the different tools for custom solution deployment in SharePoint
• Provide best practices for custom solution deployment in SharePoint

Note: For both organizations coming from non-SharePoint platforms as well as upgrading from SharePoint 2007 to SharePoint 2010.

The information is taken from various EPC Group best practices and our published online sources and as a result of hands-on testing of the of SharePoint 2010 and Visual Studio 2010.

Source Code Management– Coding Considerations

Platforms – Microsoft Office 2010, Visual Studio 2010, SharePoint Designer 2010

Features – Document Management/Records Management, User/Developer Communities, shared code/service repositories.

Security – SharePoint Server Farm – Separation of the developer and live environment allows for disassociated and secure coding that does not require prior approval or cause risk conflict. It is recommended to use this approach to ensure separation of the environments and to allow for more freedom in the development process. Live environments (QA and Production) will be subject to the SharePoint Governance Plan.

Source Code Management

Source Code Management is considered part of Software Configuration Management (SCM) and encompasses the tools, procedures and actors involved in tracking software changes over time. Myself and my team at EPC Group have seen this be an area were organizations do to always focus their budget on and can sometimes be lacking which can lead to long-term issues.

Visual Studio 2010 is the primary tool for development in SharePoint 2010 and it comes with out of the box integration support for Team Foundation Server (TFS) 2010

. TFS is more than an SCM tool as it also provides a data warehouse to store project schedules, allows for continuous integration and builds, project management and many other features. TFS is currently not an approved SCM tool per the EA SCM standards. TFS provides the basis for Application Lifecycle Management (ALM) – an area that encompasses more than just SCM.

Note: While TSF has better integration with Visual Studio, it does require licenses to be purchased.

SharePoint Deployment: Introduction

Deployment is required to take a SharePoint Customization or Development Solution and put it on a server. Development and deployment in SharePoint is different from development in .NET (ASP.NET, Silverlight etc). It involves steps that are unique to SharePoint (creating the DDF file, creating the WSP etc). These steps apply when deploying SharePoint solution to all environments (QA,Production, Contingency, etc.).

Deployment Requirements/Objectives

A SharePoint deployment solution should, in most cases, meet the following objectives:
•Implementation of packaged code in development environment: The deployment solution should be able to perform all the steps (create, build, package, deploy). for SharePoint deployments right within the development environment (usually Visual Studio).

•Modification of deployment options and parameters: Ideally developers should be able to modify the deployment parameters and options when deploying a SharePoint solution to different environments. It is not desirable to have a solution that locks developers down with an obfuscated process which is totally opaque and does not allow the team to control or modify any step of the deployment process if needed. Having a customizable solution that developers can use to create custom deployments allow flexibility and consistency across environments.

•Extensible: The SharePoint deployment process contains the following high-level steps: build, package and deploy. A deployment solution should provide ways to extend a given step to fit the specific policies of the organization or business unit for compliance, administrative/operational consistency and other reasons.

•Retraction of deployment solutions: A developer goes through multiple iterations of development and deployment while developing a solution. A deployment solution should be able to perform cyclical retraction of SharePoint solutions and support continuous dev->test cycles of SharePoint Development.

•Elimination of repetitive manual processes that reduce productivity

•Single-click build/package/deploy process integrated into Visual Studio IDE: Ideally a developer can hit a single key and SharePoint will perform all the steps for deployment within Visual Studio.

Legacy SharePoint Deployments

In SharePoint 2007, developers were introduced to the solution packaging framework which provided the ability to package multiple kinds of features or components like features, assemblies, and files into a single deployable artifact. The overall deployment process in SharePoint contained the following steps (shown in the picture below):
• Create the solution: All the required elements of the SharePoint solution are created and implemented within this step. This is usually performed within Visual Studio.

• Build the solution: Use MSBuild or custom scripts to build the solution.

• Package the solution: This involves creating the WSP for the SharePoint solution. This step involved different steps: creating DDF file, running the MAKECAB utility etc.

• Deploy the solution: This involves taking a WSP file and running the appropriate STSADM commands to deploy the WSP file into the SharePoint server so the solution is usable.

Visual Studio 2008 did not provide any standard in-built support for building, packaging and deploying a SharePoint solution to a SharePoint farm. However, there were some point solutions/add-on tools (STSDEV, WSPBuilder, and VSSWe etc) available for Visual Studio 2008 for some of these functions. All of these tools automated some of the steps in the SharePoint deployment process but fell short of the intended goal.

Legacy Deployment Limitations

In spite of these tools, the developers still had to write custom scripts to tie all the steps of the deployment process together. In a nutshell, the following were some of the limitations of the SharePoint 2007 deployment process:
• Poor Integration: The legacy deployment process required introduction of third-party tools to automate some of the steps in the deployment process. They were not integrated into Visual Studio and Visual Studio was not SharePoint aware as well. Due to the lack of integration between SharePoint and Visual Studio, each element of the solution existed in isolation with no direct relationship to each other.
• Making Communication Confusion: The legacy deployment process was confusing and did not expose the specific steps of the deployment process.
• Modification: There was no room to modify individual steps of the deployment process to meet edge case scenarios.
• Automation: There was no way to automate the deployment process.
• Retraction: There was no way to retract an existing solution other than to write a custom script.
• Upgrades: There was no way to upgrade an existing solution.
• Artifact Removal
• Conflict Resolution: The legacy deployment solution did not provide conflict resolution in deployment scenarios where the new solution creates a new version of existing SharePoint artifacts e.g. list definitions etc.

SharePoint 2010 Deployment

Deployment in SharePoint 2010 still uses the same four deployment steps as in the Legacy Deployment but the new deployment process implementation in SharePoint 2010 fixes the limitations in the Legacy Deployment process.

Visual Studio 2010 Integration

Visual Studio 2010 provides first-class support for working with SharePoint projects in the following ways:
• Single unified interface for the building, packaging and deployment processes: With the press of the F5 key, Visual Studio 2010 can perform all the necessary steps involved in the SharePoint deployment process. It can also associate the Visual Studio to the SharePoint process and thus, allow the developer to debug the SharePoint solution right after the deployment.
• Auto-discovery of mapped assets: Visual Studio 2010 now provides mapped folders that can contain things like images, CSS files etc. Visual Studio will automatically create module definitions for these artifacts and automatically include them in the WSP when built by Visual Studio 2010.
• Visual Package Explorer: Visual Studio 2010 also provides a nice visual explorer to be able to easily browse through the various components (features and folders) of the SharePoint solution (also called SharePoint Items or SPI).
• Edit-in-place configuration files (manifest.xml): Visual Studio 2010 now provides very easy-to-understand and use visual designers to work with the underlying manifest definition files. In addition, one can get to the underlying XML file and hand-edit them for complicated scenarios if needed.
• Extensible framework: Visual Studio allows creation of custom projects, SharePoint items and thus, provides ways to extend the deployment process in a customized way to fit various requirements.

Creating solutions

Visual Studio 2010 provides pre-built artifacts called SharePoint Items (or SPI). These SPIs cover most of the commonly used SharePoint solution components that one uses. These SPIs are pre-configured artifacts for common SharePoint projects and when selected and added to a Visual Studio 2010 SharePoint solution also automate generation of supporting artifacts (features, manifests, .webpart files etc) if such a feature does not already exist. The SPIs are also aware of the Visual Studio structure for a SharePoint project. As noted earlier, Visual Studio 2010 provides an extension framework where new custom SPIs can be created, added and used by others.

Building solutions

Visual Studio 2010 exposes properties to control the build process via a Property Sheet. These options include:
• URL of SharePoint site to which the solution will be deployed
• Type of deployment: Sandbox vs. Farm
• Target of deployment: Global Assembly Cache (GAC) or BIN deployment

Packaging solutions

Packaging in SharePoint still happens between the build and deploy steps.

It is possible for a SharePoint 2010 solution to contain multiple SharePoint projects. By default, Visual Studio creates a single WSP per project and so, in case of a solution with multiple projects, Visual Studio will create multiple WSP files. The default deployment process within SharePoint 2010 builds all the projects within the SharePoint solution but only the ones marked as “start up” will be deployed. Though Visual Studio 2010 provides ways to customize so we could package output from all SharePoint projects with the same solution to write to the same WSP, it is recommended as a best practice to use a single WSP per SharePoint custom "project".

Deploying solutions

The last step in the SharePoint Deployment process: the deployment step has totally changed from the previous legacy deployment process. Deployment step in SharePoint 2010 consists of many components covered in the next few sections.

Deployment View

Visual Studio 2010 provides deployment designer views that allow customization of the deployment step for a SharePoint project. Visual Studio 2010 adds a SharePoint tab under SharePoint project properties. On clicking this tab, the designer will look as follows:

Pre-deployment tasks could include initializing directories or creating directories before a SharePoint deployment while post-deployment tasks could include clean up tasks once the SharePoint deployment is complete.

Deployment configuration

Visual Studio 2010 provides deployment configurations for use in SharePoint deployment. A deployment configuration is a set of deployment steps with each step associated with a deployment action. With each deployment configuration, we can associate pre- and post-deployment tasks as shown in the Deployment View. Out of the box, Visual Studio 2010 comes with two deployment configurations: Default and No Activation. The only difference between the two is the latter does not activate the solution in SharePoint after deployment.

Typically, one will have to create custom deployment configurations to meet their specific deployment requirements. A SharePoint project can contain multiple deployment configurations but only one active deployment configuration at a time. When building a SharePoint project for deployment, the deployment configuration to use can be specified as a parameter.

Deployment Step

A deployment step implements a specific step within a deployment configuration and is associated with an action. When creating a deployment configuration, one has to specify the deployment steps included in the deployment configuration and the order of execution of the deployment steps. Each step consists of one or more tasks. Pre-defined tasks exists for common tasks like recycling the IIS Application Pool, Retracting a solution, Activate a Feature etc. but using the Visual Studio 2010 Extensibility Framework, it is possible to create custom task definitions and plug them into the deployment process by association with a suitable deployment configuration.

Conflict Resolution

One of the limitations of the legacy deployment solution in SharePoint 2007 was no way to do conflict resolution. With the integration between Visual Studio 2010 and SharePoint 2010, a developer can perform conflict resolution. There is built-in conflict resolution for SPIs provided out-of-the box with Visual Studio 2010. Developers can also customize the conflict resolution and detection logic using custom SPIs. There are three main ways for conflict resolution for in-built SPIs:
• Automatic
• Prompt
• None

Thus, conflict resolution in SharePoint 2010 deployment is very powerful when used from Visual Studio 2010 as it is specific to each SPI.

Remote Deployment

Using Visual Studio 2010 to build and deploy a solution only works for deployments to the local SharePoint Server or deploying a solution to an integration environment. Typically in most projects, the developers do not have rights to deploy a custom solution to either QA or the Production environment. A SharePoint Administrator would be responsible for deploying a custom solution to the farm in the QA or Production environments.

The development team has to hand the solution binary (WSP file) over to the SharePoint Administrator for deployment. It is recommended that the development team creates custom deployment configurations for deployments to each environment: Integration, QA and Production so that there is a consistent, well-defined and extensible set of steps associated with deployments in each environment.

Tools Involved

As indicated earlier, SharePoint 2010 provides multiple tools for deployment depending on the nature of deployment (local vs. remote). For local deployments on a developer or integration environment, Visual Studio 2010 is a recommended solution. For all QA and Production deployments, Power Shell or a custom scripting solution will meet the requirements.


1 Powershell - Windows Scripting/Administration Tool
- Allows for granular, command script control

2 STSADM - SharePoint services Command Line Tool

Solution Lifecycle In SharePoint
When deploying a custom solution (a WSP file) to SharePoint, following actions happen:

• Add the given WSP file to the Solution Store
• Solution Deployment and Distribution: Distribute, unpack and install the solution to the front-end Web Servers
• Synchronization if needed

Adding a solution to the Solution Store
Adding a solution to the solution store can be done either using the command line, Web Administrative interface or through the SharePoint Object Model.

Solution Deployment and Distribution
It is possible to use a timer job to do deployments. In such cases, the deployments create a timer job. This timer job is picked up by the timer service present in each front-end Web Server. The timer job uses the SharePoint Foundation Administrative Web Service to access appropriate privileges to deploy solution files to each computer.

Deployment Best Practices
Some of the best practices for SharePoint Deployment are:

• Use out-of-the box SPIs to implement the various parts of the SharePoint Development solution. Only create custom SPIs if the out-of-the box SPIs do not meet your requirements.

• Collaborate with the team to discuss and decide the deployment configurations to use for each environment. Ensure that the deployment configuration is agreed upon with everyone, is well documented and is extensible if necessary.

• For remote deployments, use a PowerShell like standard script language to automate and implement steps of the deployment process outside Visual Studio.

• Implement a Continuous Integration Platform and integrate that into the Deployment Process. Then, decide on the frequency and schedule of the deployments to the Integration environment and implement an automated deployment solution. This will enable the the development to quality check the integration environment on a daily basis ensuring that the latest code in SCM is working in an operational environment.

• Create a QA environment which looks exactly like Production.

• Use proper and adequate logging and exception handling in custom SharePoint solutions. This will help with troubleshooting problems encountered with a SharePoint solution in QA or Production where one does not have access to Visual Studio to debug the problem.

• Ensure that the deployment scripts and custom deployment extensions are also versioned in the SCM tool along with the source code written for the custom SharePoint solution.

• If possible, use sandbox solutions for testing a solution rather than using a farm solution.

Friday, March 2, 2012

Implementing SharePoint within Federal Government Agencies – Best Practices “From the Consulting Trenches”

Implementing SharePoint within Federal Government Agencies – Best Practices “From the Consulting Trenches” \ Tips for Government IT Executives & Appointees

Walking into Continental Airlines to implement an Enterprise SharePoint Deployment is entire different animal than walking into an organization like the Department of Justice (DOJ) for nearly the exact same initiative, or should I say Task Order. (The DOJ has had a great deal of success in SharePoint and I am merely using them as a large government institute example).

I had the privilege of having a breakfast meeting with the previous Chief Information Officer (CIO) of the United States, Vivek Kundra, last year and it was refreshing to hear his ideas about how “Big Government” should take the “lean” methodologies and best practices approach from private industry as well as his ideas on how to cut out a lot of the “fat” and red tape.

He has since left his Obama Presidential CIO (1st ever Chief Information Officer) appointment to take an Executive Vice President (EVP) position at Salesforce, but I am hopeful his ideas were passed on to administration and/or the”powers to be” as implementing a game changing and bleeding edge technology like SharePoint 2010 in a government organization can sometimes be a challenge.

I have been very vocal about my personal methodologies and best practices \ theories and those of my Sr. architects at EPC Group on how best to implement SharePoint at places such as the Pentagon, Department of Justice, multiple Air Force bases, the Department of State, Health and Human Services, the National Institute of Health, and various 3 letter organizations.

From my 10+ years of SharePoint implementation experience and 653+ SharePoint implementations (literally), I would like to express my personal 5 point plan (tips and first-hand experience advice \ mandate) on how to ensure government-sized success 100% of the time (saving millions of tax payer dollars) to CIOs, 2 & 3 Stars, high-level appointees, and the people that make the I.T. wheels turn in D.C.

My personal soapbox: Errin O’Connor’s mandates for Federal Government SharePoint I.T. Success:

1. Streamline how to get to the best SharePoint talent without the need for subcontractors to then subcontract to 2 other subcontractors… (I digress, but the amount of money I have personally seen on submitting a Statement of Work \ Proposal to a Federal Agency and then the track that has taken for 2 or 3 more firms to get involved to take an initial hourly rate from $140/hour rate to an eventual $240/hour rate blows my mind).

I am a taxpayer too and when you are talking about a 1500-5000 hour engagement, this is no small amount of money (budget) to spend (so the “good old boys club” can play contract swap and task order bingo and who has the approved 8a Stars lottery to fast track this project, to me, is personally repulsive and a taxpayer waste.

Note: I am taking nothing away from legitimate and deserving 8a Stars organizations as I believe in what they do, rightly deserve, and what they contribute but I think the GSA should take a hard look at this “status” as, in my opinion, is being abused. (Note: My personal opinion)
2. Do not be afraid to break the trends of the past | we are in the mobile age, a 1 Star General is going to get an iPad for Christmas or for their Birthday and want to use it; Have your “MMGS”, Mobile Management Governance Strategy, in order, yesterday.

3. The new employees coming into government careers are the “Facebook” generation, yes, governance is key, but stifling (internal) content development keeps the status quo’

Note: A quick sidebar: I had a conversation with a Department of Defense related member, that said, “Due to the economy, we are getting the best and the brightest candidates in the Army, Navy, Air Force, and Marines that we have had in “some” years, and they expect information now….”at their fingertips” and we are not prepared to service that request.

4. Do not just go with the consulting firm who developed the software, trust me, I have personally hired some of them and they are not always the rock stars. EPC Group, who has had over 650 successful implementations (not tooting our own horn), and sometimes will question getting involved in specific 6-8 month RFP \ proposals due to the fact that it is not worth the lengthy procurement process and contract bingo \ eBay-swap will instead take a Fortune 500 \ Fortune 1000 engagement that we know we can get executive buy-in and bring extensive SharePoint Return-on-Investment (ROI) to the table. (Note: WIthout sharing our proprietary and time tested methodologies to every IT firm interested in bidding the effort)

5. “Boutique” SharePoint Firms (like EPC Group), at the top of their game, are not always ok with sharing their proprietary methodologies with the big “7” government contractors…. something to think about and to weigh the risk factors. Why would a private firm (such as EPC Group or other respected SharePoint firms), share their time tested, “in the trenches” methodologies with an approved (massive) government contractor, who has the larger $250-500 million dollar Task Order and who can add a minuscule “SharePoint Task Order” to the overall effort (under generic verbiage) to win and staff \ deliver a project of 100-500k and risk that Big 7 firm, from turning and reverse engineering, that methodology to their own benefit. It can happen...

Note: Some of the very top talent in the SharePoint arena would, in private admit, “they would rather stay in the private sector arena and not risk their life work, i.e. risk 20k hours of 100+ employees combined knowledge and best (learned from mistakes and lessons-learned that now work time and time again), best practices for a 1 time project with zero promise of future phases with a firm who won a $250-500 million dollar “generic” task order for “IT Systems Support”.

This to me is completely frustrating and not a way to increase getting the best and the brightest to get the project done fast and efficiently. Some peoples, joking and under their breath stereotypes (about all the red tape) didn’t come from thin air and why can’t we cut out that red tape and implement a government solution for SharePoint 2010, including the NARA (The National Archives), as this information is eventually going to become part of the Freedom of Information Act.

I know personally (public information) that some of the original Space Shuttle data, now in the process of being or already archived, was originally stored in a SharePoint 2003 environment. How the National Archives (NARA) going to take that data in and make it searchable? Are we going to put it on a CD and test it (the CD\DVD) yearly to ensure it’s still valid once a year? There is such a seamless and better way this can be accomplished! (Imagine a SharePoint as a Service Platform within NARA, taking in all Government SharePoint data seamlessly regardless of what version of SharePoint that data is on?)

Final Thought…

My personal goal is for the United States government to get onboard with a solution, for not only SharePoint, but the other document management systems as well, and collaborate (led by the National Archives) to be able to take in (securely) these “content databases \ external data sources” from SharePoint 2003, SharePoint 2007, SharePoint 2010, and future versions, in a seamless manner.

There are NARA teams in Colorado that are metadata specialists that can work with all of the government agencies to develop a standard set of metadata (core content types) so that NARA can inherit content from these systems (NASA, NAVY, DoD, etc.) in a seamless and secure manner to ensure document security, seamless metadata “matching” and other requirements are stored in a 100% searchable and compliant manner.

Let’s cut the red tape with a huge set of scissors and let in some “out of the box” thinking into the many governmental :institutional" way of thinking. There are so many extreemly bright at IT folks at the top of their game in IT working at these agencies but my personally opinion from observasations as that they feel sometimes one hand is tied behind their back when they want to make real progress and real ROI changes to their organization.