The Non-Technical Art of Being a Successful DBA - Third Party Product Evaluations

Initial Analysis
Clarifying the business needs takes place during this phase. Individual, departmental and corporate requirements are evaluated. The focus of the analysis is on solving the business or technical problem as opposed to buying a product. End users may have conflicting requirements. Needs clarification helps the users in synthesizing a common statement of the overall requirements. Analysis to determine if existing products can solve the problem is performed at this time. The make vs buy analysis is also completed during the initial analysis phase.

The following questions need to be addressed during the initial analysis:

• Can the solution be provided with existing product sets?
• Should the product be purchased or built in-house?
• Are other business or technology units able to utilize the product?
• Does the product provide any other additional benefits?
• What is the impact of not solving the business or technology problem?

Determine Impact to the Information Technology Support Infrastructure
The product's affect on the support infrastructure is determined during this phase. Although the major focus should be on solving the problem identified in the initial analysis, it is proper to allow external considerations to affect the final outcome. A product is useless if it cannot be successfully implemented and integrated into the existing information processing environment. The product's impact on individuals and groups as well as its impact on existing products and technologies is determined at this time.

Collect the following information during this phase of the analysis project:

• The use of new products often increases development time and training costs. Can the additional time and higher application development costs be justified by an increase in functionality or competitive advantage?
• If the product requires support from the development area, can the application personnel effectively administer the selected product?
• Risk vs Benefit. Is the additional functionality/competitive advantage worth any additional risk the product brings to the business or IT units? This is often described as the product's comfort ratio.
• Identify changes to the organizational infrastructure required to support the product (staff additions, training, etc.)
  • Identify changes to other technologies and products.
  • Identify additional products required to support the selected product.
  • Identify changes to existing technologies and products the selected product will require.
  • Identify changes to policies and procedures the selected product will require.

Analysis Evaluation
The analysis evaluation determines the resources required to evaluate, select, implement and support the selected product. Some problems may not be cost-effectively solved by existing technologies. Evaluation personnel must estimate the costs required to evaluate the competing products. In other words, don't spend $100,000 to evaluate a $1,000 product. As a rule of thumb, the evaluation, selection and procurement costs should range from three and ten percent of the project budget. The time required to perform the evaluation is also determined during this phase.

Perform the following activities during this phase:

• Determine and estimate the resources required to further evaluate the business needs.
• Determine and estimate the resources required to perform the product selection.
• Estimate general implementation and customization costs.
• Estimate on-going support costs, including staff requirements and training.
• Determine timelines and deliverables for the evaluation, selection and implementation processes.
• Determine date when the recommendation will no longer be valid (recommendation life-cycle).

Obtain Business Unit and IT Management Commitment
One of the major activities of any evaluation process is acquiring management commitment for the evaluation, selection and implementation stages. Representatives from user and technical management need to be a part of the evaluation process to ensure their continued support. Management always asks the same questions. As a result, it is relatively easy to prepare for them. A satisfactory answer to "What business problem are you trying to solve?" is usually followed by "How much will it cost to purchase, implement and support?", and "What is the risk if we don't solve the problem?". Management wouldn't be doing their job if they didn't ask these questions. To be successful in this phase, you need to be prepared to answer them.

Obtain business unit and IT management commitment on the evaluation process, selection process and possible implementation by providing the following information:

• Reasons why you are performing the evaluation (What business problem are you trying to solve?)
• How the product solves the business problem.
• What you intend to test.
• Time and cost estimates for product selection, product purchase, implementation and on-going support (Is the solution worth the cost?)

Create Evaluation Team
The evaluation team is created in this phase. For products that do not have a wide impact and are relatively inexpensive, the evaluation team will consist of a few select individuals. But for products that have a wide impact or are expensive, it is prudent to involve a larger group, some with a technical background, others with a user perspective. Wide representation and diverse viewpoints obtained early in the decision process produce better decisions and fewer surprises later. It is imperative to include all those affected: IT personnel who will be supporting the product, IT groups affected by its usage and end-users. Team leaders are also identified during this phase.

The evaluation team consists of representatives from:

• Infrastructure support (personnel who are experienced in related technologies or who would be affected by the product implementation).
• Application developers who will be supporting the application.
• Business unit champions and end-users.

Locate Potential Vendors
Identifying the vendor offerings that will be evaluated takes place in this phase. The problem is usually too much information as opposed to too little. The best way to create a potential vendor list is to take a structured view of the marketplace and investigate all avenues to give your evaluation team the best possible chance of finding the alternatives.

A few sources of locating potential vendors follow:

• Internal personnel who are experienced in related technologies (evaluation team).
• Industry research groups- Gartner, Forrester, Burton, IDC.
• Internal research - trade journals and trade shows.
• Vendors that have existing relationships with your company (i.e. IBM, MSOFT, Oracle, etc.)
• Internet search engines.

Initial Elimination
This phase will eliminate products that should clearly not be considered as viable alternatives. The overall intent is to reduce the number of candidates being evaluated. With each additional candidate, the cost of the evaluation increases. The key to a successful elimination process is to create an in-depth set of evaluation criteria that will be used to evaluate the potential candidates. Using weighted measures helps to identify the measurement criteria that are most important to solving the business problem. If the product passes this phase, it is passed to the vendor evaluation phase for a more thorough examination.

Perform the following activities during this stage:

• Create the vendor selection process (methods used to perform the evaluation) and evaluation metrics (evaluation criteria used to compare the products). Document both the evaluation methods and metrics.

  The evaluation metrics will include:

  • Vendor Assessment (financial stability, size, alliances, etc.). The best product doesn't always win. The more expensive the products are, the more time you need to spend reviewing the vendors making them.
  • Functional requirements - evaluate product based on its ability to solve the business problem.
  • Technical requirements - evaluate product based on its technical capabilities.
  • Create a general requirements document containing a compliance matrix that uses weighted measures to show how each vendor meets the requirements.
    
    
• Create and distribute a Request for Information (RFI) document using the evaluation metrics created above as the basis for the document.
• Evaluate the vendor responses to the Request for Information request.
• Reformulate the requirements based on vendor input.
• Create a vendor short list to keep evaluation costs to a minimum.

Vendor Evaluation
An in-depth evaluation of each of the remaining products takes place in this phase. The processes and methods created in this step are more detailed than those created in the initial elimination phase. Vendor responses to the Request for Information may also require alterations to both evaluation processes and evaluation metrics. Remember that an individual product's feature set is usually a compromise of function, speed and cost when compared to its competitors. You need to determine which factors are most important to you and weight them more heavily during your evaluation. A Request For Proposal (RFP) helps to formalize the evaluation process. The Request For Proposal details the scope of the sales process and defines the procedures and rules to be followed by the vendors. Vendors are more enthusiastic about committing resources to the sales process when it is clear to them how they are to respond and how their responses will be evaluated.

Perform the following steps in the Vendor Evaluation phase:

• Create a final vendor selection process (methods used to perform the evaluation) and weighted evaluation metrics (evaluation criteria used to compare the products).
• The evaluation processes and metrics created in this phase are more in-depth than those created during the Initial Analysis phase.
• Vendor responses to the Request for Information may require alterations to both evaluation processes and evaluation metrics.
• Functional requirements - evaluate the product based on its ability to solve the business problem.
• Technical requirements - evaluate the product based on its technical capabilities.
• Create Request for Proposal (RFP)
  • The Request for Proposal is more formal than the Request for Information.
  • The document details the scope of the sales process (meetings, presentations, product demos, trial periods, timelines and deliverables).
  • It also defines the procedures and rules to be followed by vendors during the evaluation process. Ground rules are written and shared with the qualified vendors.
• Notify the vendors of how they will be evaluated (overview of compliance matrix).
• Vendor meetings and presentations are held during this phase.
• Create a questionnaire and contact the vendor supplied customer references.
• The evaluation matrix is used to measure product conformance to the business unit IT unit requirements.

Communicate Results
The results of the evaluation process are communicated during this phase. There are many communication avenues to choose from. A formal evaluation document is created and distributed. Presentations detailing the evaluation process are also given. Before you make your recommendation, be prepared to justify it. The major questions to ask yourself are "Did you choose the best product?", "Did you keep the level of risk at an acceptable level?" and "Did you achieve these objectives at a reasonable price?"

The documentation provided to business and IT management includes:

• An executive summary.
• A detailed description of the vendor chosen.
• A financial analysis of the chosen vendor.
• The reasoning behind the vendor's selection.
• What other products were evaluated and the reasons why they weren't selected.
• An overview of the evaluation process.
• The metrics used as the basis for evaluation.
• An overview of the responses to the Request for Information and the Request for Proposal.
• The benchmark results of all competing products, if any.

Wrapup
I hope you enjoyed reading this blog on third-party product evaluations. As I stated in the introduction, the intent of this blog was to not influence you to follow the steps provided verbatim. This contents of this blog should be considered as a set of general recommendations that can be used to increase the quality of the third-party product evaluation process. Happy evaluating!

Thanks for Reading,

Chris Foot
Oracle Ace