Throwing Your Organization’s Money Away Or Why Must Your Organization Pay to Test Accessibility of IT Products and Services It Purchases??


Your organization has just purchased a new enterprise IT product. And your organization is one of the ones that takes IT accessibility seriously. Yay!

But taking it seriously by no means translates into spending large sums of money, time, and other resources to thoroughly test the accessibility of the product or service that you just procured, or are in the process of procuring.

Think about it. Some of you out there know I’m a car guy, and I love car analogies…no doubt a carryover from my days in the world of Industrial Design, where car analogies were often used to convince engineers or management why a product should be designed a certain way.  So, let me use one to make an important point about accessibility testing.

Let’s say that you are in the market for a new car. One of the important features that you would be interested in is safety, obviously.

You do your research…price, brand, performance, style, warranty, and then safety of course. You select the car you want that has the right blend of all these things. You look at the safety ratings / certifications, like what you read, so you buy it.

Now that you have acquired the vehicle, would you be expected to ensure the safety specs on the car are true and accurate? Airbag deployments, side impact reinforcement, crumple zones, etc.? How would you go about doing this? Drive your car into a tree at 60 mph?!?

Of course not. Fact is, the manufacturer has already performed extensive testing of the vehicle’s safety systems and has certified that the vehicle meets these standards, and to what degree.

Accessibility testing by who? And how much?

Hopefully by now, you have figured out where I’m going with this blog.

Is it reasonable for your vendors to expect you to perform comprehensive accessibility testing on their products and online services? You might think it is because:

  1. Tradition – That’s the way it’s always been done
  2. Confidence – You feel that it’s the only way to be sure that the product meets the accessibility criteria defined in the SOW, contract, or the policies for your organization.

OK, the tradition aspect is problematic on many levels, as a “tradition based” IT model is far from the modern IT models used in both private and public sectors.

Confidence in what you are procuring is certainly valid, but there are things that can be done in the procurement process that can effectively build that confidence thereby reducing the need and costs to your organization for IT accessibility testing.

First, let’s look at some of the issues around customer testing of procured offerings:

  1. Testing needs to be performed throughout a development process, starting at unit test, to identify and correct issues early. Customer testing would typically be performed far down stream in the development or post development process making such testing a very late way to identify issues and have them resolved. This is especially true for large, complex applications so the likelihood that fixes would be integrated into the version a customer buys is low. (“We’ll put it in the next release…”)
  2. IT accessibility testing typically requires a deep understanding of the product / service to be tested in order to develop test plans and execute those plans. This means knowledge transfer from the vendor to the customer, and assumes a willingness by the vendor to do so.
  3. Testing may require a specialized IT environment to install and test the IT asset that the customer has yet to deploy. In some cases, access to a vendor site where an instance of product / service is installed may be provided, and hopefully complies with security polices of the customer organization.
  4. And last but not least, there are the cost considerations (Human resources+ time+ test tools+ documentation) associated with proper testing either through the customer internal accessibility testing facilities or the use of a third party.

When these aspects are considered, its fairly obvious where responsibility for testing should reside. The question is, how do I as the customer make this happen?

Some guidance to consider for how to minimize customer accessibility testing of vendor supplied products / services:

For Commercial Off the Shelf (COTS) products and services:

  1. In the solicitation document, SOW, etc. include language that
    1. Specifies the accessibility technical standard to which a vendor’s products need to comply. (hopefully you are doing that already!)
    2. Incorporate the requirement for accurate VPATs for any COTS procurements. If the vendor cannot provide them, that can be a good indication that the product has never been tested and accessibility levels not documented. Therefore, it’s safe to assume that the product is not very accessible.

At that point, you can ask the vendor to provide a date when an accurate VPAT(s) will be available, so that at least you have a good understanding of the risk you are assuming.

  1. Analyze the VPAT. Accuracy varies wildly from vendor to vendor and product by product. After reviewing it, asking for additional information is key:
    1. Can the vendor provide a copy of the accessibility test plan for the product?
    2. What tools / methods were used to test and complete the VPAT? The response should include accessibility tools and methods with which a knowledgeable accessibility professional would be familiar with.
  • What client platforms (operating systems (including mobile), browsers, assistive technologies, and versions of all of those) were used as test environments?
  1. Can the vendor provide the results of the accessibility testing?
  2. What issues were found and are there corrective actions in place to resolve them in this or a subsequent release…and when?

If these questions can be answered to your satisfaction, and given that the responses to these questions plus the VPAT documentation serve as “official documents”, then your organization can likely forego the arduous task of comprehensive accessibility testing.

If the vendor is unwilling or unable to provide satisfactory answers, you should assume that the offering is not accessible and factor that into your decision-making process. Your testing at this point would be a waste of time and money.

There may be cases where vendors may not be able to share such information with you as it may be deemed confidential; however, you can always ask to sign a nondisclosure agreement (NDA) with the vendor to allow them to provide the needed info.

If still, the vendor cannot supply this information, you should request a letter of certification from them stating something to the effect of “the product was tested in accordance with generally accepted accessibility testing practices (including visual inspection and with an assistive technology) and that the information in the VPAT(s) is accurate.” Note: The letter should support the level of compliance as documented in the VPAT(s) and should not be written as a statement of “full compliance”, unless the product is in full compliance, of course.

If the vendor is unwilling to provide such a letter, and the VPAT looked sketchy to begin with, then again assume the product is not very accessible and base your business decisions on that. Certainly, don’t waste your time and money testing the damn thing!

In all fairness, I do not advocate total elimination of customer (or customer paid third party) testing. Depending on the asset being procured, the customer easily “sniff’ test in a few places to validate the accuracy of the vendor test documentation or their letter of certification, and to ensure there are no assistive technology interoperability issues within the customer’s solution platforms.

For non-COTS deliverables:

Non-COTS deliverables, for the purpose of this blog, can be defined as custom development services or customization of COTS offerings. The main distinction from COTS is that there are no VPATs or other accessibility documentation because the deliverables don’t yet exist. Therefore, what you really want to ascertain before entering into an agreement with a vendor for development services is whether the vendor has the knowledge, skills, tools, policies, processes, and track record to produce accessible deliverables. (examples websites, web applications, client application, or customized front ends for COTS products, etc.).

Some of the elements described above for COTS offerings will be the same for non-COTS, but there are differences in that as the customer for development services, you have much more control of what you can require of the development services vendor.

  1. For the solicitation document, contract, or SOW, etc. here are some considerations:
    1. The accessibility technical standard that you want the deliverable to meet such as WCAG 2.0 AA (included again for completeness);
    2. Your review / approval of accessibility plans / designs at designated checkpoints throughout the development cycle;
    3. Your review / approval of accessibility test plans and tools / platforms used for testing
    4. Delivery of accessibility test results documentation;
    5. Development process corrective actions criteria (prioritization of issues, and resolution plans / dates);
    6. Accessibility related remedies and warranties; and
    7. Statement that you may request additional information as needed in support of the vendor response to the solicitation documents

Additionally, you can (and should) include accessibility specific forms for the vendor to complete that will help you gauge confidence in its ability to produce quality, accessible deliverables. Note that these completed forms should be evaluated by a knowledgeable accessibility professional.

At the Texas Department of Information Resources, we developed one such form, The Vendor Accessibility Development Services Information Request. Vendors must complete this form as part of any solicitation where development services are being procured. It asks for qualitative information regarding a vendor’s accessibility policies, organization, tools, training, corrective actions, and examples of prior accessible work. The completed form can be a significant indicator as to a vendor’s accessibility development capabilities; and can be used to gauge the level of confidence in the vendors testing and results, should that vendor be selected.

You may already be familiar with Policy Driven Adoption for Accessibility (PDAA). It is a system developed by myself and my counterparts in Minnesota (Jay Wyant) and Massachusetts (Sarah Bourne) through the National Association of State CIOs (NASCIO). To read more about it go to to download the 2-part whitepaper authored by Meredith Ward (NASCIO), Jay, Sarah, and myself. PDAA includes a tool that can be given to vendors that measures the level of accessibility maturity within an overall organization. The completed form maps responses to the PDAA maturity model, which can then be used as part of the vendor selection criteria.

Please bear in mind that it’s perfectly acceptable to ask for more detailed information about the responses you receive in these forms, as vendors should be able to substantiate them.

One thing I also like to do is visit vendor websites and run the WAVE tool against their home pages to see if they are eating their own dog food.  I have to say that the results are not always impressive, and can be another indicator of a vendor’s culture and commitment to accessibility.

  1. Prior to final delivery or go-live, the vendor should provide a letter stating that accessibility development and testing was performed in accordance with generally accepted accessibility practices (including visual inspection and with an assistive technology) and that the test results and other documentation supplied to your organization are accurate, and that the deliverable complies with the specified accessibility standards (or cites exceptions where it doesn’t with a corrective actions plan) set forth in the contract or SOW.

To state the obvious, this letter may not only be provided as part of fulfilment of the contract, it can help insulate your organization from liability should issues arise.

Again, I do not advocate total elimination of customer testing (or customer paid third party), but if the vendor’s test results are comprehensive and accurate, which can be assumed given all of the other checks and balances I’ve described, some customer “sniff” testing may still be performed for the same reasons mentioned in the COTS section.

Third party testing

One final point: My colleagues in the third-party testing business may have some concerns regarding the reduced need for customer testing, but I would argue that transferring this obligation to the vendor community creates new opportunities for third party testing on the other of the equation.








Policy Driven Adoption for Accessibility (PDAA)

Policy Driven Adoption for Accessibility (PDAA)

The development of an easy to use IT Accessibility governance model for gauging the level of IT accessibility maturity within any organization, developed with and through The National Association of State CIOs (NASCIO).

To view or download the PDAA white papers, go to



Originally posted in 2013. Happy reading!



I recently returned from DEEP 2012, a conference on ICT accessibility organized by G3ict and sponsored by the Ontario College of Art and Design (OCAD) University. The conference was held in Toronto and brought together thought leaders and experts in areas of ICT accessibility, covering an array of topics and brainstorming sessions related to moving ICT accessibility forward at a greater clip.

Although not on the agenda, one topic pervasive in both the formal and informal conference discussions was related to the frustration around the lack of solid ICT accessibility business cases.


The Accessibility for Ontarians with Disabilities Act (AODA) took center stage on day one with representatives involved in shepherding and implementing provisions of the Act speaking on various aspects of it.  AODA is applicable not only to Ontario’s government bodies, but to private sector organizations as well (effective in 2016).
  How well AODA works to increase ICT accessibility adoption remains to be seen as the implementation of many of the important provisions has just begun, and there will no doubt be a few bumps along the way.  But the fact is, Ontario has done what no one else has done to date: require entities in both the public and private sector doing business in the province to make their ICT (web specifically) accessible through a governance model that includes a requirement for organizations to report specific criteria through an online filing system operated by the province.


How was Ontario able to adopt such a law?  Do you think there was a business case that supported it?  If there was, it would have certainly been externalized by now, right?  AODA didn’t need a dollars and cents kind of business case for ensuring equality to all of Ontario’s citizens, including people with disabilities, because it views accessibility as a civil right.

For what I consider an “apples to apples” comparison, let’s look at accessibility regulations for the built environment…choose a state, province, country, etc.  Were there business cases that justified the cost of fitting new and existing buildings for accessibility?  While there  were probably numerous impact assessments done, I’m not aware of any use of business cases, because business case justification is not relevant when dealing with civil rights issues.

If you follow this rationale, and the precedent set for accessibility of the built environment, why is business case justification a valid argument for providing access to ICT? What’s different about giving people physical access as opposed to ICT access?  Civil rights wise I would argue…nothing.

Before accessibility regulations for the built environment were enacted, entities rarely would have decided to expend financial resources of their own volition for accessibility because physical accessibility was an expense that they would have rather not incurred and didn’t see the need to incur it. So if left up to the entities themselves to “do the right thing”, very little would have been done, leaving those with disabilities literally out in the cold.  To solve this problem, regulations were enacted. While enforcement was and can still be challenging as new situations arise, the initiative has been very successful given the scale of the built environment. In the US for example, you will rarely encounter a commercial building code that doesn’t require full compliance to accessibility laws. Also, oversight and enforcement is fairly strict.


Now, let’s turn to the ICT environment.  What motivation, other than “doing the right thing” do organizations have for investing (yes, it does require investment) in ICT accessibility?  Let’s be honest here.  When comparing the ratio of the number of organizations that develop, sell, or use ICT worldwide to the demonstrated business case and ROI examples for increased revenue and profitability from implementing ICT accessibility over the past 10 years, the ratio is pretty darn low.

The same holds true on the litigation side. Sure, there have been a few high visibility cases, (mostly settled and not ruled on by judge or jury) but again, not many in relationship to the number of organizations in the global ICT environment.

When private sector organizations looks pragmatically at such data points, the business case for investment in ICT accessibility adoption doesn’t look all that compelling, leaving little in the way of incentive to take ICT accessibility seriously.

So, if the ROI case isn’t clear, and the risk of inaccessible ICT is low, how can organizations be convinced that ICT is important and needs to be adopted?


Equal access to goods and services, be it physical or electronic, is not a business option, it’s a civil right.  Therefore, we must look to government to ensure equality for all citizens at all levels of ability by enacting better regulations to guarantee all individuals access to ICT based goods and services…just like the built environment.

“But there already are regulations”, you say.  While this is true, most current disability regulations (not to be confused with technical standards) with respect to ICT accessibility governance are not well defined and therefore not very enforceable.

One element missing in many accessibility regulations is a simple, yet fundamental element that could significantly increase adoption: Include provisions in ICT disability regulations requiring organizations to have and implement an ICT accessibility policy.


It might look something like AODA, with respect to requiring organizations in the public and private sectors to have a web accessibility policy and report accessibility criteria through a government operated database.

The state of Texas has rule provisions that apply to all state agencies and institution of higher education requiring them to have an ICT accessibility policy; however, the provisions do not encompass other governmental bodies or private sector organizations in the state, nor does it retain a database of agency compliance to ICT accessibility criteria.

The idea behind integrating ICT accessibility policy mandate into disability regulations and requiring the filing of those policies with a government body is to compel organizations to drive themselves towards higher ICT accessibility adoption through compliance to their own “official” (filed) accessibility policy (I call this Policy Driven Adoption).  The policy provisions would have to contain criteria as to what would be required in organizations’ ICT accessibility policies which, by the way, would be filed with the government authority responsible for accessibility regulations.

Requiring a filed policy shouldn’t be too difficult to enforce/audit, as the high level action of filing criteria may be fairly binary…an organization’s policy is filed or it isn’t. It contains the correct criteria (setting achievable deadlines, and requiring organizations to show progress in their initiative, for example), or it doesn’t.  Audit-ability of how well an organization’s ICT complies with their own policy gets a bit more challenging, which is where ICT accessibility technical standards might then come into play, but perhaps just reserved for situations when an actual complaint is filed and investigated. (similar to how complaints are handled dealing with the built environment)


I would also predict that Policy Driven Adoption would encourage innovation in ICT.  As an example, look at automotive mileage standards which are essentially policy driven. While the automotive industry was initially opposed to stricter standards, they have now fully embraced them, which has given rise to countless innovations in automotive technology, promoting healthy competition, and resulting in a better, cleaner environment that benefits everyone.

The same could hold true for ICT innovation due to the increased demand/requirement for  accessibility…invention of new developer tools to automate and facilitate the development of accessible ICT and accessibility technologies, leveraging of cloud resources to enable accessibility at the end user level (SIRI as an example), or creating other things that we can’t yet even imagine, much of which could benefit everyone.

If a Policy Driven Adoption approach takes hold and begins to be embraced by industry and innovators, the benefits (and business cases) will emerge.

To make this happen, efforts need to be focused on getting policy requirements for organizations into regulation, which isn’t a trivial task, but a highly necessary one.