Your organization has just purchased a new enterprise IT product. And your organization is one of the ones that takes IT accessibility seriously. Yay!
But taking it seriously by no means translates into spending large sums of money, time, and other resources to thoroughly test the accessibility of the product or service that you just procured, or are in the process of procuring.
Think about it. Some of you out there know I’m a car guy, and I love car analogies…no doubt a carryover from my days in the world of Industrial Design, where car analogies were often used to convince engineers or management why a product should be designed a certain way. So, let me use one to make an important point about accessibility testing.
Let’s say that you are in the market for a new car. One of the important features that you would be interested in is safety, obviously.
You do your research…price, brand, performance, style, warranty, and then safety of course. You select the car you want that has the right blend of all these things. You look at the safety ratings / certifications, like what you read, so you buy it.
Now that you have acquired the vehicle, would you be expected to ensure the safety specs on the car are true and accurate? Airbag deployments, side impact reinforcement, crumple zones, etc.? How would you go about doing this? Drive your car into a tree at 60 mph?!?
Of course not. Fact is, the manufacturer has already performed extensive testing of the vehicle’s safety systems and has certified that the vehicle meets these standards, and to what degree.
Accessibility testing by who? And how much?
Hopefully by now, you have figured out where I’m going with this blog.
Is it reasonable for your vendors to expect you to perform comprehensive accessibility testing on their products and online services? You might think it is because:
- Tradition – That’s the way it’s always been done
- Confidence – You feel that it’s the only way to be sure that the product meets the accessibility criteria defined in the SOW, contract, or the policies for your organization.
OK, the tradition aspect is problematic on many levels, as a “tradition based” IT model is far from the modern IT models used in both private and public sectors.
Confidence in what you are procuring is certainly valid, but there are things that can be done in the procurement process that can effectively build that confidence thereby reducing the need and costs to your organization for IT accessibility testing.
First, let’s look at some of the issues around customer testing of procured offerings:
- Testing needs to be performed throughout a development process, starting at unit test, to identify and correct issues early. Customer testing would typically be performed far down stream in the development or post development process making such testing a very late way to identify issues and have them resolved. This is especially true for large, complex applications so the likelihood that fixes would be integrated into the version a customer buys is low. (“We’ll put it in the next release…”)
- IT accessibility testing typically requires a deep understanding of the product / service to be tested in order to develop test plans and execute those plans. This means knowledge transfer from the vendor to the customer, and assumes a willingness by the vendor to do so.
- Testing may require a specialized IT environment to install and test the IT asset that the customer has yet to deploy. In some cases, access to a vendor site where an instance of product / service is installed may be provided, and hopefully complies with security polices of the customer organization.
- And last but not least, there are the cost considerations (Human resources+ time+ test tools+ documentation) associated with proper testing either through the customer internal accessibility testing facilities or the use of a third party.
When these aspects are considered, its fairly obvious where responsibility for testing should reside. The question is, how do I as the customer make this happen?
Some guidance to consider for how to minimize customer accessibility testing of vendor supplied products / services:
For Commercial Off the Shelf (COTS) products and services:
- In the solicitation document, SOW, etc. include language that
- Specifies the accessibility technical standard to which a vendor’s products need to comply. (hopefully you are doing that already!)
- Incorporate the requirement for accurate VPATs for any COTS procurements. If the vendor cannot provide them, that can be a good indication that the product has never been tested and accessibility levels not documented. Therefore, it’s safe to assume that the product is not very accessible.
At that point, you can ask the vendor to provide a date when an accurate VPAT(s) will be available, so that at least you have a good understanding of the risk you are assuming.
- Analyze the VPAT. Accuracy varies wildly from vendor to vendor and product by product. After reviewing it, asking for additional information is key:
- Can the vendor provide a copy of the accessibility test plan for the product?
- What tools / methods were used to test and complete the VPAT? The response should include accessibility tools and methods with which a knowledgeable accessibility professional would be familiar with.
- What client platforms (operating systems (including mobile), browsers, assistive technologies, and versions of all of those) were used as test environments?
- Can the vendor provide the results of the accessibility testing?
- What issues were found and are there corrective actions in place to resolve them in this or a subsequent release…and when?
If these questions can be answered to your satisfaction, and given that the responses to these questions plus the VPAT documentation serve as “official documents”, then your organization can likely forego the arduous task of comprehensive accessibility testing.
If the vendor is unwilling or unable to provide satisfactory answers, you should assume that the offering is not accessible and factor that into your decision-making process. Your testing at this point would be a waste of time and money.
There may be cases where vendors may not be able to share such information with you as it may be deemed confidential; however, you can always ask to sign a nondisclosure agreement (NDA) with the vendor to allow them to provide the needed info.
If still, the vendor cannot supply this information, you should request a letter of certification from them stating something to the effect of “the product was tested in accordance with generally accepted accessibility testing practices (including visual inspection and with an assistive technology) and that the information in the VPAT(s) is accurate.” Note: The letter should support the level of compliance as documented in the VPAT(s) and should not be written as a statement of “full compliance”, unless the product is in full compliance, of course.
If the vendor is unwilling to provide such a letter, and the VPAT looked sketchy to begin with, then again assume the product is not very accessible and base your business decisions on that. Certainly, don’t waste your time and money testing the damn thing!
In all fairness, I do not advocate total elimination of customer (or customer paid third party) testing. Depending on the asset being procured, the customer easily “sniff’ test in a few places to validate the accuracy of the vendor test documentation or their letter of certification, and to ensure there are no assistive technology interoperability issues within the customer’s solution platforms.
For non-COTS deliverables:
Non-COTS deliverables, for the purpose of this blog, can be defined as custom development services or customization of COTS offerings. The main distinction from COTS is that there are no VPATs or other accessibility documentation because the deliverables don’t yet exist. Therefore, what you really want to ascertain before entering into an agreement with a vendor for development services is whether the vendor has the knowledge, skills, tools, policies, processes, and track record to produce accessible deliverables. (examples websites, web applications, client application, or customized front ends for COTS products, etc.).
Some of the elements described above for COTS offerings will be the same for non-COTS, but there are differences in that as the customer for development services, you have much more control of what you can require of the development services vendor.
- For the solicitation document, contract, or SOW, etc. here are some considerations:
- The accessibility technical standard that you want the deliverable to meet such as WCAG 2.0 AA (included again for completeness);
- Your review / approval of accessibility plans / designs at designated checkpoints throughout the development cycle;
- Your review / approval of accessibility test plans and tools / platforms used for testing
- Delivery of accessibility test results documentation;
- Development process corrective actions criteria (prioritization of issues, and resolution plans / dates);
- Accessibility related remedies and warranties; and
- Statement that you may request additional information as needed in support of the vendor response to the solicitation documents
Additionally, you can (and should) include accessibility specific forms for the vendor to complete that will help you gauge confidence in its ability to produce quality, accessible deliverables. Note that these completed forms should be evaluated by a knowledgeable accessibility professional.
At the Texas Department of Information Resources, we developed one such form, The Vendor Accessibility Development Services Information Request. Vendors must complete this form as part of any solicitation where development services are being procured. It asks for qualitative information regarding a vendor’s accessibility policies, organization, tools, training, corrective actions, and examples of prior accessible work. The completed form can be a significant indicator as to a vendor’s accessibility development capabilities; and can be used to gauge the level of confidence in the vendors testing and results, should that vendor be selected.
You may already be familiar with Policy Driven Adoption for Accessibility (PDAA). It is a system developed by myself and my counterparts in Minnesota (Jay Wyant) and Massachusetts (Sarah Bourne) through the National Association of State CIOs (NASCIO). To read more about it go to nascio.org/pdaa to download the 2-part whitepaper authored by Meredith Ward (NASCIO), Jay, Sarah, and myself. PDAA includes a tool that can be given to vendors that measures the level of accessibility maturity within an overall organization. The completed form maps responses to the PDAA maturity model, which can then be used as part of the vendor selection criteria.
Please bear in mind that it’s perfectly acceptable to ask for more detailed information about the responses you receive in these forms, as vendors should be able to substantiate them.
One thing I also like to do is visit vendor websites and run the WAVE tool against their home pages to see if they are eating their own dog food. I have to say that the results are not always impressive, and can be another indicator of a vendor’s culture and commitment to accessibility.
- Prior to final delivery or go-live, the vendor should provide a letter stating that accessibility development and testing was performed in accordance with generally accepted accessibility practices (including visual inspection and with an assistive technology) and that the test results and other documentation supplied to your organization are accurate, and that the deliverable complies with the specified accessibility standards (or cites exceptions where it doesn’t with a corrective actions plan) set forth in the contract or SOW.
To state the obvious, this letter may not only be provided as part of fulfilment of the contract, it can help insulate your organization from liability should issues arise.
Again, I do not advocate total elimination of customer testing (or customer paid third party), but if the vendor’s test results are comprehensive and accurate, which can be assumed given all of the other checks and balances I’ve described, some customer “sniff” testing may still be performed for the same reasons mentioned in the COTS section.
Third party testing
One final point: My colleagues in the third-party testing business may have some concerns regarding the reduced need for customer testing, but I would argue that transferring this obligation to the vendor community creates new opportunities for third party testing on the other of the equation.