Feasibility Report - Client Satisfaction Survey

 


 
 
Home | Site Map


 

 

 

 


           
 
Board Logo
 

Feasibility Report



| Client Satisfaction Survey | Final Report |

FEASIBILITY REPORT ON A LABOUR RELATIONS BOARD CLIENT SATISFACTION SURVEY


Table of Contents

EXECUTIVE SUMMARY
FOUNDATIONS OF THIS REPORT
CLIENT SATISFACTION SURVEYS - GENERAL
What are They?
Purpose
The Critical Questions
Comments from the Auditor General's Office
Comments from Alberta Treasury, Statistics
Comments from Alberta Labour, Information Services
Agency Surveys
Exceptions
Not Cover The Substance of Decisions
Not Cover the Composition of the Board
Methods of Conducting a Survey
Factors to Consider
How Often to Do a Survey
EXPERIENCES OF OTHERS
Alberta
Ontario
Federal
United States
A Sampling
AN LRB CLIENT SATISFACTION SURVEY
Purpose
Input from the Management Team
Factors to Consider
Appendix 1 - Written Materials Reviewed



I was asked to explore and report on "the possibilities of ...(a client satisfaction survey)... with some recommendations about what could be done ... to advise (the Chair) about matters including possible outside consultants who could be of assistance (including their availability to fast-track such a project and the estimated cost) and the experience of other tribunals ... to provide (the Chair) with a range of options to consider."

Client satisfaction surveys are one way of assessing performance measurements for an organization. They can also assist in building stronger relationships and implementing change mechanisms. The most effective and useful surveys are those that address specific versus general comments and assess process oriented areas. The general concept, purpose and methodology for client satisfaction surveys are outlined in the report.

A quasi-judicial tribunal needs to consider the elements of independence and impartiality when determining whether to or what to survey clients about. Such a tribunal, such as the Labour Relations Board, should exclude any survey questions touching on the substance of the decisions made by the tribunal and the persons appointed to the tribunal and comprising any of its panels.

All of the professionals recommend the Board address and answer two questions before it decides to engage in a survey. If it cannot answer these two questions, or decides it cannot or will not act on the information obtained, it should not proceed with a survey:

1. What information do we need to gather from the clients? (Why are we doing this?)

2. What decisions will we make with the information we gather? (Will we act on the results and make changes?)

Included in the report are other items for consideration, both generally and specifically for the Labour Relations Board. Comments from other tribunals who have engaged in the surveys, as well as from Alberta Treasury, Statistics, the Auditor General's office, and Alberta Labour, Information Services are set out.

Client satisfaction surveys have become a common instrument for government in Alberta, although few, if any, quasi-judicial tribunals in Alberta have engaged in such an exercise. Across the country, Ontario is the province where these measures are next most popular because of the Agency Reform initiatives occurring there. There is some movement occurring in the federal sector as well. The Manitoba Labour Board and the Public Service Staff Relations Board are two labour tribunals who have undertaken such a survey. A sampling of the initiatives across the country is included.

The surveys which I reviewed were extremely varied, as were the experiences of the providers. Cost varies with the provider and the methodology selected. All providers agreed that the extent of the survey, both content and number of clients, would affect the cost.

Recommendations

My recommendations are set out below.

Do We Do A Survey?

From all the research I have done, I recommend the Board engage in a process oriented client satisfaction survey. Not only is this a popular instrument for client interaction at this time, but it is one which is new to the Board's clients and may generate a higher level of interaction with us outside of the application process. There is a high level of enthusiasm for the initiative among both the management team and all the staff. Although I was asked not to discuss the matter in the community or with the Board members, I am confident the Board members would enthusiastically support the initiative as well.

The management team is sufficiently cohesive on the purpose and content of the survey, as well as its commitment to act on the information obtained. The surveys done by the Manitoba Labour Relations Board, the Ontario Human Rights Commission and the Public Service Staff Relations Board cover similar areas for input as we would seek and can be valuable starting points for us.

If we engage in a survey, we should seriously consider including the Board members and staff as two distinct client groups. This may require some modification to the questionnaire and a different methodology for these two groups, but the comparison information (internal versus external perspectives) could be valuable for a small addition to the cost.

What Methodology?

The methodology will be driven by the number of questions the Board wishes to ask, the required sampling sizes to obtain representative samples and to some extent, the recommendations of the chosen provider to enable us to accomplish the goal in the desired time and budgetary restraints. From all the discussions I have had to date, I would lean towards a written or telephone survey, combined with some personal interviews or focus groups. As you will see from the details, there is a wide range of options on methodology as well as content.

Next Steps

Now that this report is complete, the next steps would be:

  • By February 1, 1999, a decision that we will proceed with a survey.
  • Agreement on the purposes of the survey (which will drive the content).
  • Commitment as to the funding to undertake such a survey.
  • Commitment from the management team that the Board will act upon the information obtained in the survey.
  • Draft and send out the RFP to these suppliers to that they can provide a more accurate and detailed tender. (They will need at least one week to reply.)
  • Deadline for RFP's to be February 16, 1999.
  • Selection of the provider and signing of contract.
  • Establishment of the survey instrument and methodology.
  • Pre-testing of the questionnaire and amendment as necessary.
  • Administration of the survey.
  • Analysis of the results and preparation of the reports.
  • Presentation of the final report to the management team by May 30, 1999.

Go back to Top



FOUNDATIONS OF THIS REPORT

In preparing for this report, I reviewed some written material (see attached list), spoke to members of the Department of Labour, solicited input from the national board of the Council of Canadian Administrative Tribunals, spoke to some members of Alberta's tribunal community, met and spoke with representatives of the Auditor General's Department and Alberta Treasury, Statistics Branch, met with the Board's management team, spoke with potential providers, spoke with representatives of Ontario's Agency Reform Management Board Secretariat and representatives of some of the Ontario tribunals, and spoke with Chairs of the Saskatchewan and Manitoba Labour Relations Boards and representatives of the BC Labour Relations Board. Bob Poburn assisted me by doing some research and by attending a seminar on the topic presented by a representative of Alberta Treasury, Statistics and other investigations.

Go back to Top



CLIENT SATISFACTION SURVEYS - GENERAL

What are They?

An organization typically will assess its performance measures in three broad areas: efficiency (objective - dollars and time), effectiveness (whether the intended public impact has occurred given the public purposes of the organization), and quality (focus on product - how well the service meets the clients needs it is designed to satisfy).

Currently, client satisfaction is seen to be one measure of the quality of performance and can be used to assess effectiveness of an organization. Although used in private sector business extensively, governments have recently adopted "client satisfaction" as a key indicator for use in business planning exercises. More recently, this view has been altered to reflect the perspective that content specific indications of client satisfaction are more useful to the organization that general indications of pleasure or displeasure.

Surveys to assess client satisfaction range from the very general "how do you like us approach" to "do you know about us" to "how well did we do last time" to specific "what could be done to improve the mediation experience - mark 1 - 7 that apply". The nature, design, focus and delivery method of the survey varies with its sponsor and its purpose.

Go back to Top



Purpose

Although client satisfaction surveys can be used as information gathering initiatives for general knowledge, it is an expensive and time consuming way to obtain "feel good" information.

In a nutshell, client satisfaction surveys are better used:

  • as instruments of accountability (for public bodies it is seen to be one method of keeping in touch with the community which the body serves and of reporting on the body's ability to meet its established standards for service by "assessing the effectiveness, efficiency and quality of service against stated objectives" [Ont]),
  • to provide specific information for use internally in setting performance targets (standards for service); and
  • to provide specific information for use in developing action plans to improve performance (best practices)
  • to provide specific information for use in the allocation of resources.

Optimumly, client satisfaction surveys should be:

  • directly related to performance goals, best practices or operational aspects of the tribunal
  • clear and concise
  • seek specific versus "feel good" information
  • content oriented
  • re-usable on a regular basis with the expectation of receiving comparable information.

Response rates to surveys (particularly mail-outs) will be affected by:

  • the format of the questionnaire
  • the timing of the survey
  • the credibility attached to the survey
  • the nature of the questions asked (if topical and interesting to client)
  • the link to the benefits for the client
  • assurances of confidentiality.

Go back to Top



The Critical Questions

It has been repeatedly stressed by the professionals who do this work that an organization should not undertake a survey unless it can answer two critical questions:

1. What information do we need to gather from the clients? (Why are we doing this?)

2. What decisions will we make with the information we gather? (Will we act on the results and make changes?)

Go back to Top



Comments from the Auditor General's Office

In his 1997 Annual Report, the Auditor General included some comments about client satisfaction surveys (page 25). Recommendation No. 7 in that report says:

It is recommended that ministries ensure their client satisfaction survey methods produce valid and reliable results. It is also recommended that standards be developed for reporting survey information.

He concluded the section by saying he would prepare additional information on the use of surveys in reporting performance. On the Auditor General's website, such a document is posted, Client Satisfaction Surveys.

Contained in these guidelines are suggestions for survey design, execution and reporting. Some highlights of those suggestions include:

  • survey design is critical to its execution and results
  • surveys should be pre-tested
  • surveys should take no more than 30 minutes to complete
  • client samples must be randomly chosen amongst the various groups to be polled
  • reliability of survey results should be shown using a confidence level of 95% +/- 5%
  • one can increase the response rate by sending a letter in advance and providing a contact person as well as the name of the survey professional
  • certain information should be contained in the report to enable the reader to understand how the information was collected and compiled and how the results relate to the goals of the organization.

Some additional comments by the Auditor General's Office were provided in the form of questions or tips for the Board to consider:

What are you trying to come to grips with about the relationship you have to manage?

What are you going to do with the information? What happens if you get negative information? (Once you get it, you have to do something with it.)

If a quasi -judicial body is not able to or willing to change the way it does business because of independence or impartiality issues, it should not engage in a survey.

The key is the intent/goals of the survey. Do the people who interact with us feel well served?

It is not necessary to ask "Are you satisfied", rather you can get the information by asking more specific questions.

To satisfy the Auditor General's office, a tribunal needs to be able to disclose the criteria which it used to decide what to survey and who to survey. If the sample is too selective (but without criteria), it is not a representative sample.

The Auditor General's Office has offered to review the draft questionnaire and provide comments for us.

Go back to Top



Comments from Alberta Treasury, Statistics

Alberta Treasury outlines three steps to completing a successful survey:

  • plan - must have a clearly defined purpose (eg. Be designed to improve service to the public) and designed to support all final analysis (identify the information needs, the data collection method, the questionnaire content, the client listing and the sample selection)
  • development - review and use existing research and surveys where applicable; decide to use interview, mail survey or telephone survey; determine the survey content which should be subjective, confirm the customer relationship, avoid assumptions, use clear, concise and consistent wording, continue minimal personal bias, measure a specific concept and allow for various degrees of satisfaction and input. The format of the survey is important: image is critical, group questions into groups, use white space, balance compactness and legibility. On the sample selection, size matters, simple random sampling is best or a systematic sample or a stratified random sample, and the sample must support the level of analysis for the final report. A sample pretest of 10 - 12 clients should be done to ask about the cover letter and questionnaire and to look for patterns in responses.
  • analysis - employ accepted testing and analytical procedures, avoid unsupported conclusions, report with precision, report specific limitations and do not exceed the data limitations.

They caution that surveys can create an unnecessary expense for the organization and burden on the respondents. They can produce inadequate or stale data. They also caution that providers must be carefully instructed and supervised to avoid abdication of the role by the organization. For the Board and the consultant, Treasury recommends everyone spend sufficient time at the front end thing about what is desired and how it will be used so that the appropriate results are obtained to support the analysis desired, which affects the design of the survey.

Alberta Treasury urges agencies to carefully seek in the RFP and examine in the proposals the specific description of methodologies used in the analysis and the types of analysis to be provides as well as the techniques used.

Go back to Top



Comments from Alberta Labour, Information Services

Alberta Labour reinforced much of what was related to us by other sources. They were able to provide assistance with some specific questions which arose from the general review.

On the topic of focus groups, they confirmed that results from these interviews are not statistically reliable, however they do provide in-depth comments of the nature that the management team is seeking. It is common practice for the consultant to pay an honorarium to focus group participants, even on government initiatives. They suggested that focus groups may be a viable way to supplement a written or telephone questionnaire. It is a useful tool for surveying all of the lawyers for example, by holding firm by firm focus groups. Personal interviews may work for the staff input.

Alberta Labour suggested our client size of 1000 is manageable enough to enable us to survey all 1000 clients by either written questionnaire or by telephone. They caution that the telephone questionnaire will, by its nature, be less in-depth (a view professed by one of the providers as well.)

Finally, Alberta Labour provided us some suggestions and comments on the use of Requests for Proposals and how to evaluate the proposals we receive.

Go back to Top



Agency Surveys

Eight common goals for tribunals, as critical to effective and efficient performance and service quality, were identified in Everyday Justice, the Report of the Agency Reform Commission on Ontario's Regulatory & Adjudicative Agencies (April, 1998). At page 18, the commission describes these goals as:

1) Fairness: The provision of service and performance of statutory functions in an impartial, lawful, unbiased and just manner.

2) Accessibility: The ability to provide information and services that are simple and easy to use.

3) Timeliness: The performance of tasks within established time frames based on reasonable expectations.

4) Quality and Consistency: The production of accurate, relevant, dependable, understandable and predictable information and results, with no errors in law or fact.

5) Transparency: The use of policies and procedures that are clear and understandable to everyone involved.

6) Expertise: The possession and use of the skill, knowledge and technical competence required to discharge all statutory responsibilities and maintain public confidence.

7) Optimum Cost: The provision of services at a cost that is based on best practices and is cost effective for everyone involved.

8) Courtesy: The demonstration of respect to everyone who come into contact with the agency.

These eight goals comprise the general areas within which a tribunal may seek client satisfaction input relative to its operations, especially the core business functions.

Go back to Top



Exceptions

Two important qualifications must be added when addressing client satisfaction surveys for adjudicative bodies, such as the Board. Those qualifications are that any survey:

1. not cover the substance of the decisions

2. not cover the composition of the Board.

Go back to Top



Not Cover The Substance of Decisions

The Board is a quasi-judicial tribunal which must at all times consider and act to preserve its independence and impartiality. Just like the courts, the Board cannot be seen to be seeking input from the clients on the results of our adjudicative efforts. The process is adversarial by design and, therefore, we can naturally expect that one party in every dispute will be unhappy with the decision - usually the party who lost. The survey content and methodology must not give the impression that the Board will be swayed by client opinion when deciding future cases; to do so would raise the spectre of a reasonable apprehension of bias.

This distinction was recently recognized in Everyday Justice, where at page 18 the Commission notes:

The commission recognizes that it is important to separate independence of decision making from agency performance accountability. Performance accountability looks at how the agency performs its work, not what decision is made in any case or cases.

One way of assessing the quality of decision making (that reasons for decision are high quality and consistent within cases and adjudicators) is to submit decisions to an external review committee (who have experience and knowledge) to review and report to the Chair on areas for improvement. This removes the determinative substance of the decision from scrutiny and focuses on the process aspects which can be addressed by training on decision writing, etc. It also removes the presumptive conclusion (and the corresponding bias) that at any time one half of the parties will be dissatisfied with a particular decision - because they lost.

Go back to Top



Not Cover the Composition of the Board

Surveying clients about their views on the members a tribunal could also create a reasonable apprehension of bias, detrimentally affecting the independent decision making authority of the tribunal. It could potentially put the Chair in a difficult position when assigning members to sit cases if the community expressed strong views about a particular member.

The judiciary has recognized this dilemma and the debate in that forum continues. However, both Manitoba and Nova Scotia have proceeded with judicial evaluation surveys. In Manitoba the survey focussed on the process, not individual judges. In Nova Scotia, the reverse occurred with the focus on individual performance, but the results were not published.

In Ontario, the Performance Management report (written by two tribunal adjudicators and which is currently being implemented in Ontario) emphasizes the need to separate performance measurement of the agency from that of the individuals within the agency, as a result of the requirement of the individuals to work independently of government.

It is more appropriate to establish and maintain a separate appointment process to ensure that qualified, competent and credible decision makers are appointed, trained, and retained. This view was also stated in the Everyday Justice report at page 15. The Labour Relations Board currently has an open, transparent and accessible process for the appointment and reappointment of Board members. We should not undermine that process by seeking opinions about appointed members. The same can be said to apply to staff members.

Go back to Top



Methods of Conducting a Survey

Client satisfaction can be determined by one or more methods, such as:

  • statistical review
  • written questionnaires
  • telephone interviews
  • in person interviews
  • focus groups
  • any combination of the above.

Each has its advantages and disadvantages. All methods depend on the purpose for doing the survey and the quality of the questions being asked to elicit productive results. Cost and time increases as you move from statistical reviews to focus other methods. In each case (outside of statistical review) the cost and time associated with the design of the questionnaire and the analysis and report are about the same. The cost varies in the field costs of the survey instrument.

In each case, the providers indicate that the time leading up to the production of the survey questions can be time consuming (which adds to the time for the survey). This time can be reduced if the agency is focussed on the purpose and content of the survey.

A statistical review could be conducted using the information available in the Board's data base. It would reveal time frames for processing matters, costs associated by case, and other information such as the number of clients using legal representatives, the number of cases of a particular type processed etc. It is essentially what we now do in the annual report. Such a review, however, reveals little or no information about client reaction to such things as communication efforts, officer assistance, ADR efforts, etc.

Books have been written about how to design a successful written questionnaire. The words of John K. Norton are worthwhile keeping in mind when considering this method. "The time of busy people is sometimes wasted by time-consuming questionnaires dealing with inconsequential topics, worded so as to lead to worthless replies, and circulated by untrained and inexperienced individuals, lacking in facilities for summarizing and disseminating any worthwhile information which they may obtain." In other words, be focussed, concise and clear about what you want answered.

Telephone and in-person interviews take less time, as do focus groups, and elicit more detailed answers and immediate follow-up. Obviously, they generate a higher rate of return because of the personal contact. These three methods require fewer completed surveys to obtain results that can be viewed with confidence.

Normally, persons who attend focus groups are given an honorarium (often $50.00) for their time. This would appear to be a questionable practice for an adjudicative tribunal however the rate of participation may depend on it. Honorariums would add to the cost of such a method. Alberta Treasury advises that focus groups are normally done in conjunction with one of the other methods, not alone. In the case of our client group size (1000), they caution that it would be difficult to sample sufficient clients through focus groups alone to obtain reliable results.

Go back to Top



Factors to Consider

Regardless of the methodology selected, the literature and my interviews reveal that all successful surveys address the following criteria:

  • freedom from bias - this often translates into someone outside the tribunal conducting the survey to remove potential bias from the questions asked and the interpretation of the results.
  • validity - the ability of the survey to measure what it purports to measure. Essentially, the survey, to be meaningful, must measure satisfaction against established tribunal goals.
  • reliability - the survey must poll sufficient clients with a sufficiently high rate of return that the results can be interpreted to be a real and true expression of client views. If less than the full client base is surveyed, the sample selected must be representative of those who actually use the service.

The Auditor General of Alberta recommends that a survey on general satisfaction receive a 60% return rate to enable the author to draw meaningful conclusions. The professionals who do this work advised that to obtain a 95% reliability with a + / - 4.8% error rate (which is considered acceptable) within a population of 1000 clients (our potential), we would need to obtain at least 400 completed surveys. This number would increase if we had distinct segments within the audience that we wished to poll and obtain confidence in the segment results.

Additional factors to consider are:

  • cost
  • ability to contact the clients (more difficult with in person interviews and focus groups)
  • how many clients can be contacted (anything more than a mailed questionnaire can be limiting)
  • whether clients need to be stratified or grouped for input and analysis
  • ease of completion (it is easiest to fill out a form on your own time)
  • ease of tabulation (written, quantitative questionnaires are easiest to tabulate)
  • uniformity (interviewers may alter the question)
  • time needed to complete the survey
  • how the survey results will be published and used.

Go back to Top



How Often to Do a Survey

How often varies, but the key is do to one regularly after the first survey is complete. Repeating the same survey assists the tribunal in assessing whether the standards it adopted and the changes it made were effective. The Auditor General's office spoke of one organization that is doing a survey every 2 years using an outside provider. This enables them to obtain good and consistent information in a less costly way and gives them the necessary time in between surveys to implement change.

If the results of this survey confirm what the Board is already able to extrapolate from our administrative reports, we may find it more useful to engage in sampling focus groups in the future. This would enable the Board to obtain the detail comments and suggestions, rather than statistical trends.

Go back to Top



EXPERIENCES OF OTHERS

Alberta

Eight departments reported client satisfaction survey results in their 1997-98 annual reports. The Auditor General predicted that 12 would do so this fiscal year. Few of the administrative tribunals have engaged in client satisfaction surveys. The WCB is one exception, but I understand that survey did not extend to the Appeals Commission. Both Alberta Treasury and the Auditor General's office have commented on the surveys conducted by the Departments and have gone so far as to cooperatively prepare and publish some guidelines about surveys. However, neither of them had been asked to consider the different circumstances arising from surveys by or for quasi-judicial administrative tribunals.

Go back to Top



Ontario

As a result of the Everyday Justice report in Ontario, all Ontario agencies will be required to adopt a process for performance measurement by March, 1999 including the use of annual client satisfaction surveys. Janet Skelton of the Ontario Agency Reform Management Board Secretariat says the Commission's recommendations were accepted by Cabinet and are being implemented. All agencies have completed at least the first draft of their performance measures. Many have included some type of client sampling in those measures, although the Secretariat is leaving it to the Auditor to comment on the quality of the client surveys or how they should be conducted.

Go back to Top



Federal

A similar approach is being taken in the federal jurisdiction, as a result of requirements that federal agencies now report to Parliament in the spring (their planning document) and in the fall (their performance reporting) every year. These agencies are to set up performance indicators for reporting, including statistical reports and client satisfaction surveys, although it is not clear that surveys are a required indicator.

Go back to Top



United States

Although popular in the US Federal government sphere as well, at least one paper, Customer Satisfaction Measurement Issues in the Federal Government by Tracey R. Wellens and Elizabeth A. Martin, has been written discussing the uses and limitations of this type of measurement in a government setting. The same concerns are identified in the US as exist here about the quality of the survey and the reliability of the results.

Go back to Top



A Sampling

Provincial agencies in Manitoba and Alberta have voluntarily used client satisfaction surveys in either general or specific ways.

A sampling of some of these Canadian national and provincial initiatives include:

  • the Public Service Staff Relations Board - in 1998 used a written survey to seek input on its entire operations. Chair Tarte advises his tribunal used Consulting and Audit Canada to complete the survey. It took about six months from start to finish. CAC met with the Chair to review the scope of the survey, designed the questions, sampled a group to test the validity of the questions, conducted a focus group on the questions, administered the survey by mail, analysed the results and prepared a written report. The PSSRB did not see the returned surveys to protect confidentiality. The Chair began the process himself by personally meeting with the heads of all the unions and employers to seek input into the Board's operations. He indicated the results of the survey were predictable, but have also been helpful in several planning and budgeting areas. For example, as a result of the conclusion that there needed to be more consistency in the Board's decisions, he has prioritized a training initiative for board members.
  • the Ontario Human Rights Commission - in 1997 used a written questionnaire given to each participant to assess its mediation initiative. The Commission has an in-house statistician who oversaw the survey and analysed the results which was all designed and delivered in-house. The Commission was very pleased with the results it obtained and was able to use them to modify its mediation system. It began an extensive training program for its staff in this area and in the intake area which, when combined with a new computerized intake system, has resulted in significant changes and productivity in the organization. They handle about 2000 new cases per year and currently have a backlog of 2500 cases which they hope to eliminate by 2000.

The Commission has just contracted a firm to conduct a quality assurance review on all aspects of the organization. It began with an internal focus and has produced a preliminary report with recommendations to re-engineer the operational side of the organization. They anticipate doing a targeted client satisfaction survey at some point in this process as well.

  • the Ontario Environmental Assessment Board - is currently drafting a client satisfaction survey.
  • the Canadian Transportation Agency - employed an outside firm, EKOS, in 1998 to conduct a survey relating to the Act, including areas of client satisfaction. The first results were obtained December 21, 1998.
  • the Alberta Environmental Appeal Board - in 1997 used a single question to solicit comments on its Rules and Procedures. (This is not uncommon to the type of practice the LRB has used in the past and is currently using on the change to the witnesses' oath procedure.)
  • the Manitoba Injury Compensation Appeals Commission - in 1998 used a written questionnaire to seek client input on the treatment of clients, its communication efforts, and the timing and processing of matters to hearing.
  • the Alberta Workers' Compensation Board - in 1997 conducted a client survey which included telephone surveys, focus groups, interviews and responses to background papers. It is currently undergoing a Policy Consultation process which it calls the largest public review in its history.
  • the Alberta Labour Employment Standards Branch - in August, 1998 used Coopers & Lybrand to assist it in conducting a review of the Regulation. The final report reveals that the Department set the overall scope and focus of the project, identified the issues and developed the specific questions, upon which C & L then provided input. The Department administered the survey and C & L input and categorized the data and provided the summary of results and a report.
  • the Manitoba Labour Relations Board - conducted its own survey in 1995 in response to resource shortages. The Board used the Department's research staff to assist with the survey design and analysis. It mailed out questionnaires to all the regular community - employers, unions and counsel, but did not include one time clients. They received a 39% overall response with very satisfactory responses to even very specific questions.

Go back to Top



AN LRB CLIENT SATISFACTION SURVEY

Purpose

If it is to engage in a client satisfaction survey, the most important thing the Board must determine is the purpose for conducting such a survey. It appears that both the Auditor General's Department and Alberta Treasury agree that a tribunal (or a department) should not engage in such an initiative unless there is a process related purpose to it. "Feel good" questions or general "how are we doing" questions are expensive and far less productive and meaningful for a tribunal such as the Board, although they may be more applicable in a different context and for a different user.

Currently, the Board has an extremely strong and reliable way of compiling information which can assist us in assessing performance and in establishing best practices -- the data base and the reports generated from it. From that information, we can draw conclusions about performance in many of the eight areas mentioned in the Everyday Justice report. For example, we can draw conclusions about fairness based on the number of objections filed and the number of reconsiderations and judicial reviews. We can assess timeliness by comparing actual processing times to those standards we have set. The cost can be broken down by case and compared to previous years.

What we cannot conclude or can at best guess at are those things within the client's knowledge. For example, are the materials we produce easily accessible, understandable and useable? How and where do clients access the materials? How do the time frames which we have set as standards in various matters relate to the client's involvement in the matter? Are they too lenient or too ambitious? Have we maintained public confidence? What drives that view? What costs does the client incur in appearing before the Board? Are those costs reasonable? What can the Board do to make its procedures "cost effective for everyone?" Do the clients believe they are being treated with respect and courtesy? What can we do to improve that? These are the types of information a survey could expose.

The Board's operation has two focuses which could drive the purpose of a survey -- the business plan and the development of internal best practices. Although the latter may be reflected to some degree in the business plan, it is more comprehensive in both scope and detail. The business plan, if we are to continue to follow the model expected by the government, is concise and limited to about four performance measures which cover the general scope of the Board's role.

Any Board sponsored survey should have as its purpose tied to either or both of these functions. That will also assist in focussing the areas of input and the questions into process or content related topics which will be more productive for the Board.

Go back to Top



Input from the Management Team

After meeting with the Board's management team, it appears that they see the purpose of such a survey could be to:

  • better utilize and balance our resources
  • plan and budget for needed resources
  • assist the Board in determining performance measures within the business plan
  • assist the Board in developing best practices
  • to assist the Board in identifying training needs
  • to assist the Board in selling change to the staff and caucus.

In addition to wanting to know where the problem/ potential change areas are, the management team would like to know how extensive the problem is: is it a one time event or a recurring situation? Is it experienced by a one time client or a regular client? They would like to hear suggestions for change or improvement. The management team would also like to be able to see an analysis of the results by category of client and who they represent, by frequency of client interaction with the Board, and by industry or sector group. These would affect the initial design of the survey.

Go back to Top


Factors to Consider

When considering whether and how to conduct a client satisfaction survey for the Board, some factors are:

  • the costs of any survey.
  • the survey needs to be underway (and if possible, complete) by March 31, 1999
  • the results should be independently validatable or supportable from other sources, such as the annual report or the data base reports.
  • the areas to be surveyed should be meaningful and useful to the Board for the purposes of establishing:
  • benchmarks towards best practices,
  • priorities for accounting and budgeting
  • performance goals in the business plan , or
  • identifying areas for improvement in the Board procedures
  • identifying ways to better utilize Board resources.
  • the potential clients should include all those who regularly appear before the Board [available on the data base] plus all those who had contact with the Board in the last six months to one year [again available on the data base]. Persons who contact the Board because they are seeking Employment Standards or the Labourers' Union should not be contacted as they do not fall within the mandated service area of the Board. It would also be impossible to identify information callers unless they subsequently filed an application with the Board.

The broadest area of "general clients" would be the members of any of the unions or employees of any of the employers who have bargaining relationships under the Board's jurisdiction. In order to identify even a representative sampling of this group, we would need the cooperation of the employers and the unions. Some of those groups may face FOIP considerations and be unable to release the information to us. Given the time and limitations of cost, this is probably too wide a scope for input.

  • the survey results should be published - probably on the Board's website - and a summary provided to every client who participated in the survey.
  • the survey results could be used to adjust the Board's business plan and budget objectives, to develop best practices and to adjust the Board's procedures and publications. If the survey is to proceed, this means that the management team has to make a commitment to using the results which are obtained.
  • the survey should produce an analysis of the results, rather than just a summary of them. From the discussions both Bob Poburn and I engaged in with the various critics, it became clear that this is an area that often produces dissatisfaction with the final product. While it is not difficult to summarize the results obtained (we did such a summary of the evaluation of the Health Care Conference), it is more difficult to analyse and compare the various responses to produce meaningful information and recommendations for action. The PSSRB survey report appears to produce the most comprehensive analysis of any that I observed.
  • The management team discussed the possibility of including the Board members and the staff as two distinct groups of "clients" because of their "inside perspective." This would further enable us to compare the internal perception with the external perception and address changes which need to be made there. As a word of caution, it is likely the survey will have an ancillary impact on the staff and Board members, if the management team does not act on the results, as it will likely decrease the confidence of those persons in the credibility of the Board (and the management team) once the expectation of action has been created.
  • An additional caution addresses the manner of selection of the clients who will participate related to the credibility of the survey. There are some clients who are so regularly before the Board that to exclude them would be to create the appearance of not seeking a valid sampling of results. This is particularly so with the lawyers. One option may be to survey all the lawyers who appear before the Board as it is a relatively small yet easily definable group.

Go back to Top

 



APPENDIX 1 - WRITTEN MATERIALS REVIEWED

Everyday Justice, Report of the Reform Commission on Ontario's Regulatory & Adjudicative Agencies April, 1998

Performance Management for Ontario's Adjudicative and Regulatory Agencies, Agency Reform Management Board Secretariat, September, 1998

Clients Give Agriculture Staff Top Marks for Service Connexus, Summer, 1998

Evaluating Judicial Evaluation, Canadian Lawyer, March, 1998

Alberta Employers: Shaping WCB Policies, Profile, Fall, 1998

Measuring Customer Satisfaction (Survey design, use and statistical analysis methods) 2nd edition, Bob E. Hayes, (1998, ASQ Quality Press)

Questionnaires: Design and Use, Douglas R. Berdie and John F. Anderson (1974, The Scarecrow Press, Inc.)

Administrative Law: Reigning in Unruly Fiefdoms or Threatening Independence? Agency Reform in Ontario: Perspectives on Independence and Accountability Brian Goodman, prepared for the Current Developments in Administrative and Employment Law Conference

Public Service Staff Relations Board Client Satisfaction Survey, March 1998

Evaluation of the Environmental Appeal Board, 1997 Stakeholder Consultation

1997 Service Satisfaction Survey: Final Report August, 1997

Manitoba Automobile Injury Compensation Appeal Commission Survey

Annual Report of the Auditor General of Alberta 1997-98

Review of Employment Standards Regulation Alberta Labour, August, 1998

Alberta Labour Relations Board Business Plan, 1999 - 2001

Manitoba Labour Board Client Survey, October, 1995

Customer Satisfaction Measurement Issues in the Federal Government,Tracey R. Wellens and Elizabeth A. Martin

Increasing Profits with Customer Surveys, James Sauer 1999, http://expert-market.com/client/seminars/strategicperformance-s.html

Customer Surveys, inforsurv, www.infosurv.com/cusotmers_surveys.htm

Competitive Benchmarking Associates, www.competitiveanalysis.com/contents.htms

Client Satisfaction Surveys, Auditor General of Alberta, October, 1998

Trial by Survey, The Canadian Bar Association National, October, 1997

Evaluating Customer Comments, The Business Research Lab, http://strategis.ic.gc.ca

Engineering Consultant Selection: Proposal Evaluation Criteria Reference Guide, Alberta Transportation and Utilities

Go back to Top

 

     
 

| Home | Disclaimer | Contact Us |
Copyright ©1996 Alberta Labour Relations Board