Category Archives: Evaluation

Method of the month

Every month we feature a participatory method/tool in our newsletter and I am always thrilled to have such positive feedback and am frequently told by readers that they really look forward to see what the ‘Method of the month’ will be!  As a result we thought it would be a nice idea to turn this feature into a monthly blog post in order to (hopefully) reach more people.

We always welcome suggestions and photographs of participatory methods that other organisations have been using, so please get in touch if you have a technique you’d like to share!

An archive of past newsletters is available on our website.

Last month’s featured method is…

Evaluation thermometer

This is a visual, participatory evaluation technique that could be used to evaluate a meeting, event or focus group. Participation Cymru recently used this tool to evaluate the regional participation networks that took place this month.


Tools needed: Flipchart paper, coloured marker pens and sticky dots.

Preparation: Draw a large thermometer onto the flipchart paper. The top should represent heat (excellent/great), the middle should be lukewarm (reasonable/OK) and bottom should be cold (disappointment/dislike).

Choose what you’d like people to rate, e.g. overall impression, how participative the event was etc.

Give sticky dots to participants and ask them to stick them where they feel they should go on the thermometer and take photos of the results.

– Sarah

New methods, good practice and evaluation dice: Our May Participation Networks

This post was written by Dyfrig just before he left Participation Cymru.

We always try and practice what we preach – that participation helps to delivered services that better meet people’s needs. February’s networks gave us a lot of scope to consider how to take our participation networks forwards. Participants told us they wanted the opportunity to learn more innovative participatory methods and also the chance to reflect on case studies, as these meant people could see what’s working in other areas or fields.

Each network began with an icebreaker, which was chosen by and put into practice by Sarah Ball (who was unfortunately unavailable for the North Wales network as she was assisting members of the Citizen’s Panel for Social Services in Wales to give evidence at the Senedd), who will be taking my role and facilitating future networks.

The icebreaker was Human Bingo, where people had to network to find other participants who had the characteristics listed on the bingo sheet we gave them. The first one to find someone who matched each one of the characteristics won a prize! A pack of Milky Way Magic Stars, which were highly sought after! We then had a bit of a discussion about it – I had initially been a bit skeptical about some of the more specific and outlandish points (such as someone who spoke a language that wasn’t English or Welsh), but it was generally felt that these more unusual points helped to get conversations started.

Our associate trainer Justine Scorrer then led a session on participative techniques. We started with some icebreakers such as Put Your Distraction in the Corner and Lucky Dip. We then put some methods into practice such as Roman Voting, Building Teams, 12 Brains are Better than 1, and Yes and……..Yes, before asking people What Do You Really Think? Justine then asked participants to reflect on their learning through Train it Forward, which was used to evaluate this part of the session.

After a short break, we looked at specific case studies. Kath Cook gave a great case study in Gorseinon about how NPT Homes had involved young people. After our featured presenter dropped out last minute, Justine Scorrer stepped into the breach in North Wales to talk about another project NPT Homes where they had engaged with tenants to set outcomes for the housing service, and in the South East we heard from Brecon Beacons National Park

We then broke up into small groups to hear from participants about work that they are undertaking, good practice they have encountered or issues they are having. I heard about innovative work across the three networks, including how City and County of Swansea had involved young people in procurement, how Hywel Dda Health Board used podcasts in their engagement, and Menter Iaith Conwy engaging with people on a community centre in the county.

DiceLast but not least, we evaluated the network. We used a technique called Evaluation Dice to do this, where people rolled a dice and they fed back on an aspect of the network, depending on the number the dice landed on.

To go back to the opening paragraph, we always try and put feedback into action. After the first network we were told that the dice we were using was too small. Fortunately in the interim we had ordered a large, novelty inflatable dice, which made the exercise easier and probably more fun too!

The next regional networks will take place in October and the theme is accessible information. Although I will have left Participation Cymru by then, I hope to attend them to share good practice in my next role with the Wales Audit Office’s Good Practice Exchange, so I hope to see you then!


Wayne Jepson reflects on our work and being a member of our Advisory Panel

This is the final video in a series of three that we conducted with Participation Cymru Advisory Panel members, following previous interviews with both Derek Walker of the Wales Co-operative Centre and Margaret Peters of the Countryside Council for Wales (which is now part of Natural Resources Wales). We filmed these as part of our Evaluation Framework, which helps us to ensure that what we do is meeting the needs of people and organisations who access our services.

Wayne Jepson on Participation Cymru / Wayne Jepson ar Gyfranogaeth Cymru from Participation Cymru on Vimeo.

Wayne is a long serving member of the Advisory Panel. He has represented NLIAH (the National Leadership and Innovation Agency for Healthcare, which has now closed, with its functions transferred to the Welsh Government and NHS Wales) for a few years now, and he has been involved in commissioning work from us as a project as well as providing a steer for our work as a member of the Advisory Panel.

When I asked Wayne to tell me how he saw the role of Participation Cymru’s Advisory Panel he said “I think the Advisory Panel is a critical element of Patricipation Cymru’s development. I think it provides a forum to inform and influence decisions that are being made by Participation Cymru and about Participation Cymru and the wider public sector. For me, the Advisory Panel not only acts as a programme board might for a project, but it also is there as that check and balance for Participation Cymru. I think that because of the make-up of the Advisory Panel – people are from a range of different organisations and different sectors – it brings a wealth of experience and knowledge to the table that can only help Participation Cymru in developing and moving forward”.

One of the National Principles for Public Engagement in Wales is to “Work with relevant partner organisations”. NLIAH firmly put this into action in their role as an Advisory Panel member by commissioning training from us in partnership with the Welsh Local Government Association. This approach ensured the best possible use of resources, but also gave added value to the training as attendees were given the opportunity to network and to learn from each other’s experiences.

NLIAH produced a range of useful resources on improving health services, including really useful guidance on involving adult NHS service users and carers, which clearly shows that listening to the voice of patients and the public is vital in order to ensure the improvement of NHS services in Wales.

– Dyfrig

Margaret Peters reflects on the work of Participation Cymru

The below video is the second in a series of three interviews that we’ve conducted with Participation Cymru Advisory Panel members that we’ve filmed as part of our Evaluation Framework.

Margaret Peters on Participation Cymru / Margaret Peters ar Gyfranogaeth Cymru from Participation Cymru on Vimeo.

Unlike the first interview with Derek Walker of the Wales Co-operative Centre, where we interviewed Derek in person, we interviewed Margaret over the phone and used images and video from our panel meeting.

When we asked Margaret about the role of the Advisory Panel she said “I think they’re there to steer Participation Cymru in the right direction and support them, and the fact that having such a range of public service organisation as panel members allows for joint working and a consistent approach to citizen and community engagement across Wales’ public service organisations”.

Margaret has represented the Countryside Council for Wales on our Advisory Panel since before the current team had even begun our jobs at Participation Cymru, and following this interview she’s changed roles and she is no longer a member of the panel. She’s worked incredibly hard to assist us with our work, and we’re all incredibly grateful for the time and effort she has put into being a panel member. We all wish her well in her new role.

Whilst I’m the first to admit that the video isn’t quite Hollywood quality, it’s nevertheless surprising that the video was put together with free software. I used Windows Moviemaker, which is a standard part of Windows packages, and I also used Audacity, which is free open-source software that enables you to edit audio files.

Using video instead of written documents allows people to get a better feel for the work we do as a project. It means that people can listen to and watch panel members talk about what they do in their own words.

Monmouthshire County Council have taken video to the next level by using YouTube to consult on their budget proposals. Their blog is a fascinating account of their work and well worth a read if you’re looking to start using video to consult and engage.

– Dyfrig

Derek Walker reflects on the work of Participation Cymru

Evaluation is vital to make sure that engagement is undertaken in an effective way, after all it’s the only way we can learn from both our successes and our mistakes. When we engaged with people around Wales to put the National Principles for Public Engagement in Wales together, you agreed with us, so it became principle 10 – “learn and share lessons to improve the process of engagement”.

We’ve implemented an evaluation framework so that we can better capture the results of the work that we’re doing. The people we work with are central to how we evaluate our success, which is why we ask for your comments after our training and networks – so that we can improve our services and ensure that they better meet your needs.

As part of our evaluation framework, we have decided to interview some of our Advisory Panel members about their work with Participation Cymru. They have a unique perspective as panel members, as they help to shape the direction of Participation Cymru but also work for organisations who access our services. First up is an interview with our Vice-chair Derek Walker, who is the Chief Executive of the Wales Co-operative Centre.

Derek Walker on Participation Cymru / Derek Walker ar Gyfranogaeth Cymru from Participation Cymru on Vimeo.

Derek was a great interviewee and believes wholeheartedly in the importance of participation. He said “Co-operative businesses seem to be doing pretty well at the moment compared to other parts of the economy, and partly that’s because they engage. They engage with their members – they speak to their members, whether they be workers, whether they be customers, whether they be stakeholders within their community, and as a result they can adjust their business. They can adjust their products in order to meet the needs of their customers, and as a result be better businesses – do better business”.

Back in 2010 we interviewed some of our other panel members about their roles, which you can also see here.

We have also interviewed Margaret Peters of the Countryside Council for Wales and Wayne Jepson of the National Leadership and Innovation Agency for Healthcare, which we’ll also share with you soon.

– Dyfrig

All Wales public services internet and social media survey 2012

I imagine that most of you have never heard of the Public Engagement Working Group, and that’s ok – that’s one of the reasons we started this blog, as there’s a lot of work that Participation Cymru are involved in that you might not get to hear about as it didn’t previously fit into the information processes that we had.

Essentially the group, which involves Welsh Government, Participation Cymru, NLIAH, Welsh NHS Confederation, WLGA, Wales Audit Office and Cardiff Business School, was put together to make sure that we’re approaching public engagement in the same way and that we weren’t duplicating each other’s work and that we collaborate where possible.

It was with all that in mind that the All Wales public services internet and social media survey 2012 was launched last September, so that we could get a good picture of what Welsh public service organisations are doing, to find out what we can do better, and to look at how we could spread the word about exciting work that is being done.

The results are in! And you can download them here.

We’re now putting the report together, which will hopefully be available by the end of the year. I have been tasked with putting together the sections on how organisations use Welsh language social media and how social media use is evaluated.

Whilst the report isn’t available as yet, there’s lots of good practice out there waiting to be tapped into! Rhodri ap Dyfrig sent me some very helpful resources around Welsh language social media. He also did a great presentation on this at a Welsh Government event that I attended in June. You can read the English presentation here, or watch the video above. There’s lots more information from the event available here. The Hacio’r Iaith project is also well worth a look at.

Esther Barrett of RSC has been a great help in helping me to look at confidence of Welsh Speaker’s. Her fantastic thesis ‘Somewhere along the line’ is well worth a read to better understand how important it is that services are provided in Welsh.

We’ve been quite timely in terms of evaluation – the UK Government Digital Service team has just written a really interesting blog on this. I’ve also found Helen Reynolds’ blogs on how we evaluate social media an inspiration – you can read more here and here.

We look forward to sharing the final report with you!

– Dyfrig

Participation Cymru Evaluation Toolkit

Evaluation of participation is vital to ensure that we learn lessons from how we engage the public and how we can do it better in the future. It’s something we often get asked about, as people want to know how they can show the value of public engagement.

Too often evaluation can be something that is tagged on to the end of participatory work. This toolkit can help you to ensure that you evaluate your work effectively and that you involve service users from the very beginning of any participatory work that you undertake.

This toolkit describes a four stage participatory process to evaluate engagement activities in relation to the National Principles for Public Engagement in Wales. The toolkit is intended to be used flexibly and users of the toolkit should feel free to adapt it to their own circumstances.

We’d like to thank Alain Thomas Consultancy Ltd, who have helped us to develop this toolkit, and also all the practitioners who have helped us to test the toolkit through various workshops, in particular the Forestry Commission Wales who piloted the evaluation workshop

We hope that you find it useful, please let us know what you think!

– Dyfrig