How industry awards are judged

StockSnap_7B52AC0C5F.jpg

Regular readers will know that I, along with a group of other people from Australia, New Zealand and the USA, recently judged the AGDA Design Awards.

Today I'd like to give you an insight into how the whole thing happens, from behind the curtain, because for me at least, awards and their inner workings have largely been a mystery.

It's important to note that this peak behind the awards curtain is simply my perspective - simply framed from my point of view. I have no doubt that others will feel differently about the whole thing.

Put plainly, I'm not a big fan of awards myself, having won a few over the course of my career they've now ceased to mean much to me. I've never met a client that has hired me for the awards I've won, and I don't believe I've lost a job for awards I haven't won.

On this last point I really can't be sure, but let's just go with it.

When I interview people to collaborate with me or work in my business, I pay very little attention to the awards listed on their CV. I'm more interested in the person.

I'm also not a member of any industry associations. Like awards, they've ceased to play a valuable role in my working life and professional development, although I do believe they're relevant, and I'm currently being interviewed to join a closed leadership network which will pretty much take my 'association dollar' as it seems to provide the type of value I'm seeking.

So with all of this in mind, you'll have a clearer perspective of the paint I'm using to illustrate this picture for you. It's transparent, honest and it comes from an authentic place.

-

For many years, I felt I wasn't anywhere near industry awards. I didn't know much about them, simply because I never saw myself winning any. The work that would be beautifully laid out in annuals was, I felt, beyond my comprehension and capability. It was there to be admired from a distance.

How on earth would I win clients like these? And if I win them, how on earth would I convince them to give me the free reign I needed to do 'award winning work?' These defeatist thoughts were a consistent presence in my mind within the first ten years of my career, at least.

Then two things happens which provided both context and answers to these questions:

  1. I won some awards
  2. I worked on the council/board of an industry association

Winning an award was a lesson in self awareness and a great bonding experience with the team I worked with. I was proud and honoured to have had our name read out on awards night and to receive the 'thing' that would sit in the foyer of our office until we deemed the date a little too far in the past to be worthy.

Volunteering my time on the industry association council was also a great lesson in humility and generosity. I realised that this industry I was part of, and was somewhat oblivious to, was actually filled with people who would happily donate their time to contribute to making a better, more equitable industry for all. I saw people who were both generous and humble in the work they did for the industry itself.

I spent seven years working on this industry council and during that time I made friends and acquaintances whom I'm quite proud to know. All of which have taught me something new and in their own way, have expanded my view of the world, as all good friends and acquaintances should.

-

As I boarded the plane for the quick, hour-long flight to Adelaide (a small, central city in the south of Australia) I saw some familiar faces getting comfortable in their seats. People that served on the AGDA council with me some seven or so years ago, and others I've met at various events. I realised that it would be a weekend of reacquaintance and familiarity.

In the Uber on the way to the hotel, myself and Andy Sargent from celebrated Melbourne design studio, SouthSouthWest caught up on the last few years. We hadn't seen one another in six years and had many stories to swap about our families, our businesses and balancing it all with what we wanted to personally achieve in life.

Mindfulness. Presence. More time with our children,  Yes, it was a deep and meaningful drive in someone else's car.

Once reacquainted with the rest of the judges, and bear-hugged from old friend and colleague Kevin Blackburn of Made By Big, we received our briefing for the weekend ahead.

Standing in an open, cold and vacant warehouse on the outskirts of Adelaide were a motley group of people who had flown from around Australia. We were cold (it was 7 degrees outside) and hungry.

The AGDA awards organisers briefed us on the criteria and how we would carry out the judging process.

As the briefing progressed, it became evident that Simon Mundy, Nic Eldridge, Steve Robbins, Shane Keane and their crew were running a tight ship. Their briefing was on point, organised and methodical. Questions were asked and answered quickly and professionally — I could tell these guys had done this before and it was evident that if we had any concerns, they were there to answer them.

I looked around the circle and saw a wide variety of design talent. Most people I didn't know (I don't make it a habit of looking too deep into my own industry) whilst others I had a fondness for from previous collaborations.

Nancy accompanied Jenny, Andy and I on the flight from Melbourne, Chris flew in from New Zealand, Wade Jeffereies and Let Sobierajski from New York, Kate from Tasmania, Kevin from Brisbane. Jo and Dan flew in late from Sydney and both introduced themselves to us as we were briefed.

Everyone was cold and smiling that smile we all smile when we meet new people and are keen to be on our best behaviour. We smiled over the criteria too.

The criteria was simple, if not light on. We were to judge all categories along the same three aspects:

  1. Creativity
  2. Communication
  3. Craft

It seemed odd to group all work along this framework — and this became evident during some of the more heated debates we held during the judging process.

The warehouse housed an array of long white tables — set out into  seven or so rows; this is where print-outs of the work would be displayed for us to review as we physically gave each a score out of 100 via our iPads. 

We were given iPads for the weekend. :)

We couldn't take them home. :(

The voting was simple:

  1. Look at the work on the iPad
  2. Walk over to the table to view it in real life, or a printed version of it.
  3. Read the supplied case study from the entrant (entrant names weren't available unless the entrant themselves included it in the summary of the project itself).
  4. Consider the three criteria and give the work a score from 0 - 100
  5. Move to the next project

Each judge also had the ability to select one piece of work as their 'Judges Choice'. I ended up choosing four — and was kindly reminded to only select one a week later.


That night we mingled over dinner and those of us who'se bed time is closer to 8pm than 3am headed off to rest before a full day of judging the following day.

There weren't any Crossfit gyms nearby. :( 


The next day, filled with coffee and quite a healthy assortment of goodies for breakfast we began to judge each category one by one as described above.

As each category was completed, we would then regroup around the long white tables where the crew had laid out the work in the order it was judged — from highest to lowest.

It was these moments I found most interesting.

This is when judges would openly comment on their surprise that a piece of work had rated so highly and talk about why they rated it low. The reverse also occurred, where some judges would debate why some work was literally on the floor, because it had rated so low that there wasn't enough room on the long white tables for it. 
  
These debates were healthy and everyone's opinion was respected and heard. 

In the weeks that followed this weekend, I feel this is what I missed the most — this healthy debate with a group of creative leaders who weren't precious about their work, or their opinions — all happy to listen and be convinced to the contrary.

Such a rare combination of talent.

As the judging went around this cycle of iPad-rating to debate and discussion, the weekend wound up and we flew home.

Although there was discussion from the organisers about 'an algorithm' which averaged up the scoring to remove the lowest and highest score (known as the Olympic method) — it was these debates and discussions which I believe, formed the core of the AGDA awards; and also illustrated the strengths, weaknesses and opportunities for the awards program itself.


During one of the initial discussions I mentioned that it was my interest and point of view to award work a high rating only if I felt it moved our industry forward in some way. I wasn't interested in a beauty contest, nor was I going to  entertain the idea that we were simply their to give our opinion and simply go home.

I felt an enormous obligation to ensure that whatever was deemed good enough to be printed in the awards annual was work that truly moved our creative industries (and in this case, design) forward. Be it a better positioning in the eyes of government, community and the business world — or an originality in the lateral, creative thinking that brought an idea to life.

It was a blend of this strength in positioning and originality in creative thinking that I was looking for.

— 
In another discussion which took most of one morning, we weighed on the relevance, and lightness of the criteria we had to work with. 

  1. Innovative expression of an idea (creativity)
  2. Execution of the concept (craft)
  3. Relevance to the intended audience (communication)

A couple of specialist categories had other aspects to their judging, but in general it was the three criteria listed here that guided all the judging.

In my mind this proved disjointed.

In the case of the Design Effectiveness category and Design for Good, it meant a heavy reliance on the validity (read: truth) of the rationale provided by the entrant. And the rationales were heavily biased towards how amazing the work was. Naturally.

At least those that were written at a level higher than grade five English — which lead me to my second concern and the highlighting of an opportunity.

The rationales which were supplied with each entry were so poorly written (generally speaking) that in some cases it was difficult to stop cringing or blushing. I'm not sure which I was doing more of.

(The opportunity? I'm going to put some effort into helping people write a decent rationale for an award entry. I don't want to put anyone through another weekend of reading...

"We were asked to ... and our strategic thinking was critical to the success ... the creative idea was applied across ... original, innovative and disruptive ..." *barf

You get the picture.

— 

Needless to say, the overall quality of the work was outstanding. The level of craft was strong in most submissions — and in a couple of them it was absolutely jaw-dropping.

There was one particular book I tried to fit into my jacket to sneak home with me, but this was noticed by one of the organising committee and I had to quickly put it back pretending I was joking.

In another case I was so moved by a piece of work that I mentioned to the group this very fact — that it was this type of work that moved us all forward. A strong application of craft, an obvious collaboration with the client and other partners, a merger of graphic, communication and technical know-how and an execution that was strong in strategic intent and simply beautiful.

— 
I came away with three key insights and I plan to follow up on each one of them:

A framework is needed

Although the quality was outstanding in some areas, in others it was at the other end of the spectrum, and I've already covered the weakness in rationale-writing.

A framework is needed that guides entrants to enter awards. 

  • Guides / courses on how to write a submission
  • Standardisation of methodologies to allow weaker entrants a guide to enter more complex categories (ie. Strategy)
  • Specific entry criteria that is unique to each category 

The judges were outstanding

In the week that followed, I missed my fellow judges. I missed the banter and the debates. I'm grateful I came away with a keen interest in staying in touch with them all. It left me thinking how influential the batch of judges were to the entries that eventually were awarded.

The association has an opportunity

AGDA in this scenario (and most industry associations) has such a huge opportunity to elevate itself and the industry, simply by the way it awards work and how it chooses to categorise and prioritise the awards it deems closer to its business goals.

put simply, if the association's goal is to improve and uphold the professionalism of the industry it represents in the eyes of our clients, then award work in a context that our clients will care about.

Submissions could be categorised into three levels:

  1. The top most tier awards work that celebrates the industry's collective values:
    • Gender equality: A single award for work that celebrates and impacts gender equality.
    • Creativity: A single award for work that shows outstanding level (98% and above) of creativity applied to solving a problem.
    • Effectiveness: A single award for work that actually worked, entered and witnessed by a third party (not the creative agency) and validated according to critical effectiveness criteria
    • Strategy: A single award for work that employs a unique and compelling strategy based on industry accepted framework for 'good strategy'.
    • Creative Leadership: A single award for an individual creative leader within the industry who has depicted each of the values in this first tier of award. Nominated by industry and voted by judges.
    • Brand Leadership: A single award for a client who meets similar criteria as the Creative Leader. Nominated by industry and voted by judges.
    • Innovation: Work that innovates a category
    • Business as a force for good
    • Young Creative Leader of the year: An emerging creative talent with less than ten years experience. Nominated by industry and voted by judges.
  2. The second tier, awards work that is strong in craft across various areas:
    • Corporate identity
    • Publication design
    • Product design
    • Packaging design
    • Book design
    • User interface design
    • Motion design
    • Environmental design
    • Experience design
    • Advertising concept
    • Advertising campaign
    • Innovative use of technology
    • Integration of media
    • Writing
    • Illustration

3. The third tier, awards work by people with less than five years experience (of any age) in the second tier categories. 
 

Naturally, there's more. But I'd be happy for your thoughts on this to begin with, 

Lipstick on a pig

The Ten-Year Hurdle