QAOrTheHighway 2014

QAOrTheHighway was held this past Tuesday in Dublin, Ohio, a city just outside of Columbus, Ohio. For the most part, this was a very regional thing. A lot of people I spoke with were from Columbus or very near by. That was sort of surprising considering the speakers (keynote and presentation) they managed to get. Despite the small/regional conference feel, there were a pretty good number of attendees present.


I was planning to get there early on Monday, hangout with friends and work maybe a little, but it turns out February is not a great time for flying. My layover was cancelled and I didn’t end up at the hotel till after 11:30pm. Luckily a few friendly faces were still up to chat and catch up despite the late hour on the day before giving a talk.

The day started at 6:30 am with breakfast in the hotel lobby and then rushing off to lean coffee. Lean coffee is a testing conference fixture at this point. People ask for it by name. This one was really enjoyable as usual. I left with some useful notes on preparing to give a conference talk. Oh, did I mention that I’ll be speaking at CAST 2014 in New York, NY?

The conference was kicked off with a keynote by Joseph Ours. The theme was some ways you could tell if you (the tester) were undervalued within your organization and some things you might do to change that. Joseph did a great job and presented some old ideas with a fresh perspective. Some things I thought were interesting was the way he thinks of testers as information brokers and also something he calls the OURS method: Observe, Understand, Review, Serve. One important thing to note is that Joseph was not the scheduled keynote speaker. Keith Klain was scheduled to talk but could not make it and Joseph did a fantastic impromptu keynote.

The first session I went to was The New Tester Skill Set by Matthew Eakin. There was some stuff in this presentation that I didn’t necessarily agree with such as an emphasis on documentation, an emphasis on tools, and very little about testing skill or how that fits into agile but I think Matt had some great points elsewhere. Mainly in emphasizing restricting WIP to be very small at any given time, and also something he mentioned about how a powerful test might tell you specifically where a problem is.

Session two was by JeanAnn Harrison on A Debate on the Merits of Mobile Software Test Automation. JeanAnn is a great speaker and conversationalist, I thought this was a fun talk. This was sort of a socratic talk, a lot of the content was posed as questions for the attendees to consider. I really enjoy this style. Some of the questions were around the idea of defining best, defining need, and asking if the project is worth the investment. She also mentioned that she doesn’t highly rely on domain expertise in new hired because that can usually be picked up quickly. I generally agree with that sentiment.

Session three was about Disintegration Testing presented by David Hoppe. This is another session that I thought was very good. The content was useful and engaging. David talked about the value of looking at problems in isolation as opposed to completely integrated products. He did this via stories about automotive repair, scenarios focusing on how a person might test their amazon home page, and also a live demo of a test program he wrote. The presentation has a little bit of everything and the attendees really seemed to respond to and enjoy that.

My last session of the day was by Scahin Mulik, Four Questions Every CXX Should Ask About Testing. This started off by modeling testing questions around Maslows Hierarchy of Needs. I thought it was quite interesting. The four questions Sachin came up with based on the hierarchy were: Is the software not doing what it is not supposed to do; It is secure, fast enough, reliable enough; Is it loved by its intended audience; Is it faster, cheaper. After this there was as a bit on testing measurements with absolutely no foundation and no reference for where the numbers came from. I wish I had written down the measurements he referenced. One I do remember was defect removal efficiency. There were also some measurements that were somehow supposed to represent industry maturity. This part left me really dissatisfied with the talk.

After this was a closing keynote given by Matt Heusser. Regretfully, I had to miss this to catch a flight back to Nashville.

I was in Columbus for about 20 hours total, not even a full day. If I go next year, I’ll try to hit the 24 hour mark.

This year and the next (2013 recap)

2013 was an exciting year for me as far as tester stuff goes. I jumped into a lot of new stuff that I hadn’t done before, that’s sort of my style. Being in an uncomfortable area seems like a good thing, it really pushes you (me) to explore and learn and grow.

I have been instructing BBST classes with AST for about a year now. I got an email from Michael Larsen one day asking if I had time to lead a class, oh and he was going to be away on vacation for the first week. I said yes of course. Why turn something fun like that down, really? That’s the brief story of how I started being a lead instructor for BBST classes.

This year marked my second CAST and first Test Retreat. For the second time, CAST was a formative event. The conference is really really good, what’s even better is getting to hang out with friends after. This is where the magic is. Test retreat is what lead to me getting to WHOSE2013. I did a session on tester skill lists, and it just happened that other people there were interested in the same thing. Who’d have thunk it. Test retreat also spawned a book club that I’ve been enjoying a lot. Next year I’d like to go to two conferences, I’m not sure what the second will be though. Maybe QAOrTheHighway.

I worked with Michael and JeanAnn a lot this year while contributing sessions for WeekendTesting Americas. Weekend Testing is a great thing for so many people. Participants get a free place to work with groups on new testing topics once a month, facilitators get a place to pursue whatever topics are interesting to them at the moment. All it takes is a good idea and 2 hours of your Saturday. I’m looking forward to more of these next year.

Writing is not something I’m particularly comfortable with, most times it is a struggle. So, because of that, I started writing more often here. Writing and reading a lot really help to make writing easier. I also started to do some writing outside of my personal blog. I’ll be writing one other place, at least for a few months, early next year. I want to thank Matt Heusser for that opportunity.

I’m not really sure what 2014 will bring, but whatever it is I probably won’t say no. There are a few really interesting in the air right now and if any one of them panned out I’d be pretty happy.

nasqpros Nov. meetup: Manual to automated shop

The quarterly Nashville Quality Assurance meetup happened this afternoon. The topic for todays talk was steps and organization can take to move from a primarily manual shop to a primarily automated shop. These sort of talks usually give me a pretty visceral reaction and the pages of (biased) notes I tool today probably reflect that pretty accurately.

I wanted to talk to the speaker but dominating his time didn’t seem appropriate for this sort of venue. So, as a result of that I’m writing this.

Here are some of the things that bugged me the most:

Not sharing your story
I have a hard time respecting talks that make sweeping generalizations and endorse specific ways of doing things without sharing failures, detailed context, and lessons learned along the way. This style of presentation gives me the impression that they are trying to share some sort of gospel. Phrases such as “this is the natural spot for X”, “This is how to get things right”, and “center of excellence” really drive that feeling home for me.

Not mentioning the gruesome details
The fellas that did this talk come from a Nashville company called Assurion. This company employs a varitable army of testers and developers to maintain the create the product. I feel like the fact that a large staff is needed to create and maintain this sort of high-volume automation was conveniently ignored. DSLs don’t magically get created, software tooling doesn’t happen magically. All of this takes significant time and development resources.

Rehashing folk wisdom
The folk wisdom is bullshit…whew I feel better now.
Here are some of the classics from today:
1 – X% of your tests should be automated.
Which percentage? Why? Why not the other percentage?

2 – Automation makes testing faster, more scalable, and able to be performed unattended.
No….just no. There is development time, maintenance time on the scripts, product, and framework, investigation for test failures, false positives, timing issues….the list goes on.

3 – The people writing code all day to make this sort of automation happen are testers.
The best I can give you here is a maybe. *Some* testers have the fluency and care enough about programming to do this. What I have seen most often though is a programmer(s) with heavy interest in tool smithing. Either way, these people will be writing code for the majority of their day be it test code or code to facilitate that.

4 – Test repeatability creates return on investment.
What is the investment? How did you measure that? What is the return? How did you measure that? Unwillingness to talk about this in terms of value as opposed to cost is baffling. If repeatability creates ROI, does that mean a test that is run once and never repeated has no return on investment?

5 – X scripts should be created per day per person
The reasoning for this aside from time accounting escapes me. I suspect it is linked to the fascination with ROI and ignoring value.

Tool fetishism
Spending time talking about all the tools you have used, all the tools you currently use, how much money you saved by switching from X to Y is uninteresting. Especially when you don’t frame the talk around what problems you were solving. Especially when you don’t show any examples of what you are actually working on.

This is a lot of stuff so I feel the need to mention a couple things. I don’t hate automation, it can be really useful in some situations. I really like the phrase tool-aided testing that has been gaining some traction lately in the community. This encapsulates the sort of automation described above as well as any other action we perform in which we use tools to amplify our ability to do something.

The speakers were charismatic and seemed like nice people. There, I said something nice 🙂

You can’t go home again

This train of though starts with Pete Walen posting an “overheard” tweet

I don’t have any experience making wine but I have made beer a few times. The thing about beer making and testing is that there are innumerable variables. If you buy a kit or a set of ingredients based on a recipe, the best case scenario is that you come *close* to what the recipe author intended. There are so many things to consider. The mineral content of your water, the yeast you’ve managed to procure, the harvest conditions of your grain and hops, temperatures at various points during the process.

This immediately brought to mind the phrase “you can’t go home again”. The general idea here is that experience changes your perspective in ways that you can’t account for. This makes it difficult to leave the place you grew up and then return and still call it home.

I think this idea applies to testing too and that made me think of a You can’t go home agian heuristic. If you have run a test once, chances are if you try to run it again you will be running a new test. The world is conspiring against your ability to do things in an identical way twice. Maybe you unintentionally take an alternate path through the software, maybe the timing of your actions are different, or maybe there are system changes that you are completely unaware of. Any of these things cause you to be performing a fundamentally different test and learning novel things about your software.