CAST2016 and AST Elections

You might already know about this, hopefully you already know about this, CAST2016 will be held in beautiful Vancouver, BC August 8th, 9th, and 10th. The theme this year is focusing on how testing helps propel software development projects forward. If you find yourself in a organization shifting toward more technical test techniques, this is the conference for you. As always, we will offer a full schedule with something special that you won’t find at most conferences, time to talk with the presenters and have deep conversations about their experiences.

TestRetreat, a peer conference, will be held the Saturday before CAST2016. TestRetreat is a little different. The attendees create the schedule and for the most part, run the event. If there is a specific topic you want to talk about, or a practical problem you want to work on in a group, this is a great forum. I have attended all but the first TestRetreat and will be there this year.

AST Board of Directors Elections

My first term on the AST board of directors will end at CAST2016. I served one year as VP of Education and Chair of the Education Committee, and my second year as President. Over the course of those two years, I was:

I will be running for my second, and last, term on the AST board this August. If I am elected again, I would like to focus on three things:

  • Continuing to help develop and promote our public education offerings like BBST, webinars, attending and reporting on academic software events.
  • Further decentralizing some aspects of work traditionally done by the BoD by fostering an engaged membership.
  • Discovering new ways to expand the reach of AST and good testing practices.

If you are happy with the work I have done so far, and what I might do over the next two years, I hope you’ll vote for me this August in Vancouver.

Downstream Bugs as a Quality Measure

I did something dumb yesterday. I got into an argument with someone on twitter. I usually try to stay away from that, it never ends well. The other person usually walks away thinking I’m dumb, and I leave not thinking much better of them. This time, the argument started with the claim that software development teams doing estimation with story points and task hours have 250% better quality.

Martin is referencing the information presented in this white paper —  ImpactofAgileQuantified2015 — where quality is defined in terms of the number of bugs found downstream. The Whitepaper was behind an email wall, and I balked all over twitter about the claim without having read first. Skin in the game is important. I registered, downloaded my copy, and will talk about their working definition of quality with some guidance from material I have read on measurement, as well as the work of Dr. Kaner.

Reliability and Validity

Meaningful measures have what we call reliability and validity. A measure is reliable to the extent that it can be performed many times by different people, and each person will get the same result. There are three types of validity, I am concerned with construct validity right now. Construct validity is the extent to which a measure (example: 250 downstream bugs reported over a period of 1 week) correlates to a theory or idea like quality.

Any kind of bug count as a measure of quality, including down stream bugs found per increment of time, has problems of both reliability and validity.

Reliability Example

Lets say that I release a software product to the market, customers are using it for a period of two weeks, and over two weeks 45 bugs are logged into a tracking system by the people using the product. Members of the development staff review those bugs over a few days and categorize 6 of them as feature requests, 3 of them as unable to reproduce, and 4 as not a bug. The remaining 32 issues are categorized by priority (how important it is to fix them), and severity (how bad the failure is).

The varying reports in this scenario point to unreliability. The customer(s) feel that they experiences 45 different problems while using the product. The development group, after reviewing those issues, feel that only 32 of those issues are bugs. Which party is right? There are also a few hidden questions to consider here:

  1. Did the customer experience bugs and not report them?
  2. Did every bug reported get documented and tracked? Maybe some got lost in email threads.
  3. If a different customer found more bugs over the same period of time, does that mean product quality is worse?

Validity Example:

Assume we have a product that is being used by 10 people over a period of two weeks. During that time, those 10 people report 20 bugs. In another experiment, that same product is used by 1000 people for two weeks and that group of people reports 200 bugs.If the first group that used the product was happy and wanted to continue to be paying customers despite the fact that group 2 found 200 bugs, does the product have good quality or bad quality?

Quality and Bugs Are Social

This points to the idea that quality is a social construct, it is a judgement people make based on their personal value system. A product can be valuable for one person, and terrible for another at the same time. The concept of a bug is also social and there is no standard definition for what it means. If you have ever taken part in a bug triage meeting where reported issues are getting routinely re-categorized as features, or working as designed, then you have experienced this first hand. Bugs are also not equal things. A server crash is not the same at a form failing to submit because of a special character, and those are not the same as a performance problem. Counting bugs and making a conversion from number to quality usually pretends that bugs are consistent things.If they were, we wouldn’t need descriptors like severity or priority to help the business make decisions about what to fix, and what to leave be.

Numbers are intoxicating. I can look at them and they seem to tell a clear, simple story. There isn’t one story there though, there are several. And, when I look at data, I am creating a story around what I see. That story may or may not represent a reality for the people using the product. I think it was Edwin Boring who said measurements are used to construct a reality for people that were not there to observe it. Whose reality are you constructing?

Inquiry or Control

I get that managers, directors, and C* level people have a need to understand how a project is going. They can not be there to understand the reality of building software, because they are busy tending to other aspects of the business. Sometimes measurement is how they get that feeling. I think using measurement as a place to begin asking questions is healthy. If a team I am working on puts a new product version into production for two weeks and gets 5 bug reports, and then two weeks later releases another new version and gets 100 bug reports the right thing to do there is ask “What’s going on here”? The change is probably a signal that something is happen with the development team, or something is happening with the customer, or maybe both. But, if we don’t start with a question, we’ll never know.

So, back to the moral of the story, it is not possible to measure a 250% improvement in quality based on a reduction in down stream bugs. It probably isn’t possible to measure a 250% quality improvement period. If there is some way, I suspect it would be a very complicated (and nuanced) anthropology experiment that would be too expensive for any company to bother performing.

Here are some of the books I have read (incompletely) that shaped the way I think about measurement.

measurement books

Recap Of WHaTDa And QA Or The Highway

Monday and Tuesday of this week (02/16 and 02/17/2015) had me in Columbus, Ohio for the Workshop on Teaching Test Design and QA Or The Highway. QA Or The Highway is a regional conference designed for software testers. This is the second running, and the second time it has sold out. This time to a bigger crowd than last year.

QA Or The Highway was great again this year. I didn’t attend a whole lot of track sessions, but the ones I did get to were high quality. Joe Ours puts on a good conference.

I did speak at the conference this year which was different from last. I did a talk on testing APIs that included and intro to REST, information on the testing VS checking dilemma, and a live demo with some code I wrote using frisbyjs. Public speaking is a new craft for me, and I’m growing into is and learning from others. Overall, I feel like it went pretty well.

WHaTDa was a one day workshop focused on teaching test design. It reminded me a lot of WHOSE, that is a good thing. We started the workshop at 9am with introductions; a little about who we were, what we were working on, and how we were planning to contribute to the workshop.

The goal, as with most Excelon Development workshops was to produce something useful to the testing community by the end of the day. Usually about halfway into the day, I start getting the feeling that producing something quickly is completely impossible.

After lunch, we split into groups and focused on an exercise we wanted to build and contribute to the wider testing community. I paired, or maybe grouped is a better word, up with Paul Harju, Megan Studzenski, and Dwayne Green.

A new Test Challenge

The 4 of us built something, and we hope it is useful to you.

We build a testing exercise based on a program that determines whether the text you entered into a field is a palindrome or not. It sounds simple, and it is, but there are a number of ways you can frame this simple program into a test exercise.

Here is the Palindrome Challenge. Feel free to use it however you like with credit to the authors.

Here are some framing examples you can use. This is what we came up with during the workshop, there are of course many other ways you can run the challenge.

For the person giving the challenge: you have to address what the scope of the exercise should be.

Here is one possibility:
1 – Design a test strategy / how would you test this (actually write the strategy down)
2 – Test for a couple of minutes using strategy
3 – What did you find? Did the stuff you found matter? Why?
4 – If you had more time, what tests would you run?
5 – Debrief

You could also do a survey of test techniques.

— Domain
— Function
— Risk
— Load
— Security

Maybe run the exercise a couple times, and see how the test strategy differs based on the identified technique.

Seeding test ideas:
I know there is a bug in X area, how would you test for that?

Megan’s Gambit:
For any given exercise, you can make another exercise by having the student identify what they were doing and why.

Test exercises are everywhere, framing and scoping are the hard part.

Have fun!

I’m running for the AST board of directors

The AST board of directors election will be happening this year between August 11th and 13th during CAST2014. I will be running, along with 6 other well-qualified candidates, for one of the four open positions.

Why I am running:

AST has been a significant catalyst for me in personal and professional development. I first discovered this group through a colleague at a company around 2006 or so. He was taking Bug Advocacy at the time, and I could see very apparent changes in the way he was working that really made sense. I wanted in on that. That lead me to signing up as a member, taking Foundations, and then Bug Advocacy, and then Test Design. It was a rabbit hole that lead me to CAST and meeting other people that genuinely cared about their work. I want to return the favor that AST has given to me by serving the community that has done so much for me.

If elected, here is what I will focus on:

These are topics dear to me, but also, I think they are very important to the continuing development of software testing as a craft.

  1. BBST is hugely popular, especially Foundations and Test Design. Those two classes are consistently booked 100% full well before the class and have wait-lists for folks interested in taking that session if a registered student is unable. I would like to grow BBST to support this demand and begin offering those two classes more frequently. This means we need more assistant and lead instructors.
  2. There is a clear market demand for testers with a technical skill set. I would like to begin developing a class to help testers meet that market need, but also to improve their ability to test software by developing technical skill and understanding. The idea of what this class is is still being conceived, but I imagine programming concepts, databases, UNIX/Linux, shell scripting, performance and load testing concepts, and Chrome developer tools / Firebug are all fair game. This could be a survey course in the same way that Test Design is a survey course. If you are interested in this or care about the development of this material, please get in touch.
  3. WHOSE is an AST sponsored event dedicated to studying and advancing self-education in software testing. The first of these was held in Cleveland, OH December 5 – 7 2013 and the result was the first version of a skills book which will be published  shortly via AST. The skills book is intended to be a living document that is updated by practicing software testers. I want to see this become a yearly event with a concrete outcome each time. A yearly revised skills book could potentially result from this.

If these initiatives are important to you, vote for me and we will make these a reality.

Stepping up as EdSig Chair for AST

About a month ago, Michael Larsen sent me an email telling me he was planning to step away from his role as Chair of the Education Special Interest Group for AST. Michael was explaining that he had served for three years now, and thought his time had come to step back from EdSig in order to put more energy into SummerQAmp and the many other interests that he wants to pursue.

I read through the email, not feeling great about it. Michael has done a fantastic job and I’ve enjoyed working with him as an instructor in BBST courses that continue to exist and thrive because of EdSig. When I got to the bottom of the email, Michael asked if I would be interested filling the role that he was leaving.

Much like Michael, when I’ve been asked to do something new that stretches a fair bit past my skill set, I have the complete inability to say ‘no’. This new role will be an experience to learn and develop new skills but more importantly, to serve a role that I think is useful for other people. I am passionate about the service role of the tester, and I think the education mission of AST is well aligned with that.

Michael posted this to his blog and the AST blog (and the relevant AST discussion groups) this past Wednesday making this change official. While Michael is transitioning out of his role as EdSig Chair, he will not be leaving AST, or BBST, or SummerQAmp, or any of the programs he has done a great job developing and fostering. I’m sure Michael will be as present as he ever was, I’ll certainly be bugging him for advice and counsel occasionally.

If there are things you would like to see AST do that help support the mission of education for software testers, please let me know. I’d love to hear about authentic problems of practicing software testers and find ways for AST to help. Following in the footsteps of MIchael, and Cem before him, is a tall order. I’m looking forward to serving to the best of my abilities.

If you are going to be at CAST2014 in NYC this year, maybe we can talk in person.

Lean Software Testing: The best 3 day conference I’ve been to

Last week I went to the best thee day conference I have ever been to, and got paid for it too! I flew from my home in Nashville to Denver, CO to assist Matt Heusser in teaching a three day class focused on lean principles in software quality and delivery.

Lean Software Testing is a one, two, or three day class created by Matt Heusser. The class combines years of research and experience on applying Lean concepts in software development with software expertise in practical problems of software quality and delivery. The result is a class that can help your test and quality team do their work smaller (we are talking about Lean, right), better, and faster. This is not an introductory class about software testing, but testers of all experience levels can certainly find value in the material and teaching. We have experience teaching this class to experience levels ranging from the newest interns all the way to the most engaged experts in the test community with 20+ years of experience.

A few weeks before class starts, Matt and I will have a discovery call with your team where we will mention a few of the concepts covered in the class and see what your teams experience level with the material is. Also, we will ask a few probing questions to discover what we like to call critical success factors: these are the things you must absolutely take away from the course to consider it a success. We want to provide a class that is timely and relevant to your team, this can help get us there.

Day one started immediately with a group exercise focused on team dynamics. This is a great way for people that may not know each other to get acquainted and also jumps right into valuable course work from minute one. After this we do a formal introduction, meet the attendees, and  explore what they want to get out of the class. The rest of the day is spent on studying and doing exercises on team dynamics, test planning, and comparing scripted and exploratory test approaches.

Day two is spent alternating between theory from Lean and exercises crafted to show exactly how the theory works. Where Agile training will tell you what you should do, we will explain exactly why approaching work in specific ways will help you deliver better software faster. This is practical information you can take back to work on Monday and use to provide real value. Value that your customers will be happy to pay for. Attendees will have a set os tools, but more importantly, they will know why those tools work so they can apply them across a variety os situations in your organization. We end the day with a one hour facilitated problem session where attendees discuss problems relevant to their environment and the group helps discover solutions.

Day three is emergent, we take lessons from the last group session on day two and mash them together with the theme of our course. We pull from our bag of LST content and present material that that is immediately relevant to your context. Some of the popular topics for day three are: tactics to help your group do regression testing faster and in a way that helps the business, a survey of test design techniques, test automation for non-technical people with live examples, and performing skilled bug reporting. There is much more, we are happy to customize content to the needs of your group.

Once the class is done, your testers are not left high and dry wondering how to apply everything they just learned. We provide lifetime support for all graduates of LST. Once class is complete, students are given access to the Lean Software Testing google group where they can talk about questions and real work problems and get support from LST instructors and graduates alike.

I can’t wait for the next teaching opportunity. If you are interested in getting Lean Software at your organization, please get in touch with Matt Heusser or myself.

Book review: This is Lean

This is Lean is a high level book about lean, the philosophy of lean, not the methods and tools. The book was written by Niklas Modig, a Swedish researcher. He spends most of the book framing lean as an operational business strategy. I’m sure there is a large amount of bias from his academic experience leading to that conclusion, so I wouldn’t mind reading another book that frames lean some other way. If you’re aware of a book like that, let me know 🙂

If I had to summarize the book in a few words, I’d say it is about two types of efficiency; flow efficiency and resource efficiency. Resource efficiency is the degree to which your resources are utilized. Are your machines running all day every day? Are your programmers spending 8 hours a day writing code? Then you have 100% resource efficiency. Flow efficiency is about getting whatever you’re making done as fast as possible once cash exchanges hands. I’m am not really sure what would constitute 100% flow efficiency though, maybe a product being completed with no wait periods.


Continue reading

I’m speaking at CAST2014

There it is, I’ll be speaking at CAST2014 in NYC this year. I’ve mentioned on twitter that I was accepted to speak but haven’t actually written about it yet.


Most simplistic measures for software productivity and quality fail, for reasons you don’t need a conference talk to explain. The problem is how to do better than that – how to “plus one” software measurement, or, at least, to choose measures and frame them in a way that will do more good than harm. Studying a little social science, specifically how social scientists do qualitative research, and measurement problems can help. Justin will talk about the development of qualitative research as a field of study, common problems with measurement in the software world, and some ideas from Lean. You will take back some tools to help you tell a more meaningful story to your business.

I’m in the session group before the last keynote on the last day. This feels a little ominous. If you include the tutorials, and TestRetreat before that, CAST is an intense 5 day marathon of deep discussions on testing and for me a bit of introspection. I will have to be sure to reserve some energy for the talk, especially since this is my first at a real conference, especially since I will be live cast over youtube. I have talks for the local testers group, and facilitated events and whatnot, but for me this is the big-time. CAST is the place.

The theme of this years conference is the art and science of software testing. My talk is themed around measurement. Mainly how it has been traditionally used in our craft, some of how it is used in the social sciences, and a bit on how we can make measurement a useful thing for software delivery and delivering value to the folks that pay for it. Measurement is a difficult problem, but I feel like talking about problems without offering alternate ideas to explore or solid solutions is not all that helpful. I’m hoping we can leave the room with some ideas on making the testers life a little bit better.

Hope to see you there!

Book Review: On Looking

On Looking: Eleven Walks With Expert Eyes is a book by Alexandra Horowitz on observation. Seeing, hearing, feeling, smelling, all that good stuff. More than that though, On Looking is a book about perspective.

The book is broken into chapters where each chapter is a narrative from the author describing a walk she took around a New York neighborhood with some expert accompanying. The experts range from her toddler, to a geologist, to a person studying pedestrian traffic, to a person studying urban wildlife.

Each chapter stands well on its own. The stories were entertaining and the characters Alexandra walked with were endearing.

Certain characteristics of each expert jump out. Alexandra’s child likes to treat inanimate objects as if they are alive. Cars that aren’t driving around are sleeping, and when leaving steps you have to big them a farewell. The guy studying urban wildlife tended to notice not just the critters but also traces of their existence when they were not around. The man studying pedestrian traffic noticed things about how people walk (hints of illness) that would be invisible to most.

This is such a nice parallel to testing. Testing is sometimes thought of as a team sport. That can be interpreted as meaning that testing isn’t exclusively the job of people that self-identify as a tester. Developers, product folks, support team, and everyone else involved in creating the product, can test.

I had always taken that for granted and didn’t think about the reason it makes sense for multiple types of people to test. At a glance, it seems like the expertise is the important thing. Product people might have a tendency to test the product from a customers perspective, and developers might have a tendency to test in a technical or programmatic way. When I refer to expertise, I mean that in the Collins and Evans sense. But after thinking on this a while, I think there is more to it.

All the biases developed while developing an expertise shade how a person observes the world. This happens to the extent that you do things in an automatic way (system 1). A person studying typography doesn’t force themselves to notice the typeface on signs, it just happens because of years spent intently looking at typefaces. This has biased them to notice text very quickly and in a very specific way.

Tester friends have often commented on their inability to turn off the critical thinking that they have developed to make them a good tester. Developing this skill has changed how they experience the world and get along day to day.

When executing scenario tests, a tester has to role play. They put themselves in a mindset to think about testing in terms of how a user might behave in the software and the benefit they might get from its use. A product manager is biased to think like this because of their daily focus on people using software and the time spent developing their product management skills.

Back to the review. Overall, I enjoyed the book. I think this is a great text for deeper thought on expertise, bias, and how people experience the world in different ways. And, I think there are a lot of ideas that can be directly applied to testing.

Deciding if test retreat is right for you

2014 will mark the third run of TestRetreat and the second time I’m attending. The first was last year. Yep, two years in a row. I had a fantastic time last year and left with some useful tools. I’m confident this year will be no different.

I wanted to share some reasons here I think you should consider attending this event.

Presenting and conferring
TestRetreat is a small group of people, last year I think attendance was capped at 30 or 35 people. There is no pre-set agenda, this is a conference for the people, by the people. What I mean is that most every one that attends TestRetreat presents on a topic or facilitates discussion on something. This isn’t required, but it is encouraged and participation is what makes peer conferences like this great. Last year, I facilitated a discussion on testing skill. This was the first time I had put myself in front of a group and talked about something. If you are interested in dipping your toes into that pool, this really is a safe place to do that.

People you will want to talk with
This peer conference attracts people that care deeply about testing and developing their craft. Being held on a Saturday before another great conference means that people in attendance are people that are serious about being there. If you want to spend time next to writers, presenters, and independent testers, this is the place. You may see this appealing to authority, and it is a little bit, but it is a great group of people and you won’t find them all in the same room very often.

The conference below the conference
TestRetreat is an event preceding CAST2014 in New York. There are a few events surrounding CAST each year: lean coffee, tester games, tester competitions, after hours conversation over beers. CAST is a great conference. For me it is the conference, but it wouldn’t be the same without all the events just below the surface. TestRetreat is the one you don’t want to miss.

Great things begin at TestRetreat
Being present last year lead to me attending WHOSE, doing a lot more writing, and now speaking at CAST 2014. What great things will you do that get started here?

I hope to see you there!