Past QA SIG Meetings

from July 14, 2010

Lightning Talks

Presented by 8-10 of your fellow QASIG participants

The format was the "lightning talk" where each of the speakers knew that they only had five minutes to present their idea (one at a time). Each speaker had a severely limited time-frame, but could speak about anything related to testing in that time – a new idea for status reporting or measurement, a lesson they learned on a project, an experience report about a new technique they tried, or a general test principle they want to share.

Lightning Talk Speakers, Topics, and Slides (where submitted):

from May 12, 2010

Presented by Jim Benson, CEO of Modus Cooperandi

What is QA?

When we seek Quality, we are looking for something more than "Does it break if I do this?" We are looking for improvement, for ascendancy, for something better.

Scripts, exploration, observation.

We look for patterns, we look for exceptions, we look for jarring transitions.

In Personal Kanban and Lean, we look for patterns, exceptions and transitions in the creation of value and the flow of work. Think of it as QA for your workload and your processes. QA for your QA.

MetaQA.

On the 12th of May, we discussed how Lean techniques like Personal and Organizational Kanban can be applied to the highly variable workload of a tester. How understanding the variation in work and making judgments based on this understanding will help testers, managers and clients schedule for testing and the impacts on QA of unreasonable demands.

We will, at last, be able to define "unreasonable" and provide meaningful service level agreements.

About our speaker: Jim Benson is a collaborative management consultant. He is CEO of Modus Cooperandi, a consultancy which combines Lean, Agile Management and Social Media principles to develop sustainable teams. Clients include the United Nations, World Bank, Microsoft, and NBC Universal. Jim created the Personal Kanban technique that applies lean thinking to individual and small team work. He blogs at evolving web and is @ourfounder on twitter.

from March 10, 2010

Presented by Lanette Creamer

We may think we are ready to move onto new and innovative features. However, if we do not deal with the past, it can easily come back to haunt, slowing down new projects, and robbing our testing time unexpectedly, often to the point that testing becomes the bottleneck that slows innovation to a crawl.

For those of us who work on software that already exists, exciting new functionality and improvements are the main things that drive upgrades, as well as compatibility with new platforms. However, if end users cannot trust the quality of the legacy features they rely on, they will be reluctant to upgrade, or worse, your new versions will get a reputation of being unstable - harming overall adoption. In some cases users may downgrade their software to an earlier version because they are unhappy with the quality of the newer release or request an earlier version they feel is reliable.

This paper presentation is about the subjective and difficult part of testing which has no provable mathematical correct answer. It is about risk management, test planning, cost, value, and being thoughtful about which tests to run in the context of your specific project. The discussion covers identifying and reducing test case bloat, when it can be done, who does it, along with a few examples used in practice. Further, it will cover one untested but under test theory, experiences shared in significantly reducing test cases to cover more than three times the applications when the test team reduced from sixteen to four testers. When facing increasingly complex software and growing software, we must balance testing existing features that customers rely on every day with new features and interactions. When balanced in a sensible way, the best of the legacy test cases can be maintained, using existing knowledge to reduce risk as much as possible.

About our speaker: Lanette Creamer is a test lead with 10 years industry experience ranging from product feature testing on early versions of InDesign to leading collaborative end to end workflow testing across the Adobe Creative Suites. Most recently Lanette has been testing on an agile team that is automating the software production process between a product build and actual shipment, to be used on all Adobe shipping products in 2010. Lanette has two published technical paper "Testing for the User Experience", voted best paper by presentation attendees at PNSQC 2008 and "Reducing Test Case Bloat", PNSQC 2009. Her magazine article "9 Ways to Recruit Testers for Collaborative Test Events" was published in Software Test & Performance Magazine, in the January 2010 issue. Lanette started writing a testing blog at http://www.testyredhead.com in 2006 and has been writing and collaborating with the online testing community non-stop since.

from January 13, 2010

Presented by Detective Brian Stampfl

Detective Stampfl from the Seattle CSI unit will be joining us to discuss the business of Crime Scene Investigation - a discipline not unlike bug investigation, with processes often used in software testing. We'll talk about parallels and get some insight into what Crime Scene Investigation is really like, straight from the source.

Detective Stampfl will cover the following:

• The make-up of the Seattle Police Crime Scene Investigation's Unit
• The type of crimes that they respond to investigate
• Steps necessary to properly document a crime scene
• The differences between the real world and how investigations are portrayed on television, and a current and look into the future of Forensic technology

Detective Brian Stampfl has been involved in law enforcement for over eighteen years. He began his career in 1991 in Southern California with the San Bernardino Police Department, and later joined the Seattle Police Department in 1995. Since joining SPD, Detective Stampfl has worked in patrol and was a field training officer, tasked with training new officers who had just graduated from the academy. Detective Stampfl went on to become an academy instructor and when the opportunity arose, he acquired the title of detective and worked for over three years in the Sexual Assault and Child Abuse Unit. Then in 2005, the opportunity of a lifetime presented itself as the Seattle Police Department sought to create its own Crime Scene Investigations Unit. Detective Stampfl was one of seven detectives chosen not only to staff this new unit, but to assist in the building of this unit from the ground up, which started simply as an idea. In addition to hours of training associated with Crime Scene Investigation, Detective Stampfl is a graduate of the National Forensic Academy in Knoxville, Tennessee and is an adjust faculty member of Seattle University where he teaches a course in Crime Scene Investigation.

from November 11, 2009

Presented by Jon Bach

Whether you are a tester or a test manager, Jon Bach assumes you have no time to do the things you want to do. Knowing that even the things you absolutely must do today is its own list of competing priority 1 items, he has ideas on how to cope. They are truly "half-baked," as in, they are still in the oven, still being tested. It is not about time management, it is about where you focus your energy - you and your team. This presentation will share some of his half-baked ideas that seemed to be valuable for him as a test manager, even if some of them didn't really work. His ideas are meant to solve common problems in test execution, reporting, measurement, and personnel - all of which are low, or no, cost and relatively easy to implement.

Jon Bach has been a software tester for 14 years, and is currently a freelance consultant for Quardev. He speaks frequently about exploratory and rapid testing, and is the co-inventor of Session-Based Test Management - a way to manage and measure exploratory testing. See his blog at http://jonbox.wordpress.com

from September 9, 2009

Presented by Harry Robinson

Remember the old MacGyver TV series? Each week, the hero solved difficult problems by combining his knowledge of applied science with everyday items such as baking soda, paper clips, and chewing gum. Test automation is currently in a rut and needs some of that outside-the-box thinking. When most testers think about automation, they think of regression scenarios and capture/replay tools. However, regression testing is the weakest, most insipid form of automation around! Regression tests are costly to maintain, rarely find important bugs, and are often obsolete by the time they are checked into the test case manager. If you are interested in approaching test automation from a fresh direction, join Harry Robinson in this innovative session and learn to create "freestyle test automation" from simple elements such as idle test machines, free programming tools and, yes, a bit of creative test thinking.

from July 8, 2009

Improving Testing Collaboration in Agile Development

Presented by Bruce Winegarden

The goal of this program is to stimulate a dialog on how QA testing professionals can best collaborate with agile development to form a quality continuum for enduring sustainable software.

Example cases from two different software products will give contrasting snapshot of before and after agile pictures. The "before" case used traditional waterfall approach with extended code, test and fix cycles. The "after" agile case used extreme programming practices such as: test first (TDD), acceptance testing (FIT), and pair programming. These examples will outline the range of QA testing strategies employed and setup discussion around topics such as:

1. Why moving testing earlier in the process and making it more integral to guiding the development effort is a good idea.
2. What quality strategies we can learn from in other industries such as statistical process control from Lean/SixSigma.
3. How can the QA testing role better complement agile development methods such as SCRUM.
4. What concepts do agile development and Exploratory Testing have in common and how can we leverage the similarities and uniqueness.

About Our Speaker:

Bruce Winegarden has extensive business and technical knowledge for software and systems development, product management and innovation. He enjoys guiding teams of people from different disciplines discover sustainable practices that smooth and accelerate their delivery. As principal consultant at Eagle Eye View, he delivers consulting, modeling, training, and coaching services applying Lean and Agile principles that enable organizational changes to: streamline product development, improve business workflows, deploy more meaningful and effective information technology.

from May 13, 2009

Data Governance - Protecting Proprietary Information from Unauthorized Release

Presented by Det. David Dunn and Seaton M. Daly III, Esq.

The most valuable asset to any organization is its "information." Quality Assurance professionals, with an emphasis in software development, critically examine a software program for functionality, compatibility, ease-of-use, stability, and integrity. This allows the QA professional to garner an understanding of a client's (or employer's) software strengths and weaknesses. But once that information is shared with a third-party, what controls or mechanisms are in place to ensure the security of that information? Are contractual provisions enough? How is the integrity of the QA industry affected by a data leak? How can the testing industry align its goals with the goals of its client(s)?

According to two of the largest security software firms, cyber-related crimes are now more lucrative than the international drug trade. While a virtual cyber-attack of a 9/11-style magnitude has not yet occurred, the need to adequately safeguard mission-critical data places network infrastructures on the frontlines. What is law enforcement doing to protect businesses from cyber-extortion and theft of proprietary information? Learn what mechanisms mitigate the chance of an unauthorized release, or disclosure, of proprietary information and how you can help investigate cyber-related crimes.

About Our Speakers:

Det. David Dunn, (Detective, Seattle Police Department, U.S. Secret Service Electronic Crimes Task Force) - Det. Dunn has been with the Seattle P.D. for 8 years, and working computer forensics and electronic crimes for 4 years. He is currently assigned to the U.S. Secret Service to assist them in investigating network intrusion and computer hacking cases at the regional, national, and international level.

Seaton M. Daly III, Esq. (Law Office of Seaton M. Daly III, P.L.L.C.) - Seaton Daly has extensive experience working in the Information Protection Industry, both as an attorney and non-attorney, implementing disaster recovery plans, and performing risk assessment audits, in both the public and private sectors. Mr. Daly owns a General Corporate Transaction and Privacy law firm in downtown Seattle.

from March 11, 2009

How To Be A Test Pilot

Rob Bach has an extensive background and expertise in the world of aviation. Parallel to his current job as a pilot for a major airline, he studies Human Factors and Ergonomics in the cockpit where they intersect with post accident/incident investigations and applying those results into preventative programs. Rob will explain a Technicolor thought process for a what is basically a black-and-white world that has many parallels to the domain of software testing. He will demonstrate why he believes that it is not discipline, but critical thinking skill that matters most in this mostly procedural world. This interactive workshop will be an exercise for the nimble brain.

About Our Speaker: Rob Bach has been a licensed pilot and flight instructor for 32 years in all types of flying machines, powered and unpowered. He is a member of the Antique Airplane Association, a member of the Experimental Aircraft Association, has recently restored an antique airplane called a Pietenpol, and considers himself an inventor and philosopher.

Click to see Rob's Slides (PDF)

from November 12, 2008

Color-Aided Test Design

Color coding theory is a method of test creation which aims at giving value by shaping, finding relevance and organizing tests based on over arching and broad testing concepts. By categorizing tests via high level test concepts (designated by color) testers can quickly and easily focus their aim and maximize reliable test coverage in a collaborative method to expose risks in the test plan. In this interactive session, Ben Brodsky, a QA Engineer at LexisNexis, talks about his experiences using a method pioneered by testing expert and McGill University professor Robert Sabourin to use color to aid test design and reporting.

About Our Speaker: Ben Brodsky is a software tester at LexisNexis in Bellevue. He started software testing in 2005 with a background in Applied Philosophy and Criminal Defense Investigations. He found that software testing contained many of the same skills that are used in the legal domain. Since moving into software investigations, he has constantly been on the hunt for ways to further the exploration and effect of test design.

Click to see Ben's Slides (PDF)

from September 10, 2008

Acceptable Acceptance Testing

This is the tale of a team of software professionals at Microsoft patterns and practices group who wrote a book on software acceptance testing. Jon Bach was one of the writers, ensuring that the project team used its own methods so the book would be acceptable to you, the reader. To develop the book, the team employed key ideas of agile projects - creating a backlog using story cards, working in short iterations, exploring requirements and expectations, building customer trust through iterative acceptance, and staying connected to the customer community through frequent preview releases, surveys, and interviews. They created a heuristic acceptance testing model for knowing when they had reached enough "acceptability," but the book is only in Alpha release so they won't really be sure it's acceptable until they talk to potential readers like you.

This will be an interactive talk where Jon will not only describe the process and content they produced, but will facilitate a group activity to discover how acceptable you might find the content of the book and if you apply acceptance testing methodologies in similar ways.

To get a free online preview of the book, go to: http://www.codeplex.com/TestingGuidance/Release/ProjectReleases.aspx

from May 14, 2008

Secrets of a Buccaneer Tester

Education has been the key to my success. Early on I chose not to outsource it. I quit high school in 1982, and have lived without the benefits and drawbacks of institutionalized learning ever since. There are no undergraduate degrees in software testing. The certification programs and graduate degree programs I've seen are disturbingly content-free. The leaders in our craft are all self-educated to some degree.

I develop and sell a methodology for software testing that I call Rapid Software Testing. At the time I created it, my ideas contradicted most of what was conventional wisdom (and still do, although rapid testing concepts are somewhat less controversial, today). This talk will present, not my testing methodology, but rather some of the methods by which I arrived at it. I believe these methods are fundamental to self-education on the job, and that self-education is fundamental to competing in a difficult and diverse economy. You can benefit from these methods even if you completely disagree me about testing and even if you are Harvard educated thrice-over and wear powdered wigs.

About our speaker: James Bach is a local self-educated software tester and author of the unpublished book How I Learn Stuff: Secrets of a Buccaneer Scholar, as well as the published book Lessons Learned in Software Testing.

James' Slides (PDF)

from March 12, 2008

Distributed Agile

Presented by: Joy Shafer, Senior Test Lead, Microsoft

This presentation reported on the challenges and successes Joy's team experienced while developing software as a service with a globally distributed team using Agile methodologies. We concurrently developed two small, first-version, online services using a development team that was located in Redmond and Moscow, and a test team that was located in Redmond and India. We encountered many interesting and difficult challenges, but were able to successfully overcome them and release high quality software on time, delighting our stakeholders. This experience report highlights our learnings, focusing on several critical success factors, such as well-defined processes, cultural awareness, the importance of relationships and communication, and continuous integration.

About our speaker:

Joy Shafer has been working at Microsoft for three years. She is currently a Senior Test Lead in the Mobile, Voice and Partner Services team. Prior to Microsoft, she tested and managed testers at Quardev, NetManage, STLabs, and Aldus Corp. She also was her own boss for many years, teaching and consulting in the area of software testing methodology, providing services to companies as well as colleges. Joy is an active participant in community QA groups and occasionally speaks on the topic of software quality. Joy holds an MBA in International Business from Stern Graduate School of Business (NYU).

Joy's Slides (PDF)

from January 9, 2008

The Strange World of Problem Solving

Presented by: Dr. John Medina

Regardless of what you may have heard, brain science has very little to say to the practical worlds of business and education, problem-solving included. The very little it can address, however, suggests that current practices haven’t moved very far from their 15th century roots. Recent findings about human memory, sleep cycles and physical activity actually suggests a few things business people and educators might try to bring their learning environments into the 21st century. And no, its not what your family doctor usually tells you.

About our speaker:

DR. JOHN J. MEDINA is a developmental molecular biologist focused on the genes involved in human brain development and the genetics of psychiatric disorders. He has spent most of his professional life as a private research consultant, working primarily in the biotechnology and pharmaceutical industries on research related to mental health. Medina holds joint affiliate faculty appointments at Seattle Pacific University, where he is the director of the Brain Center for Applied Learning Research, and at the University of Washington School of Medicine, in its Department of Bioengineering.

Medina’s books include: Brain Rules (March 18, 2008), The Genetic Inferno, The Clock of Ages, Depression, What You Need to Know About Alzheimer’s, The Outer Limits of Life, Uncovering the Mystery of AIDS, and Of Serotonin, Dopamine and Antipsychotic Medications.

To see John in action, view his short videos on www.brainrulesbook.com

from November 14, 2007

Please Break My Stuff!

Presented by: Brian Snyder, Vice President of Delivery Technology at Worktank, an integrated Advertising agency

A talk about how a developer learned to sleep well at night by looking at testing as a partnership, not a medieval battle. Hopefully giving you the tester insight in to one developer's mind, and to let you walk away with a better understanding of how to help uncover and report the nasty, lurking bugs that sometimes escape a developer's view.

About the Speaker: Brian Snyder is Vice President of Delivery Technology at Worktank, an integrated Advertising agency. Brian has had a lifelong passion for technology, gadgets, and general geekery, putting in time at Microsoft before moving to the ad agency life. He now leads a small but effective team of developers with a focus on Web sites and services. He is responsible for the company's testing strategy and implementation, and enjoys the challenge of finding ways to carve out time for testing in an insanely deadline driven industry.

from September 12, 2007

SBT Lite: Components of Session-Based Testing

Presented by: Sam Kalman, Director of Quality Assurance at OverTheEdge

Session-Based Test Management (SBTM) is an approach for managing and measuring exploratory testing. Session-Based Testing Lite (SBT Lite) is a streamlined, customizable version of SBTM. This talk will explore all the SBT Lite elements or "Components" which can be mixed and matched to optimize the test team's time. This talk does not require experience or familiarity with SBTM, as every Component will be detailed with real-world examples. This talk is intended for test leads and engineers who currently perform exploratory or ad-hoc testing on their current projects and are looking for ways to improve their testing processes or increase their accountability.

About the Speaker: Sam Kalman is the Director of Quality Assurance for OverTheEdge, creators of the Unity 3D game engine. For more than two years he worked at Quardev Laboratories with Jon Bach, co-creator of SBTM. During this time he performed both test engineer and test lead duties on two separate SBT Lite projects. Sam is currently active in using SBT Lite for his testing and technical writing duties.

Session-Based Test Lite paper

Sam's Slide Deck (Zip file of information and individual slides)

from May 9, 2007

Pre-STAREAST Presentation:
Top Ten Tendencies that Trap Testers

Presented by: Jon Bach, Manager for Corporate Intellect and Technical Solutions, Quardev, Inc.

A trap is an unidentified problem that limits or obstructs us in some way. We don't intentionally fall into traps, but our behavioral tendencies aim us toward them. For example, have you ever found a great bug and celebrated only to have one of your fellow testers find a bigger bug just one more keystroke away? A tendency to celebrate too soon can make you nearsighted. Have you ever been confused about a behavior you saw during a test and shrugged it off? The tendency to dismiss your confusion as unimportant or irrelevant may make you farsighted — limiting your ability to see a bug right in front of you. In this talk, Jon Bach will demonstrate other limiting tendencies like Stakeholder Trust, Compartmental Thinking, Definition Faith, and more. Testers can't find every bug or run every possible test, but identifying these tendencies can help us avoid traps that might compromise our effectiveness and credibility.

• How you might be susceptible to traps
• Ways through (or around) the ten most common traps
• Participate in exercises that test your situational awareness

from March 14, 2007

Panel Discussion: Steer Your Quality Career with a Career Portfolio

As Quality Professionals, we're all experienced in managing project risk. But, how good are we at managing career risk in the face of offshoring, marketplace shifts, downsizing, and distasteful company politics? One tool that you can use to provide you and your family with job security is creating a career portfolio. Just as a financial portfolio can be used to create wealth, a career portfolio can be used to develop your career assets to earn and learn throughout your life. Also, you can diversify risk so that you always have a number of career choices in front of you.

Panelists:
Brian Branagan, QA Director at Klir Technologies, will lead the panel. He will show how career portfolios are far superior to traditional resumes. He will also show how they can be used to create a personal career path and diversify risk.
Lisa Buffmire, Career Management Consultant at Right Management, will describe the basic elements of a career portfolio based on her work coaching people with the leading provider of career transition services.
Rebecca Warriner, Owner of Woodland Recruiting, will discuss her experience in placing candidates who use a career portfolio approach in their dream jobs.

Related Links:
Tips on Building a Career Portfolio: http://www.quintcareers.com/your_career_portfolio.html
Johanna Rothman article on how to develop a Career Portfolio: http://hiring.inc.com/columns/jrothman/20051130.html
Text of Rebecca Warriner's Talk (PDF): Developing a Career Portfolio

from January 17, 2007

CSI Seattle

Detective Hanf from the Seattle CSI unit will be joining us to chat about crime scene investigation - a study not far from bug investigation and one which surprisingly correlates closely to software testing. We hope to find some parallels and learn some new techniques - straight from a CSI Unit.

Detective Hanf will briefly cover all of the following areas:

Describe the make-up of the Seattle Police Crime Scene Investigation's Unit;
Explain the type of crimes that they respond to investigate;
Outline the steps that are necessary in properly documenting a crime scene;
Provide some examples on how it is different from what is shown on TV; and
Explain some of the various areas of forensic science & the technology that they deal with.


About our speaker: Detective Hanf is a twenty-six year veteran of law enforcement, with the last twenty-three years being with the Seattle Police Department. He has been an investigator in the Special Assault Unit, Domestic Violence Unit, Robbery Unit, and the Homicide & Assault Unit. He is currently an investigator in the Seattle Police Department’s Crime Scene Investigations (CSI) Unit. Throughout his career, he has gained both the experience and specialty training in the processing of various types of major crime scenes. He is a graduate of the National Forensic Academy in Knoxville, TN. In the Spring of 2004, he was tasked with the formation of the Seattle P.D. CSI Unit, which is now a permanent entity within the department.

Jon Bach's Notes (PDF)

from November 8, 2006

Lightning Talks

On November 8, there was no single speaker… there were 10!

The catch is, the format is the “lightning talk” where each of the speakers know that they only have five minutes to present their idea (one at a time). When each speaker was invited, they were told that although their time was severely limited, they could speak about anything related to testing in that time – a new idea for status reporting or measurement, a lesson they learned on a project, an experience report about a new technique they tried, or a general test principle they want to share.

Strictly timed by a moderator, each speaker will speak either extemporaneously or with slides. When the last “lightning talk” ends, the floor will open to three more audience members who are inspired enough to present their own lightning talk. After that, there will be an open networking session between speakers and attendees.

Lightning Talk Speakers and Presentation Materials

from September 13, 2006

An Integral Approach to Creating Quality Software

Presented by, Brian Branagan and John Forman.

This presentation discussed how software quality professionals can master the complexity of projects and organizational politics by making it possible to identify, organize, and effectively operate upon all the truly critical factors they face.

About our speakers:
Brian Branagan is a software quality management professional with over 19 years experience in assuring products meet customer and business requirements. He has used an integral approach to his software testing and quality engineering projects to develop his cross-functional expertise in project definition, planning, execution and customer delivery for both start-ups and Fortune 500 corporations including Adobe Systems, Getty Images and Real Networks.

John Forman is a partner in Integral Development Associates, a Seattle-based consultancy specializing in organizational leadership and management development. He is particularly knowledgeable about the role of organizational culture and values, and the many ways that people make meaning of their work-lives. His work has been included in the curriculum of graduate-level classes at the University of Washington and the National Defense University, where he has been a guest lecturer. John is also a founding member of the Integral Institute in Boulder Colorado.

For more information on Integral Theory see the following links (provided by Brian and John):

Links about Integral Theory
http://integralinstitute.org/public/static/abtapproach.aspx
http://integraldevelopment.com/theory.html

A bibliography of books about Integral Theory and Integral Business
http://integraldevelopment.com/resources_biography.html

Articles about Integral Theory in Practice
http://integraldevelopment.com/resources_articles.html

from May 10, 2006

Measurement Pitfalls

Presented by, Steve Smith, Technical Business Consultant for EMC, a manufacturer of intelligent storage systems, software and services and an independent consultant.

Steve has experienced many measurement program failures as well as a few successes. In this talk he explored with you what separates the successful programs from the unsuccessful. As part of his presentation, Steve led an experiential exercise to demonstrate a typical pitfall in thinking about measurements. He debriefed the exercise to show how to navigate around the pit or, if we are already in it, how to climb out.

About the speaker: Steve Smith works at the intersection of people, management and systems, helping organizations to become more productive.

Although most of his career has been technical — starting in 1975 as a Systems Programmer — over the years, his context has widened. Today, he tackles the most difficult problem businesses face -- managing technical teams so they produce results, rather than merely justifying and deploying the latest technology.

Steve is a consultant, author speaker, and expert facilitator and coach who honors and enhances the wisdom that lies dormant in groups. He works as a Technical Business Consultant for EMC, a manufacturer of intelligent storage systems, software and services; and sometimes freelances as an independent consultant to lead project retrospectives. He is also a host and session leader for the Amplifying Your Effectiveness (AYE, www.ayeconference.com), an annual conference held each November in Phoenix, Arizona.

from March 8, 2006

Breaking Down (and Building Up)Exploratory Testing Skill

Presented by, Jonathan Bach, Manager, Corporate Intellect and Technical Solutions, Quardev Laboratories

Scenario #1: TesterX is a superhero. For the third time today, he is hot on the trail of a great bug that may save the company millions. The Bug Council, the Triage Team, those War Room generals, they're about to make a decision today on whether to ship. At his desk TesterX works without a net, honing his critical thinking skills and scientific mind to find those last buzzer-beater bugs. Then, he finds it -- the severity 1 stop-ship crash he suspected was there -- and he files it like a scoop reporter on deadline. Other testers look on in awe. "How does he do it so reliably?"

Scenario #2: Program Manager Michelle has been in the triage meeting all day. There are 43 new bugs to look at (the day started with 103) and the minutia and nuance of bugs needs to be hashed out. It is tiring and rigorous, but rewarding. As the list narrows to 42 bugs, up pops a new one to bump it back to 43. She sees it was filed by TesterX, and immediately there is a hush in the room. He has earned a lot of credibility on the project for his abilities to find not only severe bugs, but also many less severe bugs that lead to rich discussions about quality. The new bug is read and the staff in the War Room groans collectively. This one is a meaningful issue that will take time to discuss, as they wonder, "How does he do that?" while thanking the Fates that he is on staff to protect the company from shipping unknown bugs.

In this talk, I will explain the factors that make exploratory testers valuable. It's not that they luckily "stumble" onto a bug or find it "randomly". Exploration is both craft and science -- a teachable set of skills and techniques that we all can learn. It makes no difference if we're a PhD in Advanced Computer Science from CalTech or a Journalism major from a small Maine college. We have these skills within us and use them in our everyday lives. I will describe what these skills and techniques are as well as strategies to practice them.

About the Speaker: Jon Bach is the Corporate Intellect Manager and Senior Test Consultant for Quardev Laboratories, a Seattle test lab specializing in rapid, exploratory testing. He is most known for being co-inventor (with brother James) of Session-Based Test Management (a way to manage and measure exploratory testing) and most recently the developer of Open-Book testing. In his ten-year career, he has led projects for many corporations, including Microsoft, where he was a test manager on Systems Management Server 2.0 and feature lead on Flight Simulator 2004. He has presented at many national and international conferences and is a Program Chair for the 2006 Conference for the Association for Software Testing. He is a featured keynote speaker at this year's STAREast Conference in Orlando (www.sqe.com/stareast).

from January 11, 2006

See the Slides

Evolution of Structured Exploratory Testing at Philips Ultrasound

Presented by, Doug Carlton, System Integration Test Manager and LeAnn Coker, Integration Test Lead, Philips Ultrasound.

Philips Ultrasound is a subsidiary of Philips Royal Electronics, producing consumer electronics, lighting, semi-conductors and medical devices. Philips Ultrasound has facilities in Bothell Washington and Andover Massachusetts for the research, design, development and testing of ultrasound imaging systems.

This presentation will discuss the evolution of exploratory testing starting with the original company, Advanced Technology Laboratories (ATL).

In 1997, exploratory testing was "in the closet" and referred to as "random stress testing." As an FDA regulated medical device company, Philips was required to have an independent Verification and Validation team that focused its efforts on providing documented evidence that top level system requirements were covered. The released ultrasound systems were safe and effective, but quite often, full of latent defects.

In 1999, Doug Carlton approached the Director of Software Development with a proposal to embed software testing professionals in the software development department. This proposal was accepted and two testers were hired. The charter of this group was still focused on generating manual test scripts from requirements, but exploratory testing was now out-of-the-closet as a secondary directive. This Software Integration Test (SIT) team added another tester and became very successful, identifying 75% of the severity 2 or greater defects in the tracking system.

In 2001, the SIT team had pretty much abandoned the requirements-based test scripts in favor of full-time exploratory testing. Doug Carlton and SIT lead Joe Germani teamed up to hire a tester who had a strong background in exploratory testing. Doug contacted James and Jon Bach who forwarded the name of LeAnn Coker.

By 2005, most of the original SIT team had left the company. The two remaining team members were transferred to the formal System Verification and Validation team. Immediately, upon transitioning, one of the original SIT members joined a software team as a developer. This last year, the team was re-organized and with some training by Jon Bach, is now formally devoted to exploratory testing using the following charter:

The Integration Test team emphasizes heuristic exploration of ultrasound features as opposed to stringent coverage of system requirements. The IT team uses a structured exploratory testing methodology that is driven by emerging and evolving device functionality.

The testing target focus is risk-based, using the following priorities:

1. Identified system and software risks along with derived mitigation requirements.
2. Identified negative impacts on business goals defined in the business plan.

• Collects and analyzes system and software “intelligence” data for use in identifying targets for exploratory testing
• Coordinates with the System Engineering Team to identify Feature level and Feature Interaction Matrix (FIM) focus test targets
• Creates, maintains and publishes a testing results Dashboard.

About the Speakers:
Doug Carlton holds degrees in Electronic Technology and Computer Science. He has 27 years of hardware, software and test engineering experience in the Boston, Silicon Valley and Seattle high tech locales. During that time, Doug has designed, developed and tested mass storage, process control and medical imaging systems. Prior to joining Philips Ultrasound 8 years ago, Doug spent 9 years at a start-up medical imaging company developing system software and test automation. Currently, Doug manages two teams at the Bothell,Washington, USA campus; Automation Test and Integration Test. Doug received R&D excellence awards in 2000 and 2001 and was named a Philips Ultrasound Technical Fellow in 2003.

LeAnn Coker is a Senior Development Engineer and Integration Test Team Lead at Philips Ultrasound Bothell, WA. She has more than ten years of experience in the IT industry including time spent as an Instructional designer, computer-based training developer and development engineer. During her six year testing career, she has primarily focused on exploratory testing including four years integrating software exploratory testing in a medical device company.

from November 9, 2005

Powerpoint Slides (PDF)

Anatomy of an Attack: Learning testing techniques to help secure your applications against hacker attacks.

Presented by Joe Basirico, Manager, Technology & Security Services with Security Innovation (http://www.securityinnovation.com)

This talk will discuss the basic methodology a hacker would use to compromise a server. Each step will be outlined to explain why the hacker is taking this step and the information or access the hacker hopes to obtain. After the step has been outlined the techniques and tools the hacker will use to complete this step will be covered. The techniques describe, in detail, what the hacker can discover using that method. Tools tie in directly with the techniques to aid the hacker in the discovery of sensitive information.

There are five categories including: Data Gathering, Exploitation, Elevate Privileges, Cover Tracks, Install Tools, and finally Gathering Sensitive Information from the target machine. Each attack category will be covered in depth and arm the tester with the education and tools it takes to help secure software from attack.

About our speaker: Joe has spent the majority of his educational and professional career studying security and developing tools that assist in the discovery of security vulnerabilities and general application problems.

At Security Innovation his primary responsibility is to train testers and developers from HP, EMC, Microsoft and other reputable fortune 100 organizations to find security and reliability flaws in applications and to write secure code. Basirico has delivered “Building Secure Applications in an Insecure World” at Compuware’s annual application development conference, OJ.X. He was also chosen to give two presentations for Microsoft’s Professional Developers Conference this year. He is the author and content provider of the company’s “SI Security Report”, a quarterly intelligence report that provides an in-depth analysis of the techniques and tools that an attacker could use to compromise enterprises. He has written numerous security whitepapers that focus on vulnerabilities at the source code level, including a detailed 20-page “Static Analysis Tools” report.

from September 14, 2005

Powerpoint Slides (PDF)

It's Too Darn Big To Test

Presented by Keith Stobie Test Architect at Microsoft Corporation

Structuring test designs and prioritizing your test effort for large and complex software systems are daunting tasks, ones that have beaten many very good test engineers. If you add concurrency issues and a distributed system architecture to the mix, some would simply throw up their hands.

At Microsoft, where Keith Stobie plies his trade, that is not an option. Keith and others have reengineered their testing, employing dependency analysis for test design, model property static checking, “all pairs” configuration testing, robust unit testing, and more. They employ coverage to successfully help select and prioritize tests and make effective use of random testing including fuzz testing security. Finally, models of their systems help them generate good stochastic tests and act as test oracles for automation.

About the speaker: Mr. Stobie plans, designs, and reviews software architecture and tests for Microsoft. His current project, Indigo, involves XML Messaging and Web Services using SOAP. Keith is also active in the Web Services Interoperability organization’s (WS-I.org) Test Working Group creating test tools for Profile analysis and conformance.

Keith directed and instructed in QA and Test process and strategy at BEA Systems where he worked on BEA WebLogic Collaborate, and WebLogic Enterprise. Keith was Test Architect at Informix, designing tests for the Parallel Server product, and Manager of Quality and Process Improvement. With over 20 years in the field, Keith is a leader in testing methodology, tools technology, and quality process. He is a qualified instructor for Systematic Software Testing and software inspections. Keith has been active in the software task group of ASQ, participant in IEEE 2003 and 2003.2 standards on test methods, published several articles, and presented at many quality and testing conferences. Keith has a BS in computer science from Cornell University.

from May 11, 2005

A Theory of Variation in Software Development, Architecture and Project Management

Presented by David Anderson of Microsoft Corporation

Traditional software engineering methods are built on an assumption that software engineering is deterministic and can be accurately planned in advance. Recent agile methods rebel against planning and adopt reactive adaptation to change. However, there is a better way - the introduction of a theory of variation into software engineering. By using lessons from Shewhart, Deming and Wheeler, it is possible to create predictive methods for software development, architecture and project management which embrace uncertainty and absorb change gracefully. The result is a system of quality assurance and continuous improvement for software engineering through the reduction of variation.

from March 9, 2005

Powerpoint Slides

Agile Product Management

Presented by Greg Patrick of TechMeth, Inc.

Does the narrow bandwidth, high focus and short term cycling of "Agile" development methods undermine the continuity of product management and effective market planning? Experiences with several "Agile" product development environments are examined to identify critical success factors that may help when Product Managers need to survive and thrive with these dynamic methods. One thing seems certain: Product Managers can have nightmares trying to positively influence an "Agile" product development process. One way to deal with this tendency is shifting more responsibility for detailed product definition and vision into the Product Management realm. However, this adjustment creates issues for deriving product specifications that work well for "Agile" processes and product validation.

from January 12, 2005

Powerpoint Slides

The Revolution in Manufacturing Quality - What can we learn?

Presented by Larry Schuiski, President/CEO, AGILEAN Corporation, Productivity for Performance™

There are many stories coming out of manufacturers recently about 90% improvements in quality, 70% reductions in cycle times, 30% increases in productivity. And these revolutionary results are from manufacturers that everyone thought were already pretty efficient. Agile, Lean, Six Sigma, Theory of Constraints, and Total Quality Management are more than buzz words for these manufacturers; they are radically changing how they do business. The topic of this discussion is to review what is happening out on the shop floor and what does, and does not, have a direct impact on quality software manufacturing.

from November 10, 2004

TeamTest 2005 Demo

Presented by Tom Arnold and Jason Anderson

Visual Studio 2005 Team System is an extensible life-cycle tools platform that helps software teams collaborate to reduce the complexity of delivering modern service-oriented solutions and ensure their applications are “Designed For Operations.”
Microsoft’s offerings now include a comprehensive set of proven process frameworks, best practices, prescriptive architecture guidance, and integrated life-cycle tools that enable IT organizations to successfully deliver custom solutions on the Windows Server System.

With Visual Studio 2005 Team System, organizations can:

• Reduce the complexity of delivering modern service-oriented solutions that are designed for operations.
• Facilitate collaboration among all members of a software team, including business users, project managers, architects, developers, testers, and operations managers.
• Customize and extend the Team System with their own internal tools and process frameworks or choose from over 400 supplemental products from over 200 partners.

from September 8, 2004

Powerpoint Slides
Triangle:(the dll and ocx file in your windows dir) - Zip file
TaskMaster 1.0
http://workroom-productions.com - James Lindsay's "exploratory" machines
When You're Tested - Article by Jonathan Bach
Meeting Review - by Kevin Dargie

When *You’re* Tested: Techniques for the Testing Interview

Presented by Jonathan Bach

Software testing is a process of information-discovery. But because not all problems are easily found in the user interface, it takes a broad combination of skills to unearth and report them -- creative, technical, scientific, diplomatic, and communicative, to name a few.

The same is true with job interviews. Finding a person with all of these skills takes, well… the same combination of skills! Just like software, candidates come to the interview, the bulk of their problems hidden, and the interviewer has to use their skills to quickly obtain good information about the candidate's strengths and weaknesses. Jon Bach, a managing test lead at Quardev Laboratories, tells of his experience interviewing hundreds of candidates over the past 10 years and will demonstrate a variety of proven interactive techniques (and a few unproven ones) to help you make the most of the testing interview, no matter which side of the clipboard you're on.

from July 14, 2004

Powerpoint Slides
Rapid Test Planning Zip Files

Rapid Test Planning with Quardev's Joy Shafer

Presented by Joy Shafer

Joy's talk focused on how to plan and execute a reasonable testing effort for short-term or very time-critical projects where there is no time or budget for formal planning. Very valuable and excellent presentation!

from May 12, 2004

Presentation Zip File

Exploratory Testing Bootcamp with Microsoft's Noel Nyman

Presented by Noel Nyman

Noel Nyman will reprise his semi-popular “Exploratory Testing” presentation. An avid exploratory tester, Nyman brings a unique perspective to the field, building on the seminal work of Cem Kaner, James Bach and others to provide unusual insights. He addresses such burning questions as, “How many tests do we really need for Myers’ triangle problem?” “How do I test Windows Notepad?” And, “What does the state of Kansas have to do with Exploratory Testing?” Be sure to put this meeting on your calendar and come join us for pizza, even if you decide to leave before the presentation starts.

About our speaker
Many years ago, Noel Nyman discontinued his exhaustive study of the undocumented op-codes in 65xx series microprocessors and devoted his unusual talents to creating a digital gas engine throttle controller for QS Products. The company declared bankruptcy after selling three units. Noel next moved to Aldus Corporation, the creator of PageMaker. Within six months the company was acquired by Adobe. Undaunted, he moved to Microsoft and joined the Semantic Platforms group, testing proof-of-concept applications using cutting edge technology. The group disbanded a year later. Noel then moved the Application Compatibility group, on special assignment to produce the test frameworks for the Certified for Windows 2000 logo program. The test plans, called the most rigorous testing outside the military, proved too rigorous for most ISVs and very few apps received the logo. When the logo program terminated, Noel became a regular member of AppCompat. The team immediately started a complex series of reorgs which continue to this day. In April of 2003, Noel joined the Microsoft Identity Integration Server group as a test lead. He uses minimalist heuristic, synergistic, managerial paradigms to enable the Test team to make effective contributions to the quality of MIIS. In his spare time Noel counts cars on the New Jersey Turnpike using traffic Webcams and listens to his extensive collection of Marcel Marceau vinyl records.

from March 10, 2004

Presentation slides

Model-based testing in the key of C#

Presented by Harry Robinson and Michael Corning

What would you give to be able to run zillions of distinct tests on your applications around the clock?

Model-based testing lets you generate software tests from descriptions of an application's behavior. Creating and maintaining a model is easier and more productive than writing individual tests.

We will show you how to create models and then generate and execute tests from those models, using nothing other than the C# language!

About the speakers:
Harry Robinson is Test Architect for Microsoft's Enterprise Management Division. He also writes a regular column for StickyMinds.com.

Michael Corning is Harry's minion whose day job includes testing Microsoft Operations Manager and developing model-based testing tools for general use. The publisher who published his regular column (Wrox Press), "Confessions of a Modeling Bigot," has gone out of business.

from January 14, 2004

Presentation Overview
Presentation slides
Presentation Zip File
Meeting Review - by Jonathan Bach

The Three R's of Software Testing

Presented by James Bullock and Brian Branagan

Sometimes software testing goes sideways, with nothing good happening and everyone involved simply pissed-off. Why is that? Can we do anything about it as testers, test managers, people in development, or even as people who depend on software and want more, better and sooner?

"The Three r's of Software Testing" follows the story of John, a test manger in a difficult situation, to illustrate how moods like the "three r's"—resignation, resentment and righteousness—influence behavior in software testing. The talk outlines some ways to manage mood, along with why software testing is particularly prone to these problems. James and Brian will discuss examples of language, thinking, and behavior that can help with the three r's, and will provide a "crib sheet" of tools for managing your own mood while on a project. Because these examples are not silver bullets, references are included for investigating the ideas behind the examples and techniques.

The session will include some simple exercises to work with your experience, connecting the "three r's" directly to what is going on for you.

About the speakers:
James Bullock has been successfully building systems for over 20 years. His experience includes building high-volume embedded control software for heat pumps, plant-floor manufacturing automation, enterprise scale data warehouses, software engineering tools for managing multi-million SLOC code bases, creating testing frameworks and tools and testing large-scale IT & web deployments. He has also run small software development projects, major IT deployments, conversion projects, development tool teams, project specific QA teams, QA departments supporting multiple projects, and an automated testing consulting group.

James is the lead editor for Dorset House's Roundtable on Project Management and co-editor of the Roundtable on Technical Leadership.

James lives in Seattle, where he offers consulting on software engineering processes and methods, project assessment, management or recovery, and IT department planning or reviews. You can email him at: jbullock@rare-bird-ent.com.

Brian Branagan is the Director of Quality Assurance for Getty Images in Seattle. He has successfully lead test teams to deliver great products under tight timelines at Aldus, Adobe, and PictureIQ over his 16-year history in software testing and quality engineering.

Brian has studied with Jerry Weinberg and completed his year-long Systems Effectiveness Management training in 1998. He completed Enterprise Performance's two-year "Action in Management" Executive Development program in 2003. Brian is also a member of the Project Management Institute and a certified Project Management Professional and is a Certified Software Test Engineer through the Quality Assurance Institute.

from November 11, 2003

Presentation slides
Meeting Review - by Jonathan Bach

When All is Said and Done—hosting a meaningful project retrospective

Presented by Christina Chen
Microsoft Corporation

During the life of a project, teams make mistakes and create good practices, but nobody has the time to analyze and understand them. Too often after the project ends, people don't learn from their experience and repeat earlier mistakes on subsequent projects. A well moderated retrospective helps a team constructively examine their recently completed project to carry forward knowledge and prevent problems from reoccurring. This presentation discusses how to effectively moderate a retrospective to help teams get the most out of their experiences.

About the speaker: Christina Chen has been a Program Manager at Microsoft since 1997, working on Combat Flight Simulator and the Midtown Madness series. Her hobbies include playing video games, cooking and moderating retrospectives.

from September 23, 2003

Measuring up: QA and the FDA

Presented by Andrew Jelen
Encompass International Inc.- www.encompassintl.com

Computers and computer software play a critical role in the pharmaceutical industry and are used extensively in the discovery, manufacture, and delivery of drug products. This industry is heavily regulated by the Food and Drug Administration (FDA) and, in the case of computer technology, each firm is accountable for accurate and reliable computer systems. Making this especially challenging is that these companies are increasingly dependent on external suppliers for their computer systems including both hardware and software.

As a prerequisite for FDA compliance, companies providing computer products and services to regulated manufacturers, including commercial off-the-shelf (COTS) software, are required to undergo audits of their product development process. The Parenteral Drug Association (PDA, www.pda.org) a international non-profit pharmaceutical industry association has established the Audit Repository Center (ARC, www.auditcenter.com) to help both manufacturers and suppliers more efficiently and cost effectively meet this audit requirement, avoiding the cost and duplicate effort associated with carrying out multiple audits of a company's software development operations.

Andrew Jelen, Managing Director of Encompass International (www.encompassintl.com) a WSA member company will provide an overview of ARC and the criteria used from PDA Technical Report #32 (TR#32) for evaluating a software suppliers product development process.

Andrew has over 20 years domestic and international experience in Quality Engineering, Regulatory Compliance, and Process Improvement in technical and management positions with major companies in the Medical, Software and Electronics industries. His undergraduate studies were in Engineering Technology and Business Management and his graduate work is in Information Systems. In addition to being a PDA TR#32 certified auditor he is also a ASQ Certified Quality Engineer, Certified Software Quality Engineer and an IRCA Certified Quality Management Systems Lead Auditor.

from July 8, 2003

Presentation slides
Meeting Review - by Jonathan Bach

It's a Beta... What Do You Expect?

Presented by Hal Bryan
Microsoft Corporation

Beta testing is one of the most frequently misunderstood aspects of software development. Many companies put too much trust in their Beta testers, others don't put nearly enough. In both cases, a Beta test will provide little value, and prove distracting at best.

However, when managed effectively, a Beta program can be equal parts quality assurance, market research, and public relations, significantly impacting quality at very low cost. Hal Bryan, software test lead at Microsoft, and Beta Coordinator for the long-running Flight Simulator franchise, provides an overview of their Beta programs of the past five years. Hal will discuss working with Beta testers, how to set their expectations and yours, what to do with their feedback, and provide an opportunity to learn from some of his most spectacular mistakes.

About the speaker: Hal Bryan is a software test lead for Microsoft Flight Simulator. He's been testing software for 6 years, as both a contractor and full-time employee at Microsoft, working on Windows 98, Flight Simulator, Combat Flight Simulator, and other game/simulation titles. For 5 years, he's been known for transforming Beta programs into valuable partnerships with some of Microsoft's long-standing customers. In addition, Hal wrote the "Beta Best Practices" document that is being adopted by test leads in the Microsoft Games Group.

from May 13, 2003

Presentation slides
Full article
Meeting Review - by Andrew Custer

Testing In Session: A Way to Manage Exploratory Testing

Presented by Jonathan Bach

Exploratory testing is unscripted and unrehearsed. Like the music in a jam session, its agile and unrestricted nature makes it a widely-used test method. But that nature also carries a stigma—it's often mistaken for "random testing" because it appears to have no structure, which can make it seem an inappropriate method to use on test projects that invite intense scrutiny or require extensive test documentation.

Jonathan Bach is a senior test lead at Quardev Laboratories in Seattle. He's been testing software for 8 years, half of which was for Microsoft as both contract employee and full-time test manager on Systems Management Server. Additionally, Bach worked with Satisfice, Inc. where he was lab manager and test consultant. Bach has published testing articles in STQE and Computer magazines, as well as presented testing talks at Microsoft, the University of Washington, STAR West 2000 in San Jose, and the 2001 Testing Computer Software conference in Washington, DC.

from March 11, 2003

Defect Management: Building a Better Bug Trap

Presented by BJ Rollison
Microsoft Corporation

A discussion about bug tracking best practices — building a taxonomy for bug classification in order to derive better metrics. Mr. Rollison will present ideas on how to write better bug reports and improve the defect tracking system. He will cover general ideas for improving the defect management database, discuss a few common problems with defect tracking systems, outline common problems with bug reports, and provide guidelines for creating effective and meaningful defect reports that can be addressed quickly and accurately. He will also outline a typical bug lifecycle and discuss resolution negotiation.

BJ Rollison is currently the test training manager at Microsoft. He has been with the company for 9 years. He started in '94 as the setup test lead on International versions of Windows 95, then became a test manager for the Internet client division before moving into Microsoft's internal training department about 4 years ago. Prior to Microsoft he worked for an OEM company in Japan building and testing Japanese computer and network solutions. He has published several articles on DOS memory management and Windows optimization in Japan and Canada, and is currently working on a book on International Software Testing.

from January 14, 2003

Meeting Review by Chris Bridenbaugh

The Revolution in Manufacturing Quality – What can we learn?

Presented by Larry Schuiski
President/CEO
Agilean Corporation
www.agilean.com

There are many stories coming out of manufacturers recently about 90% improvements in quality, 70% reductions in cycle times, 30% increases in productivity. And these revolutionary results are from manufacturers that everyone thought were already pretty efficient. Agile, Lean, Six Sigma, Theory of Constraints, and Total Quality Management are more than buzz words for these manufacturers; they are radically changing how they do business. The topic of this discussion is to review what is happening out on the shop floor and its impact on the future of quality software manufacturing.
Please contact Larry Schuiski if you would like to receive a copy of the presentation.

Mr. Schuiski is the founder and CEO of Agilean Corporation. Agilean's consulting services deliver common sense, low-cost, low-tech improvements to business processes. Prior to Agilean, he worked for Attachmate Corporation as the SVP of Worldwide Products and Support. Managing an organization of 400 people, he was responsible for all aspects of the company's strategic planning, product development, and technical support. Mr. Schuiski brings over 20 years experience in business process improvement, enterprise system implementation, and management consulting. He is the current Chairman of the Board for the WSA. Mr. Schuiski graduated with honors from the University of Washington with a Bachelor of Science in Physics.