The smart coverage framework

SmartQA smart coverage pic

T Ashok @ash_thiru on Twitter


Coverage, an indicator of test effectiveness is really multidimensional and has not been dealt with rigour most often(excepting for code coverage). This article outlines a “Smart coverage framework” that looks at coverage from multiple angles summarising it as a beautiful picture. 

The original article is at CLICK HERE to read the complete article.

Signup to receive SmartQA digest that has something interesting weekly to becoming smarter in QA and delivering great products.

CIO views on Quality

CIO Views on Quality Featured Image

This article is about views on quality from CIOs curated from a list of interesting articles. It is felt that solution quality is one of the Top-3 challenge doing DevOps adoption with reducing technical debt as a key focus area for 2019. Some of the interesting views from CIOs are “there’s no way you can satisfy the demands of digital transformation without DevOps, Continuous Testing”, “address testing and ensure it advances your digital transformation initiatives rather than holds them back”, “can’t risk disrupting frequent deployment,  this is where Continuous Testing comes in.”

Quality of solutions is a challenge during DevOps adoption
Based on Gartner’s 2019 DevOps Survey, ensuring the quality of solutions is among the top 3 challenges encountered during the adoption of DevOps. According to them,application leaders guiding a digital transformation initiative must make continuous quality the technical, organizational and cultural foundation of their strategy.

Many organizations are on a journey with DevOps, practicing continuous development and continuous deployment, yet a continuous approach to quality is often missing. Basic functional quality goals, is not sufficient to satisfy the quality expectations of the users, the business or the market. The growing pervasiveness of mobile, web, cloud and social computing scenarios has raised end users’ expectations for application quality. The notion of what constitutes superior quality has become much broader and includes overall user experience, quality of service (QoS), availability and performance, as well as security and privacy. It is no longer sufficient that the application just works. It must provide an optimised experience that leaves the user wanting to engage more and interact again. 


Reducing technical debt needs increasing focus
When asked CIOs “What are your top priorities for 2019?”, reducing Tech debt was 2nd most popular response. CIOs say reducing technical debt needs increasing focus. It isn’t wasting money. It’s about replacing brittle, monolithic systems with more secure, fluid, customizable systems. CIOs stress there is ROI in less maintenance labor, fewer incursions, and easier change. (Ref:

CIO’s views on Digital Transformation
Here are some views from CIO on Digital Transformation, consumer expectations, and hence the changing expectations from testing.

Rajeev Ravindran, SVP & CIO, Ryder System, Inc.
“Oftentimes, when people talk about digital transformation, they are really talking about technology. For me, taking the company “digital” is both about technology and a mindset shift. As a part of this mindset shift, we are moving from an applications-focused environment to a product-focused environment. In our new model, we look at every application as a product that has a life cycle determined by a product owner, who is typically in a business function other than IT.”

“In IT, we are moving from a linear thinking perspective to design thinking, and we are moving from waterfall to iterative. The goal of these changes is to create a customer-centric culture, whether those customers are internal or external to Ryder. The customer centric culture along with a product mindset will help with operational efficiency and revenue growth.” (Ref:

Andy Walter (Procter and Gamble)
“I think Continuous Testing is going to be core to companies being able to dynamically evolve their structures, their M and A, joint ventures, all these types of areas. While we were doing the Cody divestiture, we started a covert project of “how are companies going to be structured in the future?” And there’s no way you can satisfy the demands of digital transformation without DevOps, Continuous Testing, and the speed and agility they enable.” 

Jennifer Sepull (USAA, Kimberly Clark, American Honda)
“I think the beauty of creating a DevOps model is that you have a powerful team that is empowered to really connect with the consumer. When those teams come together in that powerful way, and they own the entire end-to-end process, there’s opportunities for innovation. Application development and testing are absolutely critical to making sure that those innovations, or that connection with the consumer, can happen.” This means that you have to address testing and ensure it advances your digital transformation initiatives rather than holds them back.” 

Robert Webb (Etihad)
“.. I know that software testers can make the CIO’s survival rate higher—but they can make the company more profitable, make it safer, and help it grow faster. If they make your testing faster and get your new apps out there, you can be more competitive. And if they can do that while lowering costs, that’s remarkable..” “..Transforming testing is pivotal for accelerating how software is digitising the business.” 

Vittorio Cretella (Mars)
We have to really understand how the user is reacting and how to achieve this optimal customer experience..”  “.. To accomplish this, we need constant deployment. But we also have to ensure that deploying functionality daily or hourly always improves the user experience. We can’t risk disrupting it—so this is where Continuous Testing comes in.” 

Andreas Kranabitl (SPAR ICS)
“..I believe that the most important element in digital transformation is people. We cannot have people spending their time on software testing tasks that can and should be automated. There is much higher-level work to do. We need future-oriented staff, and we can’t afford to make them suffer by asking them to do needless manual testing.” 

Robert Webb ( Etihad Aviation Group)
“..Can you make my testing faster and get my new apps out there so I can be more competitive? Can you do that in a way that makes testing more automated and safer, and can you do that while you’re lowering costs? This is something that is very, very unique, and I think we all have a wonderful opportunity to be part of this revolution..” (Ref:

Signup to receive SmartQA digest that has something interesting weekly to becoming smarter in QA and delivering great products.

To express well, choose the right medium

The medium that we choose to express our thoughts that emanate from our language-driven thinking is key to expressing well. Good expression is key to being clear and ensuring clarity in communication to others.

Any friction that can impede the free-flowing thoughts is very unwelcome, hence the medium of expression plays a vital role in the final transformation of thoughts to actionable ideas. A loose frictionless paper medium is very suited for early-stage ideas whilst a strict template/tool is more suited for capturing ideas fully and clearly.

Click here to read the full article published in Medium

Signup to receive SmartQA digest that has something interesting weekly to becoming smarter in QA and delivering great products.

Three communication approaches to brilliant clarity

Clarity is often a reflection of the mind and great clarity implies an uncluttered mind. The style & structure of sentence matters to the way we think, understand, design, act and analyse. How we string the sentences and communicate is key to seeing everything and spotting the missing.

A judicious mix of unstructured human friendly(descriptive) storytelling approach with structured action-oriented prescriptive (rule/criteria based) approach and fact-rich visual approach expands the cognitive abilities enabling us to see the expansive full picture and also allowing us to swoop down as necessary.

Click here to read the full article published in Medium

Signup to receive SmartQA digest that has something interesting weekly to becoming smarter in QA and delivering great products.

3 failures of poor system operationalisation

Featured image of 3 failures of poor system operationalisation article

Successful large system deployment failures is not merely due to poor testing of software. It is about poor operationalisation of software. This article outlines three major failures – Poor transition of software to end users, messed up business procedures and Data issues, the result of poor operationalisation. These have been curated from two articles.

FAIL #1 Avon : Poor transition of software to end users

In 2013 Avon’s $125 million SAP enterprise resource planning project failed after four years of work, development and employee testing.

ERP software can brag all it wants about functionality and all of the magical modules and apps you can use to make your business processes easier, but that won’t mean anything if your software isn’t actually usable. It’s all about aligning your software to your business processes, and if you can’t get staff to use your ERP, they won’t be carrying out the processes necessary to keep your business running. Make sure your employees are properly trained and transitioned into the new software, and that they want to use that system in the first place.

Read about this in detail at

FAIL #2 Woolworth : Business procedures messed up

The Australian outpost of the venerable department store chain, affectionately known as “Woolies,” also ran into data-related problems as it transitioned from a system built 30 years ago in-house to SAP. 

The day-to-day business procedures weren’t properly documented, and as senior staff left the company over the too-long six-year transition process, all that institutional knowledge was lost — and wasn’t able to be baked into the new rollout.

Read about this in detail at

FAIL #3 Target Canada : Data issues

Many companies rolling out ERP systems hit snags when it comes to importing data from legacy systems into their shiny new infrastructure. The company’s supply chain collapsed, and investigators quickly tracked the fault down to this supposedly fresh data, which was riddled with errors -items were tagged with incorrect dimensions, prices, manufacturers, you name it.

Thousands of entries were put into the system by hand by entry-level employees with no experience to help them recognise when they had been given incorrect information from manufacturers, working on crushingly tight deadlines. It was later found that only about 30 percent of the data in the system was actually correct.

Read about this in detail at

About SmartQA The theme of SmartQA is to explore various dimensions of smartness to leapfrog into the new age of software development, to accomplish more with less by exploiting our intellect along with technology.  Towards this, we will strive to showcase interesting thoughts, expert industry views through high-quality content as articles, posters, videos, surveys outlined as a SmartQA Digest weekly emailer. SmartBites is “soundbites from smart people”. Ideas, thoughts and views to inspire you to think differently.

Signup to receive SmartQA digest that has something interesting weekly to becoming smarter in QA and delivering great products.

Role of human intellect in the digital age

T Ashok @ash_thiru

Over the last few months, as a producer of SmartBites for the SmartQA channel, I asked a question to EIGHT senior software folks spanning across the roles of management, technology& architecture, QA and development.

“In this digital age, what do you believe is the role of human intellect in QA?”

The answers were different and very interesting, summarised as :

  • Increased need for QA intervention at back end
  • Intelligent UI automation requiring less human intervention
  • Focus on back end
  • Intelligent systems will augment our capabilities
  • It is QA as a mindset that will prevail, may not be the department
  • AI/ML may not understand business context, be the voice of customer
  • Tools cannot come up with negative scenarios
  • Intellect systems will foster more test innovations
  • To suspect everything, think like Sherlock Holmes
  • Intelligent systems will assist in making decisions
  • Intellect is needed to judge adequacy
  • Behave as first client
  • 1000s of configuration, 1000s of customer varieties

Now continue reading in-depth from the horse’s mouth!

Sudhir Patnaik says:
With the way AI & ML coming into the play, my instinct says that there will come a time where a developer doesn’t need to write code. You just tell the chat bot the business logic and it spits out the code. In fact, to some extent it’s happening today – where the template code is ready  and you just plug in the business code. I think even in the testing world that is going to happen. Someday automation is going to be SO intelligent, point it to the URL (UI) and it will do everything. There is no need for human intervention. It self-corrects whenever the things change in the website or URL!

I think that’s where it is headed. Sometimes when we say test automation, we unfortunately focus only on UI automation. A big work piece is platform automation and companies around the world are transforming themselves to be known as platform organizations. There is more technology on the platform side. I think the human interface on the UI side will minimise, and this is where the transformation is. I see it more on the back end, as the front end is going to be driven through intelligent systems. So, our intervention as quality engineers on the front-end side will get minimised over a period of time. 

With respect to a piece of code, unit testing requires you to focus on code. And it is not easy for any computer system to understand the piece of code and magically spit out unit test code.

An intelligent software can write positive and negative test cases, but that may be about 50% of the coverage with the remaining 50% from a human. I think that is where the intellect is still needed. I don’t think chatbots will take over. Click here to watch this smartbits video.

Anuj Magazine says:
That’s an interesting question and I’d like to answer this with an analogy. Playing chess is often associated with great deal of mystique as someone having a great deal of intelligence. Garry Kasparov who was the world champion in chess for a long time wrote a very insightful book titled “Deep Thinking”, where he captures a part of his journey. He talks about his experience when he decided to play with computers for the first time despite being a world champion on a public stage in 1980s. The first match was a walkover, the first series he won without much effort and that proved that humans are still superior to computers. He played a few public matches till around 1988 and 1989 and then a computer named “Deep Blue” beat Kasparov. People wrote obituaries of Kasparov, his skill as a chess player and also of chess as a sport . We are twenty years from that time and we have the champion Magnus Carlsen who is winning almost everything that chess has to offer and a lot of people are terming him as the greatest chess player of all-time. 

If you rewind this whole sequence of things, first humans fought with the machines and won, and machines kept getting better and then machines were able to beat humans at that time. And then we are at a stage where especially in sports like chess where Magnus Carlsen, Vishwanathan Anand use computers to augment their already supernatural abilities that they have. Machines and humans are working together to make the whole sport better with lot of invincibility and lot of new people players are emerging. So this really tells a lot about what is going to happen in the existing professions as well. 

So the role of human intellect is not going to go down in any way. What will happen is, with all the automation that is happening (my hypothesis), it will return more time back to the human beings and it is up to the human beings to utilise that time to make them more better at what they do,  their professions. 

Interestingly from what I have read and observe like machines don’t need to mimic human beings to get better than them. If you look at aeroplanes, they don’t need to flap their wings, helicopters don’t need wings at all. So they will find numerous ways to get better than what we do. The engines replacing bullock cart was another example so we don’t need to have someone pushing the engine like the bullock used to do earlier. In order to stay relevant, human intellect is going to be really important. 

Even from QA standpoint it is very important for QA professionals to adapt QA more as a mindset than a department. Let me give you an example gleaned from interacting with one of the departments in my organisation hat takes care of FedRAMP certification a security certification needed to sell to the Federal Government in the US. If you look at the structure of FedRAMP certification, there are criteria that are stringent, criteria that has to be met and the people who do the certification for you are the people who are well read and well aware of the criteria and what happens around it and they follow the procedure as well as apply their knowledge to certify.

They do a lot that a QA person will do in a different context, but they are not called as QA Engineers, they are called as the enablers of certification. People should be using their intellect to enhance the QA mindset and apply that while going in different direction that QA mindset is not going to get extinct, may be department may get overhauled depending upon the way the different organizations think about the value of it. Click here to watch this smartbits video.

Vivek Mathur says:
Understanding the business context, acting as the voice of the customer are things that are not reproducible as yet and ultimately may never be because AI, ML will depend on the quality of the data that you get. What we get is a biased data, that is not representative sample of the world, but a sample of a certain subset of people and if you try to extrapolate that to everybody, you end up pushing that bias into the system and into the results of the AI as well. So it’s that human understanding, the ability to say whether this is the right thing, it is ethics of the situation, it is the humanness, it is the empathy, the emotional connect with the end user , which is the problem that we as intellectuals need to solve and that’s what we need to propagate, and that’s what we need to be the as spokesperson for, as part of the QA function. Click here to watch this smartbits video.

Srinivasan Desikan says:
Human intellect cannot be replaced at all. We talk about negative testing, we talk about positive testing, and the plethora of tools & processes can assist in positive testing, not the negative testing.  None of the case tools cannot give you the negative test scenarios. If we look at the Boeing crash that had happened, it was not a testing failure. It was only a failure of a negative scenario, which was not explored while developing the product itself. It was a nose problem, the autopilot that was running the aeroplane was trying to adjust the nose so that it gets the elevation. Rather, it put the nose too much down, and it went into a crash.

So the testing is something which should be left to 90 percent automated and at least 10% manual so that we can catch those kind of errors. So the human intellect in QA will continue to grow and we are going to have more and more automation. We are going to have more and more innovative testing that may not be in an ‘automated way’, but in a ‘manual way’ because innovative testing happens only once in a while. It is not something which is mundane that you will repeat, for if you repeat it is not termed innovation. So from that perspective the human QA intellect will continue to not only survive but grow. Click here to watch this smartbits video.

Jawahar says:
Our industry is not like other parallel Industries where after the design and validation, pretty much things can be manufactured . Machines can do this job. There are predictions that quite a few things will be taken up by machines  even in the software industry, but I strongly believe that  the ultimate deliverable of our industry is something still produced by humans and it has to be validated by humans. Pretty much the validation team will have to behave like Sherlock Holmes and the kind of intellect that he has is what I would expect the teams to have – “to suspect everything”.  It is not a lack of trust , it is trust-but-verify principle there. So I think intellect plays a very important role in both teams – construction as well as for validation teams for a long time to come. Click here to watch this smartbits video.

Arun Krishnan says:
I always maintain that analytics is a platform, AI or ML is a platform that is going to enable humans to make decisions. For example, there are already models that can predict based on looking at X-rays, the propensity of somebody having a cancer for instance, but would we  completely stop using human intellect? I think that would be a mistake, in any field. 

Recent case in point is the air crash that took place in Ethiopia, where the plane completely controlled by an algorithm. If only the humans had disengaged this, the crash may have been averted. A recent Twitter spat between Elon Musk and Mark Zuckerberg was if AI will be beneficial or pose an ethical issue. Well I am the side of Elon Musk, while Zuckerberg has a very rosy vision, which I don’t think it is at all. 

I grew up reading Asimov, the robot series, and the three laws of robotics got into me when I was a kid.  In the book the those laws of robotics were circumvented in very unique ways in certain circumstances. I read that  Google is starting to think about the ethics of AI, which means you do not only build in the ethics programmatically, but also have the human override.

 While I am all for AI helping testing, I think it still is a role for human intellect.  It might sound a little wishy-washy but I think you still have to ensure that human intellect has a veto power so that you can shut off the AI switch if you think it is isn’t telling, if it can be catastrophic. Click here to watch this smartbits video.

Sriramadesikan says:
It is always needed, it plays a great role irrespective of whatever be the level of automation, people are talking bringing in a lot of machine learning, artificial intelligence and so on, but in the end as ultimately one needs to put their brains to see what the adequacy is,  how much of the flexibility is needed in the given time with the accelerated development. Click here to watch this smartbits video.

Raja Nagendra Kumar says:
Today’s products are used by thousands of folks, here the QA role has become 10X.  QA has to start behaving as a first client themselves. It is not just about getting into the requirement, it is about wearing the customer which in this digital age which could be 100s of different varieties of customers. It is not more about “I have complied to SRS”.

Second, the cloud has given us too many configuration approaches. How do you deploy? How do you scale? How do you perform? we need to consider both the software side and the hardware side. Now all these configurations are not easy to be done,  even by QA if they want to do it manually, they have to become coders. So basically, DevOps are QA.

Need to use one’s intellect to handle so many varieties of deployment ,so many customers intent to bring to developer’s notice.. If they have any problem in testing it they have to communicate to development saying that, this is a requirement I am not able to test. Make sure that the developer addresses it because that is a good input for a great developer. If it is not testable, if it is not maintainable  which means it is not verifiable, which is not a good way to say ‘I have done’, as a developer. 

Lot of information in enterprise is coming from various sources, one is the QA, one is the support, other is production, all  they are supposed to do is the first level of filtering . “Why they were not able to capture that” and then give it in a way where developers are able to consume it faster. If they can really do that well, they are fulfilling not just a QA job, they are  representing 1000s of varieties of customers, 1000s of configurations in the cloud. All this complexity requires a lot of intellect. Need to have a dev mindset, know to comfortable with  coding. Click here to watch this smartbits video.

About SmartQA The theme of SmartQA is to explore various dimensions of smartness to leapfrog into the new age of software development, to accomplish more with less by exploiting our intellect along with technology.  Towards this, we will strive to showcase interesting thoughts, expert industry views through high-quality content as articles, posters, videos, surveys outlined as a SmartQA Digest weekly emailer. SmartBites is “soundbites from smart people”. Ideas, thoughts and views to inspire you to think differently.

Signup to receive SmartQA digest that has something interesting weekly to becoming smarter in QA and delivering great products.